Part 10 — Fan-Out / Fan-In: Running AI Tasks in Parallel

Part of the series: Production-Grade Concurrent AI Systems in Go

→ Full code for this post: github.com/madmmas/go-concurrent-ai-systems/tree/part-10 → Diff from Part 9: compare/part-09...part-10 → Run it: go run ./cmd/news-processor -articles=3 -workers=2 inside arc-2-production/part-10-fan-out-fan-in

Arc 1 closed with a solid worker pool. Articles flow into a jobs channel, workers process them one by one, results come out the other end. It is race-free, bounded, and handles context cancellation cleanly.

But look at what each worker actually does per article:

Summarisation      → wait ~700ms
Sentiment Analysis → wait ~500ms
Keyword Extraction → wait ~300ms

Total: ~1,500ms of waiting, in a line

Three tasks. Three separate LLM calls. Each one waits for the previous to finish before starting. The CPU is idle for almost all of it. And here is the thing — there is no reason these three tasks depend on each other. The sentiment analyser does not need the summary. The keyword extractor does not need the sentiment score. They are three independent calls that happen to be running on the same article.

That is the problem Part 10 fixes.

The Pattern: Fan-Out Then Fan-In

Fan-out means launching multiple goroutines from a single point, each doing independent work. Fan-in means collecting all their results back into one place before proceeding.

Applied to our pipeline:

                 ┌─ Summarisation goroutine ─┐
Article worker ─┤─ Sentiment goroutine      ─┼─→ fan-in → AIResult
                 └─ Keyword goroutine        ─┘

All three tasks start at the same moment. The article result is ready when the slowest task finishes — not when all three finish in sequence.

The math is straightforward. If each task takes 300–700ms:

Same work. Half the wall-clock time. The tasks run in the same workers from Part 9 — only what each worker does inside its article-processing function has changed.

Implementing Fan-Out

The outer worker pool from Arc 1 is untouched. The change is entirely inside processArticleFanOut — the function each worker calls per article.

A typed channel carries results from each task goroutine back to the collector:

type taskResult struct {
    name  string
    value string
    err   error
}

Then the fan-out — three goroutines launch simultaneously, each sending one result when done:

func (p *FanOutPool) processArticleFanOut(ctx context.Context, article model.Article) model.AIResult {
    taskCh := make(chan taskResult, 3)
    var wg sync.WaitGroup

    tasks := []struct {
        name string
        fn   func(context.Context, int) (string, error)
    }{
        {"summarise", p.summarise},
        {"sentiment", p.sentiment},
        {"keywords",  p.keywords},
    }

    for _, t := range tasks {
        wg.Add(1)
        go func(name string, fn func(context.Context, int) (string, error)) {
            defer wg.Done()
            val, err := fn(ctx, article.ID)
            taskCh <- taskResult{name: name, value: val, err: err}
        }(t.name, t.fn)
    }

    go func() {
        wg.Wait()
        close(taskCh) // signals the fan-in loop to exit
    }()

    // Fan-in: collect until channel is closed and drained
    result := model.AIResult{ArticleID: article.ID}
    for tr := range taskCh {
        if tr.err != nil {
            result.Err = tr.err
            continue
        }
        switch tr.name {
        case "summarise": result.Summary   = tr.value
        case "sentiment": result.Sentiment = tr.value
        case "keywords":  result.Keywords  = []string{tr.value}
        }
    }
    return result
}

Three details worth pointing out.

taskCh is buffered with capacity 3. Each goroutine sends exactly one result, so all three can complete and send without blocking — even if the fan-in collector has not started receiving yet. Without that buffer, a goroutine finishing early would stall waiting for the range loop to catch up, and if the channel were full, you would have a deadlock.

The closer goroutine is in a separate goroutine for the same reason as Part 5's channel close: if wg.Wait() ran inline before the range loop, it would deadlock — the wait blocks until all three goroutines send, but the goroutines are blocked waiting to send with nobody receiving.

The fan-in is a plain range taskCh. Results arrive in whatever order the tasks finish — keyword extraction might beat summarisation on a given run. The switch on tr.name routes each result to the right field regardless of arrival order.

What the Output Looks Like

Fan-Out pipeline: 3 articles, 2 workers
─────────────────────────────────────────────────────────
[article 1] fanning out 3 tasks
[article 2] fanning out 3 tasks
  [1] Summarisation started (689ms)
  [1] Sentiment Analysis started (447ms)
  [1] Keyword Extraction started (286ms)
  [2] Summarisation started (253ms)
  [2] Sentiment Analysis started (534ms)
  [2] Keyword Extraction started (302ms)
  [2] Summarisation completed
  [1] Keyword Extraction completed
  [2] Keyword Extraction completed
  [1] Sentiment Analysis completed
  [2] Sentiment Analysis completed
[article 2] fan-in complete — all tasks done
[article 3] fanning out 3 tasks
  [1] Summarisation completed
[article 1] fan-in complete — all tasks done
...

Duration  : 1.23s

Articles 1 and 2 start their tasks simultaneously — all six "started" lines appear before any "completed" lines, because both workers fan out at once.

Completions arrive out of order. Article 2's summarisation completes before article 1's keyword extraction, even though article 1 started first. Keyword extraction for article 1 finished first because it drew the shorter latency (286ms vs 689ms for summarisation). The fan-in handles this naturally — range taskCh takes whatever arrives next.

Three articles in 1.23 seconds. In Arc 1 the same three articles would have taken roughly 4–5 seconds. The worker pool itself is unchanged — the only difference is what happens inside each worker per article.

Two Levels of Concurrency

Part 10 introduces a second level of concurrency sitting on top of Arc 1's first:

Level 1 (Arc 1, unchanged): Multiple articles process concurrently across workers. Worker 1 handles article 1 while worker 2 handles article 2 at the same time.

Level 2 (Part 10, new): Multiple tasks process concurrently within each article. Summarisation, sentiment, and keyword extraction all run simultaneously inside the same worker.

These compose naturally. With 5 workers and 3 tasks each, up to 15 LLM calls can be in-flight at the same moment. The two levels are independent: worker count is tuned against rate limits and article volume, fan-out within each worker happens automatically.

One Failure Does Not Stop the Others

With serial tasks, a failure short-circuits everything. If summarisation fails, sentiment and keyword extraction never run.

With fan-out, all three tasks launch before any result is known. If summarisation fails and returns an error, sentiment and keyword extraction keep running in their goroutines. The fan-in collects all three:

for tr := range taskCh {
    if tr.err != nil {
        result.Err = tr.err
        continue // record the error, keep collecting the rest
    }
    // ... assign successful results
}

Whether this is the right behaviour depends on the downstream consumer. If partial results are useful, collecting everything and flagging the failure is the right call. If all three fields must be present for the result to be valid, you would want to cancel the sibling tasks the moment one fails — that is what errgroup does, and Part 12 covers it.

What's Next

Fan-out within a worker gives us parallelism at the task level. But each article still flows through a single monolithic function that handles everything from fetch to summarisation, and all workers are configured identically.

Part 11 introduces pipeline stages. Instead of one function per article, work splits into discrete stages — scrape, clean, embed, summarise — each running concurrently with the others and each configurable with its own worker count. Scraping is IO-bound and can run with 10 workers. LLM calls are rate-limited and should run with 3. That asymmetry cannot be expressed in the current design.

See you in Part 11.

This is Part 10 of the series "Production-Grade Concurrent AI Systems in Go," and the first part of Arc 2 — Production Concurrent AI Systems. Read Part 9 — Cancellation and Graceful Shutdown or continue to Part 11 — Multi-Stage Pipeline.