Part of the series: Production-Grade Concurrent AI Systems in Go
→ Full code for this post: github.com/madmmas/go-concurrent-ai-systems/tree/part-18 → Diff from Part 17: compare/part-17...part-18 → Run it: go run ./cmd/news-processor -articles=8 -workers=3 inside arc-2-production/part-18-goroutine-leaks
A goroutine leak is a goroutine that starts and never exits. It is not a crash. It is not an error. The pipeline keeps running, articles keep flowing, and nothing looks wrong — until you notice the process is using 2GB of memory when it should be using 200MB, or the scheduler is visibly struggling, or the Kubernetes pod keeps getting OOM-killed every few hours.
Goroutine leaks are one of the most common production bugs in concurrent Go programs, and AI pipelines create them easily. A streaming goroutine whose consumer exits early. A worker blocked on a channel send after the receiver has gone. A goroutine waiting on ctx.Done() from a context that was never cancelled. Each one is a tiny memory footprint — a goroutine stack is a few kilobytes — but a pipeline processing thousands of articles per minute can accumulate thousands of leaked goroutines before the effect becomes noticeable.
Part 18 shows how to detect them, how to reproduce them intentionally so you understand the failure mode, and why the patterns used throughout this series prevent them.
Detection: runtime.NumGoroutine()
The simplest goroutine leak detector is a count before and after:
before := runtime.NumGoroutine()
pool.ProcessAll(ctx, articles)
time.Sleep(100 * time.Millisecond) // allow goroutines to exit
after := runtime.NumGoroutine()
if after > before+2 {
fmt.Printf("possible leak: %d goroutines remain\n", after-before)
}
If after is significantly higher than before, goroutines that should have exited when ProcessAll returned are still running. The +2 tolerance accounts for Go's own background goroutines (GC, finalizer) that may appear and disappear around test boundaries.
The 100ms sleep is important. When a function returns, the goroutines it launched do not exit instantaneously — they need a scheduler cycle to run their cleanup code. Without the sleep, you can get a false positive: goroutines still in their exit path that would have finished cleanly given a moment.
Running this in a test:
runtime.GC()
time.Sleep(50 * time.Millisecond)
before := pipeline.CountGoroutines()
pool.ProcessAll(context.Background(), articles)
time.Sleep(100 * time.Millisecond)
runtime.GC()
after := pipeline.CountGoroutines()
// before=1, after=1 — no leak
What the binary shows in production:
Goroutines before: 1
[article 1] done
[article 2] done
...
Goroutines after: 1 (delta: +0)
✓ No goroutine leak detected
A Leaky Pipeline
The LeakyPool in this part demonstrates the most common leak pattern: an unbuffered results channel whose consumer exits before it is fully drained.
func (p *LeakyPool) ProcessAllLeaky(ctx context.Context, articles []model.Article) []model.AIResult {
resultsCh := make(chan model.AIResult) // unbuffered
var wg sync.WaitGroup
for _, a := range articles {
wg.Add(1)
go func(art model.Article) {
defer wg.Done()
time.Sleep(50 * time.Millisecond)
resultsCh <- model.AIResult{ArticleID: art.ID} // BLOCKS if consumer is gone
}(a)
}
go func() {
wg.Wait()
close(resultsCh) // LEAKED if workers are blocked
}()
var results []model.AIResult
for r := range resultsCh {
results = append(results, r)
}
return results
}
Under normal conditions this works fine. But if the caller times out, panics, or returns early — anything that causes it to stop ranging over resultsCh — every worker goroutine blocks on resultsCh <- result forever. The closer goroutine blocks on wg.Wait() forever, because the workers are blocked and never call wg.Done(). The entire set of goroutines is frozen.
This is circular: workers need the consumer to receive before they can call Done, consumer has exited, workers are stuck, closer is stuck waiting for workers.
The Three Fixes
Fix 1: Buffer the results channel.
resultsCh := make(chan model.AIResult, len(articles))
With a buffered channel sized to the article count, workers can send their results without a receiver present. Even if the consumer exits early, workers complete their send and call wg.Done(). The closer goroutine unblocks and closes resultsCh. The goroutines exit cleanly.
Fix 2: Propagate context cancellation.
for article := range jobs {
select {
case <-ctx.Done():
resultsCh <- model.AIResult{ArticleID: article.ID, Err: ctx.Err()}
continue
default:
}
// ... process article
}
When the pipeline context is cancelled, workers stop processing new articles and send cancellation results into the (buffered) channel. They do not get stuck waiting for a receiver. They drain the jobs channel and exit.
Fix 3: Use a jobs channel, not a direct goroutine per article.
The worker pool pattern — a fixed set of goroutines ranging over a jobs channel — is inherently leak-resistant. Workers exit when the jobs channel closes. The jobs channel closes when the producer finishes feeding articles. The producer finishes when it has fed all articles or its context is cancelled. Every exit path is defined and reachable.
The leaky pattern spawns one goroutine per article. Each of those goroutines is a separate entity that must individually find a way to exit. With a jobs channel, there are only Workers goroutines in total, and their exit condition is simple: the jobs channel closes.
Production Leak Detection with pprof
runtime.NumGoroutine() tells you that a leak happened. pprof tells you which goroutines are stuck and where they are blocked.
Add the pprof HTTP handler to any long-running service:
import _ "net/http/pprof"
go func() {
http.ListenAndServe("localhost:6060", nil)
}()
Then query the goroutine profile:
go tool pprof http://localhost:6060/debug/pprof/goroutine
The output shows every goroutine and its current stack trace. Leaked goroutines appear as goroutines blocked on channel operations:
goroutine 47 [chan send, 5 minutes]:
main.(*LeakyPool).ProcessAllLeaky.func1(...)
/app/pipeline/leaky.go:38
chan send, 5 minutes means the goroutine has been blocked trying to send on a channel for 5 minutes. That is the signature of a leak caused by an abandoned channel. If you see dozens or hundreds of goroutines with the same stack trace and a long block time, you have found the leak.
For a streaming pipeline, the tell-tale sign is a goroutine blocked on a channel send inside a token producer goroutine — the consumer exited, the producer is stuck, the streaming session is frozen but the goroutine is still running.
Why the Series Patterns Prevent Leaks
Looking back at Parts 10 through 17, every pipeline uses the same structural pattern that makes leaks unlikely:
Buffered results channels — make(chan AIResult, len(articles)) means workers never block on send. Even if the collector exits early, workers complete and call their deferred cleanup.
Context propagation on every blocking call — every llm.Call(ctx, ...), every time.Sleep replaced by time.After in a select, every channel send in a loop preceded by a ctx.Done() check. If the context is cancelled, the goroutine has a path to exit.
Closer goroutines always run — go func() { wg.Wait(); close(out) }() in every stage of Part 11's pipeline. The closer is a separate goroutine specifically so it can wait on the WaitGroup without blocking the collector. As long as every worker eventually calls wg.Done() — which they do because they either process successfully or exit via context cancellation — the closer fires and the downstream stage exits cleanly.
Deferred cancels — defer cancel() on every context.WithTimeout. The cancel releases resources from the context machinery, including any goroutines that the context runtime tracks internally.
None of these patterns are complicated individually. The complexity is in remembering to apply all of them, every time. The test in Part 18 that checks runtime.NumGoroutine() before and after ProcessAll is the automated enforcement: if any of these patterns are missing, the goroutine count after the run is higher than before, and the test fails.
What's Next
Part 18 completes the diagnostic part of Arc 2. You now have tools to detect, reproduce, and prevent the most common concurrency failures in a production AI pipeline.
Part 19 is the culminating part of the arc: a full concurrent RAG pipeline — chunk, embed, retrieve, generate — that applies every pattern from Parts 10 through 18 in one system. It is the closest thing in this series to a complete production AI backend.
See you in Part 19.
This is Part 18 of the series "Production-Grade Concurrent AI Systems in Go." Read Part 17 — Token Streaming or continue to Part 19 — Concurrent RAG Pipeline: Putting It All Together.