Part of the series: Production-Grade Concurrent AI Systems in Go
→ Full code for this post: github.com/madmmas/go-concurrent-ai-systems/tree/part-16 → Diff from Part 15: compare/part-15...part-16 → Run it: go run ./cmd/news-processor -articles=8 -workers=2 -queue=2 inside arc-2-production/part-16-backpressure
Parts 10 through 15 focused on what happens when individual LLM calls fail — concurrent task management, retries, rate limiting, circuit breaking. Part 16 steps back and looks at the system from the outside: what happens when the pipeline itself receives work faster than it can process it?
Consider the pipeline from Part 11. A scrape stage with 10 workers feeds a summarise stage with 3 workers. Under light load, this works fine — the 10 scrapers produce articles faster than the 3 summarisers can handle them, but the queue between stages absorbs the difference. Under heavy load, the queue grows. If the queue is unbounded — just a regular slice or an infinitely buffered channel — it grows without limit. Eventually memory runs out, the process crashes, and every article in the queue is lost. You have traded a slow pipeline for a dead one.
Backpressure is the mechanism that prevents this. Instead of letting the queue grow, you cap it. When the queue is full, the producer blocks. The slow signal propagates upstream naturally: summarisers are slow, so the scrape-to-summarise queue fills, so scrapers block waiting to enqueue, so the scraping rate drops to match the summarisation rate. No explicit coordination, no polling, no "please slow down" message. The bounded channel does it automatically.
The Mechanism
In Go, a buffered channel is the backpressure mechanism. The capacity is the contract.
jobs := make(chan model.Article, queueDepth) // bounded queue
The producer sends into it:
go func() {
defer close(jobs)
for _, article := range articles {
select {
case <-ctx.Done():
return
case jobs <- article: // blocks here when queue is full
fmt.Printf("[producer] queued article %d\n", article.ID)
}
}
}()
When jobs has queueDepth items in it and the producer tries to send the next one, the send blocks. The producer goroutine suspends. It will not resume until a consumer removes an item from the channel, creating space. When the consumer is slow — because each article takes a long time to process — the producer is automatically slowed to match.
The workers on the consumer side never change:
for w := 0; w < p.Workers; w++ {
wg.Add(1)
go func() {
defer wg.Done()
for article := range jobs {
// process article
}
}()
}
They range over jobs until it closes. They do not know or care whether the producer is fast or slow. The channel handles the synchronisation.
This is one of the most elegant properties of Go channels: they are inherently backpressure-capable. You do not need a separate backpressure library, a reactive streams framework, or explicit flow control messages. The bounded channel send blocks when full. That is all you need.
Seeing It in Action
Run the same 8 articles with different queue depths and 2 workers:
queue=8 (no pressure): Duration: 1.847s
queue=2 (moderate): Duration: 2.163s
queue=1 (extreme): Duration: 1.568s
The timing differences are small here because the bottleneck is the LLM calls, not the queuing. The real effect of backpressure shows up in memory, not timing. With queue=8, all 8 articles are enqueued immediately — 8 Article structs in memory simultaneously, waiting. With queue=1, only 1 article is enqueued at a time — the producer is blocked waiting for the worker to consume the current item before it can send the next.
For a pipeline processing millions of articles per day, the difference between "queue everything immediately" and "queue one at a time" is the difference between stable memory usage and an out-of-memory crash at 3 a.m.
Choosing Queue Depth
Queue depth is a tuning parameter, not a constant. The right value depends on two things: how much memory you are willing to use for buffering, and how much latency variation you can absorb.
A deeper queue absorbs more burst. If your producer occasionally fires a batch of 50 articles in a short window, a queue depth of 50 lets all of them enqueue immediately without blocking the producer. Workers drain the queue at their own pace. The cost is memory: 50 articles sitting in a channel.
A shallower queue keeps memory usage tight but makes the pipeline more sensitive to rate mismatches. With queue=1, the producer can only be one article ahead of the consumers. If a consumer takes longer than usual on one article — a slow LLM call, a timeout — the producer blocks until it finishes. The pipeline runs at exactly the pace of the slowest worker.
In practice, a queue depth of roughly 2× your worker count is a reasonable starting point. It allows one full round of workers to be busy while the next round queues up, without letting the queue grow arbitrarily deep. Measure actual queue depth under production load and adjust from there.
What Backpressure Does Not Solve
Backpressure controls the rate at which work enters the pipeline. It does not make slow consumers faster. If your LLM calls take 800ms each and you have 3 workers, your pipeline can process roughly 3.75 articles per second regardless of how shallow the queue is. A tight queue depth slows the producer to match — it does not speed up the consumers.
If the producer is consistently faster than the consumers and you need higher throughput, the answer is more consumer workers, a faster provider, or fewer tasks per article — not a deeper queue. Backpressure is a safety valve, not a performance booster.
It also does not help when the producer itself is the bottleneck. If your scraper is the slow component and the LLM workers are idle waiting for input, a bounded queue between them helps nothing — there is no pressure to propagate. In that case, more scraper workers or faster scraping is the fix.
The Interaction with Rate Limiting
Part 14's rate limiter and Part 16's backpressure operate at different points in the pipeline and serve different purposes.
The rate limiter sits at the LLM call boundary — it controls how many requests leave the pipeline per second, protecting the external provider from being overwhelmed. The bounded channel sits at the input boundary — it controls how many articles enter the pipeline, protecting the pipeline itself from being overwhelmed by its own producer.
Both are needed. A pipeline with rate limiting but no backpressure can still be flooded with articles from a fast producer, queuing millions of items in memory before the rate limiter even sees them. A pipeline with backpressure but no rate limiting queues articles properly but still sends LLM calls at whatever rate the workers can achieve, which may exceed the provider's limit and trigger 429s.
Together they bracket the pipeline: backpressure caps what enters, rate limiting caps what exits.
What's Next
Part 16 completes the flow control story. The pipeline can now handle fast producers, slow providers, transient failures, rate limits, and sustained outages without losing work or running out of memory.
Part 17 turns to the output side: what happens when the LLM response itself is not a single value but a stream of tokens arriving incrementally. Modern LLM APIs stream their responses — you receive the first token in hundreds of milliseconds, and the rest arrive one by one over the next several seconds. Processing a streaming response requires a different approach from everything built so far, and channels turn out to be the natural fit.
See you in Part 17.
This is Part 16 of the series "Production-Grade Concurrent AI Systems in Go." Read Part 15 — Circuit Breaker or continue to Part 17 — Token Streaming: Processing LLM Output as It Arrives.