Part of the series: Production-Grade Concurrent AI Systems in Go
→ Full code for this post: github.com/madmmas/go-concurrent-ai-systems/tree/part-17 → Diff from Part 16: compare/part-16...part-17 → Run it: go run ./cmd/news-processor -articles=2 -workers=2 inside arc-2-production/part-17-token-streaming
Every LLM call in the series so far has been modelled the same way: send a request, wait for the response, receive the full text as a single string. That is how the simulator works, and it is how many tutorial examples work. But it is not how real LLM APIs work.
OpenAI, Anthropic, and most production LLM providers stream their responses. When you call the API with streaming enabled, you do not wait for the full summary to be generated before receiving anything. Instead, you receive a stream of tokens — individual words or word fragments — as they are generated. The first token might arrive in 200ms. The full response might take 3 seconds. In between, you receive dozens of tokens one by one.
This changes the architecture significantly. You can no longer treat the LLM call as a blocking function that returns a string. You need a way for the response producer to emit tokens as they arrive while the consumer processes them incrementally. In Go, this is a channel.
Why Streaming Matters
The user experience difference is stark. In a non-streaming system, a user who requests a summary waits 3 seconds staring at a blank page before the full text appears. In a streaming system, the first few words appear almost immediately, and the rest follow as they are generated. The total time is the same, but the perceived responsiveness is completely different.
Beyond user experience, streaming enables early processing. If you are looking for specific keywords or extracting structured data from a summary, you can start scanning from the first token rather than waiting for the full response. If the first few tokens indicate the article is not relevant, you can cancel the stream and save the tokens you would have spent generating the rest.
The Architecture
The streaming architecture adds one layer to the per-article processing: a token channel sits between the streamer goroutine and the consumer.
Worker (article)
│
streamArticle()
│
[streamer goroutine] ──→ tokenCh (buffered) ──→ [consumer]
│
accumulates tokens
│
StreamingResult
The streamer goroutine produces tokens into tokenCh as they arrive. The consumer ranges over tokenCh and accumulates the full text. When the streamer closes tokenCh after the last token, the range loop exits and the result is returned.
Both sides run concurrently — the streamer does not wait for the consumer to process each token before generating the next one. The channel buffer absorbs small speed differences between them.
The Implementation
func (p *StreamingPool) streamArticle(ctx context.Context, article model.Article) StreamingResult {
start := time.Now()
// Buffer of 10: streamer can be up to 10 tokens ahead of consumer
tokenCh := make(chan Token, 10)
// Streamer goroutine: produces tokens as they arrive
go func() {
defer close(tokenCh) // signals consumer to stop ranging
tokens := p.generateTokens(article.ID)
for i, tok := range tokens {
select {
case <-ctx.Done():
tokenCh <- Token{ArticleID: article.ID, Err: ctx.Err(), Done: true}
return
default:
}
// Simulate inter-token latency (10–60ms per token)
p.rngMu.Lock()
delay := time.Duration(p.rng.Intn(50)+10) * time.Millisecond
p.rngMu.Unlock()
time.Sleep(delay)
tokenCh <- Token{
ArticleID: article.ID,
Text: tok,
Done: i == len(tokens)-1,
}
}
}()
// Consumer: accumulates tokens until channel closes
result := StreamingResult{ArticleID: article.ID}
for tok := range tokenCh {
if tok.Err != nil {
result.Err = tok.Err
break
}
result.FullText += tok.Text + " "
result.TokenCount++
}
result.Duration = time.Since(start)
return result
}
The buffer size of 10 is deliberate. Without a buffer, every token send would block until the consumer receives it — the streamer and consumer would run in perfect lockstep, eliminating the concurrency benefit. With a buffer of 10, the streamer can generate up to 10 tokens without the consumer having processed any of them. In practice this means the streamer runs slightly ahead of the consumer, which is the correct behaviour for a streaming pipeline: generate as fast as possible, consume as fast as possible, the channel absorbs any momentary difference.
The Token struct carries a Done flag for the last token and an Err field for failures. Both allow the consumer to distinguish between a normal stream completion and an interrupted one:
type Token struct {
ArticleID int
Text string
Done bool
Err error
}
What the Output Looks Like
Two articles streaming simultaneously, two workers:
[article 2] token: "Breaking"
[article 2] token: "news"
[article 1] token: "Breaking"
[article 2] token: "article"
[article 2] token: "#2"
[article 1] token: "news"
[article 2] token: "discusses"
[article 1] token: "article"
[article 2] token: "the"
[article 1] token: "#1"
[article 2] token: "latest"
[article 2] token: "developments"
[article 1] token: "discusses"
...
[article 2] streaming complete: 13 tokens in 436ms
[article 1] streaming complete: 13 tokens in 609ms
Article 2: 13 tokens in 436ms
Breaking news article #2 discusses the latest developments in AI and Go concurrency.
Article 1: 13 tokens in 609ms
Breaking news article #1 discusses the latest developments in AI and Go concurrency.
Total: 609ms
Tokens from article 1 and article 2 interleave freely. Article 2's streamer generates a few tokens, then article 1's streamer gets scheduled and produces some of its own. Both consumers accumulate their respective tokens independently. The articles finish in 436ms and 609ms — the total pipeline time is bounded by the slower article, not their sum.
The full text assembled by each consumer is correct despite the interleaving. Each token carries ArticleID so there is no ambiguity about which article a token belongs to, even when tokens from multiple articles appear in the same output window.
Context Cancellation in the Streamer
The streamer checks the context at the start of each token generation:
select {
case <-ctx.Done():
tokenCh <- Token{ArticleID: article.ID, Err: ctx.Err(), Done: true}
return
default:
}
If the context is cancelled — pipeline shutdown, per-article timeout — the streamer sends one final error token and exits. The consumer receives it, records the error, and stops ranging. The channel is closed by defer close(tokenCh) in the streamer, so the consumer's range loop exits cleanly.
This is the correct behaviour: a cancelled stream does not leave goroutines blocked. The streamer exits, the consumer exits, the article result carries the cancellation error. No goroutine leak.
Without the context check, a streaming goroutine would continue generating tokens after the pipeline shuts down — potentially for several seconds — burning CPU and holding references to the article data in memory. With the check per-token, the goroutine exits within one token generation cycle of the cancellation signal.
The Buffer Size Trade-Off
The token channel buffer size controls the decoupling between streamer and consumer. Choosing it involves a real trade-off.
A small buffer (1–2) keeps the streamer close to the consumer. If the consumer is slow — writing tokens to a database, rendering them in a UI — the streamer blocks frequently. Memory usage stays low because tokens are processed almost immediately after generation. The cost is that the streamer loses its head start.
A larger buffer (10–50) lets the streamer run well ahead of the consumer. This smooths out processing jitter — if the consumer has a momentary delay, the streamer fills the buffer and keeps generating. Memory usage is proportionally higher because more tokens sit in the channel unprocessed.
For in-memory accumulation — the pattern in this part — a buffer of 10 is more than enough. The consumer is just appending a string, which takes nanoseconds. For a consumer that writes to a network connection or a database, a larger buffer absorbs network jitter and keeps the streamer running smoothly during momentary latency spikes.
Connecting to Production APIs
The generateTokens function in the simulator returns a slice of words and sleeps 10–60ms between them. Real LLM API streaming replaces this with an HTTP SSE response:
// Real streaming (OpenAI SDK pattern)
stream, err := client.Chat.Completions.NewStreaming(ctx, params)
for stream.Next() {
chunk := stream.Current()
for _, choice := range chunk.Choices {
token := choice.Delta.Content
tokenCh <- Token{Text: token}
}
}
if err := stream.Err(); err != nil {
tokenCh <- Token{Err: err, Done: true}
return
}
The channel-based pattern is identical. The for stream.Next() loop replaces for i, tok := range tokens, and the SDK's streaming iterator replaces the simulated sleep. The consumer code — ranging over tokenCh and accumulating text — is completely unchanged.
This is the value of building the architecture around channels rather than SDK-specific types: swapping from a simulator to a real API requires changing only the producer goroutine. The consumer, the worker pool, the context propagation, and the result collection all remain the same.
What's Next
Token streaming completes the output side of the pipeline. Articles go in, tokens come out incrementally, results are assembled correctly even when multiple articles stream simultaneously.
Part 18 takes a diagnostic perspective on everything built so far. Goroutine leaks — goroutines that start and never stop — are one of the most common production problems in concurrent Go programs, and streaming pipelines create them easily if the context is not respected. Part 18 shows how to detect leaks using runtime.NumGoroutine(), how to reproduce them intentionally, and how the fix pattern used throughout this series prevents them.
See you in Part 18.
This is Part 17 of the series "Production-Grade Concurrent AI Systems in Go." Read Part 16 — Backpressure or continue to Part 18 — Goroutine Leaks: Finding and Fixing Stuck Goroutines.