Part of the series: Production-Grade Concurrent AI Systems in Go
→ Full code for this post: github.com/madmmas/go-concurrent-ai-systems/tree/part-14 → Diff from Part 13: compare/part-13...part-14 → Run it: go run ./cmd/news-processor -articles=5 -workers=5 -rate=3 inside arc-2-production/part-14-rate-limiting
Part 13 handled 429 rate limit errors reactively: the provider pushed back, the worker waited, and it tried again. That works — but it is the wrong sequence. By the time you receive a 429, you have already wasted a round trip. The provider spent resources rejecting your request. Your worker spent time waiting for a response that carried no useful data. And if you have 20 workers all hitting the rate limit simultaneously, you get 20 retries all starting at roughly the same moment, which can trigger another round of 429s.
The better approach is to enforce the rate limit on your side before the call is ever made. A rate limiter sitting in front of every LLM call caps outbound volume regardless of how many workers are running. Workers that would exceed the limit simply wait their turn. The provider sees a smooth, controlled request rate. You stop receiving 429s because you stop sending more than the provider allows.
Part 14 implements this with a token bucket — the algorithm behind nearly every production rate limiter you have ever encountered.
How a Token Bucket Works
A token bucket has three properties: a capacity (the burst allowance), a refill rate (tokens per second), and a current token count. The rules are simple:
- Every second, the bucket gains
ratenew tokens, up to its maximum capacity. - To make a call, a worker must acquire one token first. If a token is available, it is consumed immediately and the call proceeds.
- If no token is available, the worker waits until the next token arrives.
The burst capacity is what makes it a bucket rather than a pure rate limiter. If the bucket holds 5 tokens and no calls have been made for 2 seconds, the bucket refills to 5. The next 5 calls can proceed immediately without waiting. After that, calls throttle to one per 1/rate seconds.
This models real provider behaviour accurately. OpenAI, Anthropic, and most LLM providers allow short bursts above their stated rate limit before throttling. The token bucket captures that nuance — a cold start can fire several requests at once, sustained load is smoothed to the stated rate.
The Implementation
type TokenBucket struct {
mu sync.Mutex
tokens float64
maxTokens float64
rate float64 // tokens per second
lastFill time.Time
}
func NewTokenBucket(ratePerSec float64, burst int) *TokenBucket {
return &TokenBucket{
tokens: float64(burst),
maxTokens: float64(burst),
rate: ratePerSec,
lastFill: time.Now(),
}
}
The bucket starts full — tokens == burst. That is the cold-start burst: the first burst calls proceed without any wait.
Acquire is where workers block:
func (tb *TokenBucket) Acquire(ctx context.Context) error {
for {
if err := ctx.Err(); err != nil {
return err
}
tb.mu.Lock()
now := time.Now()
elapsed := now.Sub(tb.lastFill).Seconds()
tb.tokens = min64(tb.maxTokens, tb.tokens+elapsed*tb.rate)
tb.lastFill = now
if tb.tokens >= 1.0 {
tb.tokens--
tb.mu.Unlock()
return nil
}
waitSec := (1.0 - tb.tokens) / tb.rate
tb.mu.Unlock()
wait := time.Duration(waitSec * float64(time.Second))
select {
case <-time.After(wait):
case <-ctx.Done():
return ctx.Err()
}
}
}
Every call to Acquire first refills the bucket based on elapsed time since the last fill. This lazy refill means you do not need a background goroutine ticking tokens in — refills happen on demand, only when a worker actually tries to acquire. The math is: tokens earned = elapsed seconds × rate, capped at maxTokens.
If a token is available after refilling, it is consumed and Acquire returns. If not, the worker calculates how long until the next token arrives — (1.0 - tokens) / rate seconds — and sleeps that long. When it wakes, it tries again.
The select during the wait is the same pattern from Part 13's retry backoff: context cancellation interrupts the sleep immediately. A worker waiting for a rate limit token responds to pipeline shutdown without hanging until its next token would have arrived.
Where the Token Is Acquired
The token acquisition happens before the LLM call, not after. This is the critical placement:
func (p *RateLimitedPool) processArticle(ctx context.Context, article model.Article) model.AIResult {
result := model.AIResult{ArticleID: article.ID}
// Acquire first, then call
if err := p.bucket.Acquire(ctx); err != nil {
result.Err = err
return result
}
if err := p.llm.Call(ctx, "Summarisation", article.ID); err != nil {
result.Err = err
return result
}
result.Summary = "AI-generated summary"
// Acquire again for the second call
if err := p.bucket.Acquire(ctx); err != nil {
result.Err = err
return result
}
if err := p.llm.Call(ctx, "Sentiment Analysis", article.ID); err != nil {
result.Err = err
return result
}
result.Sentiment = "Positive"
result.Keywords = []string{"AI", "Go", "RateLimit"}
return result
}
Each LLM call has its own Acquire. Two calls per article means two tokens consumed per article. With rate=3 and burst=1, the pipeline can sustain three LLM calls per second — so it can fully process one and a half articles per second at this rate. That is exactly what the timing shows.
Acquiring per-call rather than per-article is the right granularity for LLM pipelines. A provider's rate limit is expressed in requests per minute or tokens per minute — not articles per minute. If you acquire once per article but make three calls, you are sending three times the requests the token permits.
The Timing Proof
The test that proves the limiter actually enforces its rate:
// 4 articles × 2 calls each = 8 calls at 2 calls/second → should take at least 3s
pool := pipeline.New(
simulator.New(fastCallConfig),
5, 5*time.Second, 2.0, 1, // rate=2/s, burst=1
)
start := time.Now()
pool.ProcessAll(context.Background(), pipeline.GenerateArticles(4))
elapsed := time.Since(start)
// Assert at least 3s elapsed
if elapsed < 3*time.Second {
t.Errorf("rate limit not enforced: finished in %v", elapsed)
}
Running this test:
--- PASS: TestRateLimit_EnforcesLimit (3.51s)
Eight calls at two per second takes four seconds in theory. The test saw 3.51s — burst=1 lets the first call through immediately, so the clock effectively starts on the second call. The math holds. The limiter is not advisory — it physically prevents calls from proceeding faster than the configured rate.
Compare the effect at different rates on five articles (two calls each = ten total calls):
rate=1000/s burst=3 → 1.369s (effectively unlimited, calls take ~270ms each)
rate=3/s burst=1 → 3.734s (10 calls at 3/s = ~3.3s minimum)
rate=2/s burst=1 → 4.430s (10 calls at 2/s = 5s minimum, burst saves ~0.5s)
The limiter enforces a hard ceiling. Adding more workers does not help — five workers all blocked on Acquire still only send three calls per second. The rate is controlled by the bucket, not by worker count.
Rate and Burst in Practice
Real provider documentation usually looks like this:
Rate limits: 60 requests per minute, burst up to 20
That translates directly to token bucket parameters:
// 60 requests/min = 1 request/second sustained
// burst = 20 (allow up to 20 simultaneous before throttling)
bucket := NewTokenBucket(1.0, 20)
A few practical notes on choosing values.
Set rate conservatively, not maximally. If the provider allows 60 RPM, configure 50 RPM. The 10-request headroom absorbs burst from other clients sharing your API key, leaves room for retries (Part 13) without immediately exhausting the limit, and keeps you safe if your pipeline momentarily sends a cluster of requests.
Burst should match your startup pattern. If you start the pipeline and want the first N articles to process immediately without queuing, set burst to N. If you would rather the pipeline start slowly and ramp up, set burst to 1.
Per-provider, not per-pool. If you are calling multiple providers from the same pipeline, each provider needs its own bucket. Sharing a bucket across providers would throttle OpenAI calls because Anthropic calls are consuming tokens.
Rate Limiting vs Retries — Both Are Needed
It is tempting to treat rate limiting and retries as alternatives — either you prevent 429s or you recover from them. In practice you need both, for different reasons.
Rate limiting keeps your steady-state traffic under the limit. It does not protect against traffic spikes caused by a burst of new articles, a delayed batch processing queue draining all at once, or a bug that fires requests in a tight loop. When those spikes happen, 429s will still arrive.
Retries (Part 13) handle the 429s that slip through. The combination means your pipeline is both proactive — it sends less than the limit during normal operation — and reactive — it recovers when the limit is still hit.
The relationship is also temporal. Rate limiting works at the call level, in the microsecond-to-millisecond range. Retries work at the article level, over hundreds of milliseconds to seconds. Circuit breakers (Part 15) work at the provider level, over seconds to minutes. Each layer handles a different failure mode at a different timescale.
What's Next
Rate limiting prevents 429s under normal operating conditions. But what happens when the provider goes down entirely — returning 503 for minutes at a time? Retries keep trying. The rate limiter keeps allowing calls through. Workers queue up at Acquire, retry, fail, retry again, building up pressure that cannot be released because the provider is not recovering.
The circuit breaker is the answer: after enough consecutive failures, stop calling the provider entirely. Return errors immediately — without a network round trip, without waiting for a timeout, without consuming retry budget. When the provider recovers, probe it carefully with a single call before reopening.
See you in Part 15.
This is Part 14 of the series "Production-Grade Concurrent AI Systems in Go." Read Part 13 — Retries and Exponential Backoff or continue to Part 15 — Circuit Breaker: Fail Fast When the Provider Is Down.