Part 13 - Retries and Exponential Backoff: Handling Failure Gracefully

Part of the series: Production-Grade Concurrent AI Systems in Go

→ Full code for this post: github.com/madmmas/go-concurrent-ai-systems/tree/part-13 → Diff from Part 12: compare/part-12...part-13 → Run it: go run ./cmd/news-processor -articles=5 -workers=2 -rate-limit=0.3 inside arc-2-production/part-13-retries

Parts 10 through 12 dealt with failure as something to detect and propagate. When a task returned an error, the article was marked failed and the pipeline moved on. That is correct behaviour for the pipeline itself — nothing leaks, nothing hangs, every article produces exactly one result.

But in production, most LLM failures are transient. A 429 rate limit is not the provider telling you the article is broken. It is the provider telling you to slow down and try again in a moment. A 503 server error is not permanent either — it is a signal that the provider is temporarily overloaded. Discarding the article on the first failure would mean losing work that would succeed on a second attempt a few hundred milliseconds later.

Retries are the standard answer. But naive retries — immediate, unlimited, without any backoff — can make things significantly worse. If 50 workers all hit a rate limit simultaneously and all retry immediately, they generate another burst of requests that triggers another rate limit, which generates another retry burst. This is the thundering herd problem, and it turns a provider that is briefly struggling into one that stays overwhelmed.

Part 13 adds retries the right way: exponential backoff, jitter, a hard attempt limit, and a distinction between errors that are worth retrying and errors that are not.

Two Categories of Error

Not every error warrants a retry. The simulator defines two kinds:

var (
    ErrRateLimit   = errors.New("llm: rate limit exceeded (429)")
    ErrServerError = errors.New("llm: server error (503)")
)

func IsRetryable(err error) bool {
    return errors.Is(err, ErrRateLimit) || errors.Is(err, ErrServerError)
}

ErrRateLimit (HTTP 429) and ErrServerError (HTTP 503) are retryable — they reflect transient provider state, not a problem with the article itself. Retrying them makes sense.

Context cancellation and deadline exceeded are not retryable. If the pipeline context was cancelled, retrying would violate the caller's shutdown signal. If the per-article timeout fired, retrying with the same timeout against a provider that already timed out once is unlikely to succeed.

This distinction matters more than the retry mechanism itself. A pipeline that retries everything — including context cancellations — will keep workers running after graceful shutdown, undermining everything Part 9 built. A pipeline that retries nothing wastes the resilience that makes LLM-backed systems usable in production.

Exponential Backoff

The backoffDelay function computes how long to wait before the next attempt:

func (p *RetryPool) backoffDelay(attempt int) time.Duration {
    delay := p.Retry.BaseDelay * (1 << uint(attempt-1))
    if delay > p.Retry.MaxDelay {
        delay = p.Retry.MaxDelay
    }
    p.rngMu.Lock()
    jitter := time.Duration(p.rng.Float64() * p.Retry.JitterFrac * float64(delay))
    p.rngMu.Unlock()
    return delay + jitter
}

With BaseDelay=100ms, MaxDelay=2s, and JitterFrac=0.2:

The delay doubles each time until it hits MaxDelay. The cap matters — without it, a pipeline that keeps failing would eventually wait minutes between attempts, holding a worker hostage for an article that has probably become stale by then.

The jitter is the piece that prevents thundering herd. Without it, workers that fail at the same moment all wait the same 100ms and then all retry at the same moment, creating another burst. With jitter, retries are spread across a window — 100ms to 120ms, not all at exactly 100ms. The spread is small enough that individual articles do not wait much longer, but large enough that concurrent retries no longer land as a synchronised wave.

The Retry Loop

processWithRetry wraps each article attempt and handles the retry logic:

func (p *RetryPool) processWithRetry(ctx context.Context, article model.Article) ProcessResult {
    var lastErr error
    for attempt := 1; attempt <= p.Retry.MaxAttempts; attempt++ {
        if ctx.Err() != nil {
            return ProcessResult{Result: model.AIResult{ArticleID: article.ID, Err: ctx.Err()}}
        }

        articleCtx, cancel := context.WithTimeout(ctx, p.Timeout)
        result := p.processArticle(articleCtx, article)
        cancel()

        if result.Err == nil {
            result.Retries = attempt - 1
            return ProcessResult{Result: result}
        }

        lastErr = result.Err

        if !simulator.IsRetryable(result.Err) {
            return ProcessResult{Result: result, DeadLetter: true}
        }

        if attempt < p.Retry.MaxAttempts {
            delay := p.backoffDelay(attempt)
            fmt.Printf("[article %d] attempt %d failed (%v), retrying in %v\n",
                article.ID, attempt, result.Err, delay.Round(time.Millisecond))

            select {
            case <-time.After(delay):
            case <-ctx.Done():
                return ProcessResult{Result: model.AIResult{ArticleID: article.ID, Err: ctx.Err()}}
            }
        }
    }

    fmt.Printf("[article %d] exhausted %d attempts — dead letter: %v\n",
        article.ID, p.Retry.MaxAttempts, lastErr)
    return ProcessResult{
        Result:     model.AIResult{ArticleID: article.ID, Err: lastErr, Retries: p.Retry.MaxAttempts - 1},
        DeadLetter: true,
    }
}

Three exit paths:

Success — result.Err == nil. The result is returned with Retries set to how many attempts were needed beyond the first. A caller can use this to measure retry pressure: articles that consistently need two or three attempts signal that the provider is under stress before rate limits start firing.

Non-retryable failure — IsRetryable returns false. The article is marked DeadLetter: true and returned immediately. No further attempts, no waiting.

Exhausted — the loop reaches MaxAttempts without a success. The article is marked DeadLetter: true with the last error. Something is persistently wrong — possibly a malformed article, a provider outage lasting longer than the retry window, or a rate limit that is not recovering.

The select during the backoff wait is important:

select {
case <-time.After(delay):
case <-ctx.Done():
    return ProcessResult{...Err: ctx.Err()}
}

Without the ctx.Done() case, a worker waiting 400ms on its third retry would not respond to pipeline shutdown until the timer fires. With it, cancellation takes effect immediately — the worker returns with a cancelled error and the worker goroutine is free to exit.

What the Output Shows

With 30% rate limit probability — realistic for a provider under moderate load:

  [2] Summarisation → 429 rate limited
[article 2] attempt 1 failed (llm: rate limit exceeded (429)), retrying in 117ms

  [1] Sentiment Analysis → 429 rate limited
[article 1] attempt 1 failed (llm: rate limit exceeded (429)), retrying in 103ms

  [2] Sentiment Analysis → 429 rate limited
[article 2] attempt 2 failed (llm: rate limit exceeded (429)), retrying in 206ms

...

Retry pipeline: 5 articles, rate-limit=30%
Succeeded: 5 | Failed: 0 | Dead letter: 0
Total retries: 3 | Duration: 3.042s

All five articles eventually succeeded despite repeated rate limits. The total retries across all five articles was 3 — on average, less than one retry per article. The pipeline absorbed the transient failures without losing any work.

With 100% rate limit — provider is broken:

[article 1] attempt 1 failed (llm: rate limit exceeded (429)), retrying in 105ms
[article 2] attempt 1 failed (llm: rate limit exceeded (429)), retrying in 110ms
[article 1] attempt 2 failed (llm: rate limit exceeded (429)), retrying in 236ms
[article 2] attempt 2 failed (llm: rate limit exceeded (429)), retrying in 236ms
[article 1] exhausted 3 attempts — dead letter: llm: rate limit exceeded (429)
[article 2] exhausted 3 attempts — dead letter: llm: rate limit exceeded (429)

Retry pipeline: 3 articles, rate-limit=100%
Succeeded: 0 | Failed: 3 | Dead letter: 3
Total retries: 6 | Duration: 693ms

Three articles, three attempts each, six total retries, all dead-lettered. The pipeline gave up cleanly in 693ms rather than hanging indefinitely. The dead letter count tells the operator exactly how many articles need attention — they can be re-queued manually or processed from a DLQ once the provider recovers.

The Dead Letter Queue

ProcessResult.DeadLetter is a flag, not a queue. In this implementation the caller decides what to do with flagged articles — log them, write them to a database, emit them to a Kafka DLQ topic, or simply count them and alert on the rate.

results, dur := pool.ProcessAll(ctx, articles)

ok, dead := 0, 0
for _, r := range results {
    if r.DeadLetter {
        dead++
        // re-queue, log, or alert
    } else if r.Result.Err == nil {
        ok++
    }
}

The separation is deliberate. A pipeline that handles DLQ processing inline becomes harder to reason about — the retry logic, the DLQ routing, and the normal processing path are all entangled. Keeping the retry loop focused on retry decisions and leaving DLQ handling to the caller keeps each piece of code doing one thing.

In production, a dead letter rate above some threshold — say, 5% over a five-minute window — is a signal that the circuit breaker in Part 15 should have fired already. The two patterns complement each other: retries handle transient failures within a call, circuit breakers handle sustained provider degradation at the pool level.

Tuning RetryConfig

DefaultRetryConfig is a starting point, not a prescription:

var DefaultRetryConfig = RetryConfig{
    MaxAttempts: 3,
    BaseDelay:   100 * time.Millisecond,
    MaxDelay:    2 * time.Second,
    JitterFrac:  0.2,
}

MaxAttempts — three is conservative for rate limits; some teams use five or more. The right number is determined by how long your provider's rate limit windows last and how much latency increase your users can absorb. Each retry adds at minimum BaseDelay to the article's processing time.

BaseDelay — 100ms works for providers with second-scale rate limit windows. If your provider's documentation says to back off for at least 500ms, your BaseDelay should be at least 500ms.

MaxDelay — caps the wait so no article is held for unreasonably long. Two seconds means no worker is idle for more than two seconds waiting to retry. Adjust upward if your retry budget allows longer waits, downward if article freshness is more important than retry success.

JitterFrac — 0.2 adds up to 20% random spread on each delay. Increasing it spreads retries more widely, decreasing it tightens them. 0.0 means no jitter — only appropriate if you have a single worker and the thundering herd problem does not apply.

What's Next

Retries handle individual call failures. But what happens when a provider is not just occasionally slow — it is consistently returning 503 for minutes at a time? Retrying against a broken provider wastes time, adds latency to every article in the pipeline, and generates load that the provider cannot handle, making its recovery slower.

Part 14 adds a rate limiter — a token bucket that proactively limits how many calls per second we send, preventing 429s before they happen. Part 15 adds a circuit breaker — a mechanism that detects sustained failure and stops calling the provider entirely until it recovers, failing fast instead of retrying into a wall.

Together, retries, rate limiting, and circuit breakers form the resilience layer that every production AI pipeline needs.

See you in Part 14.

This is Part 13 of the series "Production-Grade Concurrent AI Systems in Go." Read Part 12 — errgroup and Structured Concurrency or continue to Part 14 — Rate Limiting: Controlling Outbound Call Volume.