← Building Concurrent Ai Pipelines In Go

Phase

Production Concurrent AI

Fan-out/fan-in, multi-stage pipelines, errgroup, retries, rate limiting, circuit breakers, backpressure, streaming, and observability for production AI systems.

11 published · 0 upcoming

  1. Part 10Published

    Part 10 - Fan-Out and Fan-In: Concurrent Tasks Per Article

    Arc 1 ran three AI tasks per article one after another. Part 10 runs them all at once — fan-out launches them concurrently, fan-in collects all three results, and per-article time drops from the sum to the slowest.

  2. Part 11Published

    Part 11 - Pipeline Stages: A Worker Pool Per Bottleneck

    Part 10 ran tasks concurrently inside one worker. Part 11 runs stages concurrently across the whole pipeline — scrape, clean, embed, and summarise each get their own worker pool, tuned to their specific bottleneck.

  3. Part 12Published

    Part 12 - errgroup: When One Task Fails, Cancel the Rest

    Part 10 fanned out tasks and collected results. Part 12 adds the missing piece: when one task fails, cancel the rest immediately. errgroup does this in a handful of lines — and understanding how it works is more useful than treating it as a black box.

  4. Part 13Published

    Part 13 - Retries and Exponential Backoff: Handling Failure Gracefully

    Part 12 showed how to cancel sibling tasks on first failure. Part 13 shows what to do next: retry with exponential backoff and jitter, distinguish retryable from permanent errors, and route exhausted articles to a dead letter queue.

  5. Part 14Published

    Part 14 - Rate Limiting: Controlling Outbound Call Volume

    Part 13 added retries for when the provider returns 429. Part 14 prevents the 429 from happening in the first place — a token bucket rate limiter that caps outgoing LLM calls per second, regardless of how many workers are running.

  6. Part 15Published

    Part 15 - Circuit Breaker: Fail Fast When the Provider Is Down

    Rate limiting smooths steady-state traffic. Retries recover from transient failures. Neither helps when a provider is down for minutes. The circuit breaker detects sustained failure and stops calling entirely — failing fast, protecting the pipeline, and probing for recovery.

  7. Part 16Published

    Part 16 - Backpressure: Letting the Queue Push Back

    When a producer generates work faster than consumers can process it, the queue grows without bound. A bounded channel is the fix — when it fills, the producer blocks automatically. No explicit signal, no coordination: the channel capacity is the backpressure contract.

  8. Part 17Published

    Part 17 - Token Streaming: Processing LLM Output as It Arrives

    Every previous part treated an LLM response as a single value that arrives when the call completes. Real LLM APIs stream tokens incrementally — you receive the first word in hundreds of milliseconds and the rest follow one by one. A channel between producer and consumer is the natural Go model for this.

  9. Part 18Published

    Part 18 - Goroutine Leaks: Finding and Fixing Stuck Goroutines

    A goroutine that starts and never exits is a leak. Unlike memory leaks, goroutine leaks are invisible — no crash, no error, just a process that slowly accumulates blocked goroutines until memory and scheduler overhead degrade everything. This part shows how to detect them, reproduce them intentionally, and fix them.

  10. Part 19Published

    Part 19 - Concurrent RAG Pipeline: Putting It All Together

    The flagship of Arc 2: a full concurrent RAG pipeline — chunk, embed, generate — where every stage from Part 11, every resilience pattern from Parts 13-15, and every flow control mechanism from Parts 16-18 comes together in one coherent system.

  11. Part 20Published

    Part 20 - Observability: Seeing What the Pipeline Is Actually Doing

    A pipeline that cannot be observed cannot be operated. Part 20 adds throughput, latency percentiles, and error breakdown by type — the three metrics that tell you whether the pipeline is healthy, where it is slow, and what is failing. This is the foundation for Prometheus and OpenTelemetry integration.