Part 9 — Cancellation and Graceful Shutdown: Stopping Safely

Part of the series: Production-Grade Concurrent AI Systems in Go

→ Full code for this post: github.com/madmmas/go-concurrent-ai-systems/tree/part-09 → Diff from Part 8: compare/part-08...part-09 → Run it: go run ./cmd/news-processor inside arc-1-foundations/part-09-cancellation

Part 8 gave every article a per-article deadline. If a single LLM call hangs, the article fails fast and the worker moves on. The pipeline never blocks indefinitely on one slow provider.

But here's what Part 8 can't do: stop.

Send a SIGTERM — from Kubernetes rolling out a new deployment, from a developer hitting Ctrl+C, from an orchestrator scaling down — and ProcessAll ignores it. It keeps running until the last article completes or times out. For a large batch, that could be minutes. A Kubernetes pod that doesn't stop in time gets SIGKILLed, and work in flight disappears with no record.

This part fixes that. We wire the pipeline to OS signals, propagate cancellation through the worker pool, and build a ShutdownReport that accounts for every article — whether it succeeded, failed, was mid-flight when shutdown arrived, or was still waiting in the queue.

What Graceful Shutdown Means

"Graceful" has a precise meaning here. It does not mean "stop immediately" — that's what SIGKILL does, and it's what we're trying to avoid. It means:

Workers finish the task they are currently executing, up to a configured timeout
Articles that haven't started yet are drained from the queue and reported as not processed
The process exits only after every goroutine has had a chance to clean up

The result is a clear inventory: you know exactly what made it through and what didn't. That's the difference between a system you can safely operate and one that drops work silently under pressure.

Signal Handling in Go

Go's signal.NotifyContext is the cleanest way to wire OS signals to context cancellation:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()

ctx is a standard context.Context. When the process receives SIGTERM or SIGINT, Go cancels ctx — exactly as if you'd called cancel() yourself. Every piece of code that holds a reference to ctx or a child of it sees ctx.Done() close and can exit cleanly.

The elegance here is that there's nothing pipeline-specific about this. The signal handling is one line at main(). The pipeline just receives a context and respects it. The two concerns are completely separate.

Two Levels of Cancellation Revisited

Part 8 introduced two context levels: per-article and pipeline-wide. Part 9 makes the pipeline-wide level real and useful.

// Pipeline-wide context — cancelled by OS signal or -cancel-after flag
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()

// Per-article context — derived from the pipeline context
articleCtx, cancel := context.WithTimeout(ctx, p.ArticleTimeout)
defer cancel()

The hierarchy matters. A per-article context is a child of the pipeline context. When the pipeline context is cancelled:

All per-article child contexts are cancelled immediately, regardless of their own timeout
Workers see ctx.Done() close on the next iteration of their job loop
Articles in the queue are drained and marked as not started

The cancellation flows top-down through the tree automatically. You don't have to explicitly tell each worker to stop — they all inherit the signal from the root context.

The ShutdownReport

Part 8's results slice told you what happened to each article individually. Part 9 adds a ShutdownReport that gives a summary of the whole pipeline's state at the moment it stopped:

type ShutdownReport struct {
    Succeeded int  // completed successfully
    Failed    int  // timed out or hard error
    Cancelled int  // context cancelled while in-flight
    Queued    int  // never dequeued — pipeline stopped first
    Duration  time.Duration
}

The invariant that must always hold:

Succeeded + Failed + Cancelled + Queued == total input articles

This is not just a nice-to-have. If this equation breaks, work has gone missing silently — you don't know what was processed and what wasn't, and you can't safely resume or retry. Every article must produce exactly one outcome, even under abrupt cancellation. Our tests enforce this invariant explicitly across all scenarios.

Draining Queued Articles

The subtle part of graceful shutdown is what happens to articles that are still sitting in the jobs channel when cancellation arrives. Workers are checking the pipeline context at the top of their loop:

func (p *WorkerPool) worker(ctx context.Context, id int,
    jobs <-chan model.Article, results chan<- model.AIResult) {

    for article := range jobs {
        // Has the pipeline been cancelled?
        select {
        case <-ctx.Done():
            fmt.Printf("[worker %d] pipeline cancelled — skipping article %d\n",
                id, article.ID)
            results <- model.AIResult{ArticleID: article.ID, Err: ctx.Err()}
            continue  // keep draining — don't break
        default:
        }

        // Context still live — process normally
        articleCtx, cancel := context.WithTimeout(ctx, p.ArticleTimeout)
        result := p.processArticle(articleCtx, article)
        cancel()
        results <- result
    }
}

The continue instead of break is deliberate. When we detect cancellation, we don't exit the loop — we skip the LLM calls but keep pulling articles from jobs and emitting skipped results. This is what empties the queue and ensures every article eventually produces a result. Without it, articles left in jobs when the pipeline stops would never get a result entry, breaking the invariant.

What You See

Normal run — everything completes:

Pipeline: 6 articles, 3 workers, 4s per-article timeout

[worker 1] starting article 1
[worker 2] starting article 2
[worker 3] starting article 3
[worker 3] article 3: done
[worker 3] starting article 4
[worker 1] article 1: done
[worker 1] starting article 5
...
[worker 2] exiting
[worker 1] exiting
[worker 3] exiting

═════════════════════════════════════════════════════════
Succeeded : 6
Failed    : 0 (timeout/error)
Cancelled : 0 (in-flight when shutdown arrived)
Queued    : 0 (never started)
Duration  : 5.3s
═════════════════════════════════════════════════════════
Total accounted for: 6 / 6

Simulated cancellation after 1 second, with 10 articles in-flight:

⏱  Will cancel pipeline after 1s

[worker 1] starting article 1
[worker 2] starting article 2
[worker 3] starting article 3
  [1] Summarization started (1.2s)
  [2] Summarization started (900ms)
  [3] Summarization started (1.4s)

[worker 3] pipeline cancelled — skipping article 4
[worker 3] pipeline cancelled — skipping article 5
...
[worker 3] pipeline cancelled — skipping article 10
[worker 3] exiting
[worker 1] article 1: context deadline exceeded
[worker 1] exiting
[worker 2] article 2: context deadline exceeded
[worker 2] exiting

═════════════════════════════════════════════════════════
Succeeded : 0
Failed    : 0 (timeout/error)
Cancelled : 3 (in-flight when shutdown arrived)
Queued    : 7 (never started)
Duration  : 1.000s
═════════════════════════════════════════════════════════
Total accounted for: 10 / 10

Three articles were in-flight when the 1-second cancellation fired. They were mid-way through their first LLM task, got cancelled, and reported as Cancelled. Seven articles were still in the queue — they got drained by workers 3, reported as Queued, and never had any LLM calls made. The pipeline stopped in exactly 1 second. Ten in, ten accounted for.

The Real-World Scenario

Replace the -cancel-after flag with OS signal handling and this is exactly what happens when Kubernetes sends SIGTERM before rolling out a new deployment:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()

_, report := pool.ProcessAll(ctx, articles)

// After SIGTERM:
// Succeeded : 847
// Cancelled :   3   (these 3 need to be retried in the next deployment)
// Queued    : 150   (these 150 were never started — resume from here)

You know exactly where to resume. Article IDs 848 through 1000 were either queued or cancelled — re-queue them. Articles 1 through 847 are done. No guessing, no re-processing things that already completed.

Arc 1 Complete

This is the final part of Arc 1. Here's what the whole arc built, and what each part added:

Part	Concept	What you couldn't do before it
1	Sequential baseline	Nothing — this is the starting point
2	Goroutines + WaitGroup	Process more than one article at a time
3	Mutex	Share results across goroutines safely
4	Deadlocks	Recognise blocking failure modes before using channels
5	Channels	Remove shared memory from the design entirely
6	Buffered channels	Decouple producers and consumers under burst load
7	Worker pool	Control how many goroutines run at once
8	Context + timeouts	Stop waiting for a hung provider
9	Cancellation + shutdown	Stop the whole pipeline cleanly from the outside

The pipeline at the end of Part 9 is meaningfully different from Part 1. It's concurrent, bounded, race-free, timeout-protected, and gracefully stoppable. You could run this in production — within the limits of a single process.

Those limits are what Arc 2 addresses. Single-process pipelines have no way to spread work across machines, no way to recover from a process crash mid-batch, and no way to handle back pressure from a downstream system that's slower than the producers feeding it. Arc 2 starts with those problems: fan-out/fan-in pipelines, concurrent scraping, retries, circuit breakers, and streaming — the patterns that take a solid single-process design and make it production-grade in the distributed sense.

This is Part 9 of the series "Production-Grade Concurrent AI Systems in Go," and the final part of Arc 1 — Concurrency Foundations. Read Part 8 — Context and Timeouts or jump ahead to Arc 2 — Production Concurrent AI Systems.