Part of the series: From Zero to Kubernetes Operators with Kubebuilder (KubeAgent Journey)
→ Full code for this post: github.com/madmmas/kubeoperator-journey/tree/blog-02 → Run it: go run ./cmd/control-loop inside the repo root
In Blog 1 we saw what life looks like without operators — runbooks, drift, 2am pages. In this post we build the mechanism that makes operators possible: the control loop.
We're not touching Kubebuilder yet. That's Blog 3. Here we build a working Kubernetes-style controller from scratch, in plain Go, so that when Kubebuilder generates code for you in the next post, you understand every line of it — not just what it does, but why it's designed that way.
The code in cmd/control-loop runs six scenarios that would take a non-trivial amount of cluster setup to observe in a real Kubernetes environment. You can run them locally in seconds.
The Three Pieces of Every Kubernetes Controller
Before writing any code, the mental model:
┌─────────────────────────────────────────────────────────┐
│ │
│ etcd (source of truth) │
│ "desired state lives here" │
│ │ │
│ │ watch events │
│ ▼ │
│ Watch Loop │
│ "translates events → work queue entries" │
│ │ │
│ │ key (namespace/name) │
│ ▼ │
│ Work Queue (deduplicated) │
│ "buffers keys needing reconciliation" │
│ │ │
│ │ key │
│ ▼ │
│ Reconcile() │
│ "the only function YOU write" │
│ fetch desired → observe actual → act │
│ │
└─────────────────────────────────────────────────────────┘
That's it. Every Kubernetes controller — the Deployment controller, the ReplicaSet controller, every operator you'll ever write — is a variation on this three-piece pattern. Kubebuilder generates the watch loop and work queue. You implement Reconcile().
Let's build all three.
The Store: Our etcd Analogue
etcd is Kubernetes' distributed key-value store. Every resource you create — Pod, Deployment, your custom DatabaseCluster — gets written to etcd. Controllers never talk to etcd directly; they go through the kube-apiserver. But the fundamental contract is: etcd is the source of truth for desired state.
Our internal/watcher/store.go is a single-process in-memory version of that contract:
type Store struct {
mu sync.RWMutex
data map[string]interface{}
revision int64
subscribers []chan Event
}
Two things worth noting here. First, revision — etcd maintains a global, monotonically increasing revision counter that increments on every write. This is how watch clients know they haven't missed any events: they resume from a specific revision number, not from a timestamp. Second, subscribers — this is the watch fanout. Every controller that registers a watch gets its own channel and receives every event.
The Set method shows how a write flows through:
func (s *Store) Set(key string, value interface{}) {
s.mu.Lock()
old, exists := s.data[key]
s.data[key] = value
s.revision++
// ... copy subscribers while holding lock ...
s.mu.Unlock()
eventType := EventAdded
if exists {
eventType = EventModified
}
// fan out to all watchers
for _, ch := range subs {
select {
case ch <- evt:
default:
// watcher too slow — in real Kubernetes: "too old resource version"
}
}
}
When you run kubectl apply -f cluster.yaml, this is what happens inside the kube-apiserver. The write happens, the revision increments, and every registered watcher gets notified within milliseconds. There is no polling.
The Event Types
type EventType string
const (
EventAdded EventType = "ADDED"
EventModified EventType = "MODIFIED"
EventDeleted EventType = "DELETED"
)
These map directly to watch.EventType in k8s.io/apimachinery. When your controller is running and someone applies a new CRD instance, you get ADDED. When they change a field, MODIFIED. When they delete it, DELETED. Your Reconcile() function handles all three — because it works from current state rather than event payloads, it doesn't need to branch on event type.
The Watch Loop: From Events to Work Queue
This is the piece most operator tutorials skip because Kubebuilder makes it invisible. Understanding it is what separates engineers who can debug controllers from engineers who can only use them.
func (c *Controller) watchLoop() {
events := c.store.Watch()
for {
select {
case evt := <-events:
fmt.Printf("[watch] %s event for %q → enqueuing\n", evt.Type, evt.Key)
c.enqueue(evt.Key)
case <-c.stopCh:
return
}
}
}
Notice what we enqueue: just the key, not the event or the object. This is deliberate and important. Reconcile() will fetch fresh state when it runs — it doesn't use the event payload. This means if 10 events arrive for the same key before Reconcile() runs, they all enqueue the same key, and the work queue deduplicates them down to a single reconciliation.
In controller-runtime, this logic lives in the Informer's event handler callbacks (AddFunc, UpdateFunc, DeleteFunc). They're generated for you, but they do exactly this: extract the key, enqueue it.
Deduplication in the Work Queue
func (c *Controller) enqueue(key string) {
select {
case c.workQueue <- key:
// enqueued
default:
// queue full
}
}
Our work queue is a buffered channel. If the same key is already sitting in the queue, adding it again creates a duplicate — which is fine, because Reconcile is idempotent. But controller-runtime's actual work queue (k8s.io/client-go/util/workqueue) does proper deduplication: it tracks which keys are currently queued and drops duplicates. The result is the same: many events, few reconciliations.
Run Scene 4 and watch this happen:
━━━ SCENE 4: 10 rapid updates — watch deduplication in action ━━━
[etcd] revision=4 MODIFIED key="default/production-postgres"
[etcd] revision=5 MODIFIED key="default/production-postgres"
[etcd] revision=6 MODIFIED key="default/production-postgres"
[etcd] revision=7 MODIFIED key="default/production-postgres"
[etcd] revision=8 MODIFIED key="default/production-postgres"
[etcd] revision=9 MODIFIED key="default/production-postgres"
[etcd] revision=10 MODIFIED key="default/production-postgres"
[etcd] revision=11 MODIFIED key="default/production-postgres"
[etcd] revision=12 MODIFIED key="default/production-postgres"
[etcd] revision=13 MODIFIED key="default/production-postgres"
[reconcile #6] key="default/production-postgres"
[reconcile #6] desired: replicas=5 version=15.5
[reconcile #6] actual: replicas=5 version="15.4" healthy=true
[reconcile #6] ACTION: upgrading → 15.5
[reconcile #6] requeuing in 500ms to verify
[reconcile #7] ✓ in desired state — no action needed
10 etcd revisions. 1 meaningful reconcile action. The rest of the queue was drained against already-correct state. This is the scalability property that makes controllers work at Kubernetes scale — you don't get O(events) reconciliations, you get O(distinct-keys) reconciliations.
This design has a name: level-triggered reconciliation, borrowed from hardware interrupt design. You respond to the current level (state), not the edge (event). The alternative — edge-triggered, where you process each event and try to derive what changed — is fragile because you can miss events, receive them out of order, or process them against stale state.
The Reconcile Loop: The Heart of Every Operator
This is the function you implement in Kubebuilder. Everything else — the store, the watch loop, the work queue — exists to call this function at the right times.
In controller-runtime, the signature is:
func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
Our version is structurally identical:
func (c *Controller) Reconcile(key string) (ReconcileResult, error)
The body has four sections, and every production Reconcile() you'll ever write follows this same structure:
Section 1: Fetch Desired State
raw, exists := c.store.Get(key)
if !exists {
// Resource was deleted — clean up and return
fmt.Printf("[reconcile] resource %q not found — may have been deleted\n", key)
delete(c.actual, key)
return ReconcileResult{}, nil
}
desired := raw.(*DesiredState)
Fetch fresh state every time. Never use the event payload — it may be stale by the time Reconcile() runs. The resource may not even exist anymore: it could have been deleted between the watch event and now. Handle that case first.
In Kubebuilder: r.Get(ctx, req.NamespacedName, &myResource). If it returns errors.IsNotFound(err), handle deletion and return.
Section 2: Observe Actual State
actual, hasActual := c.actual[key]
if !hasActual {
actual = &ActualState{RunningReplicas: 0, CurrentVersion: "", Healthy: false}
}
Query what's actually running. In a real operator this means listing Deployments, checking Pod readiness, or calling an external API. The gap between desired and actual is what your operator's business logic closes.
In Kubebuilder: r.List(ctx, &deploymentList, client.InNamespace(req.Namespace), ...).
Section 3: Compare and Act
if actual.CurrentVersion != desired.Version {
if err := c.simulateUpgrade(key, desired.Version); err != nil {
return ReconcileResult{}, fmt.Errorf("upgrade failed: %w", err)
}
actual.CurrentVersion = desired.Version
}
if actual.RunningReplicas != desired.Replicas {
actual.RunningReplicas = desired.Replicas
}
Each if block handles one type of drift. Returning an error causes a requeue with exponential backoff — the controller-runtime equivalent of "try again later." This is how transient failures (network blips, API rate limits, nodes not ready) self-heal without manual intervention.
Notice the upgrade function has a 15% random failure rate:
func (c *Controller) simulateUpgrade(key, newVersion string) error {
time.Sleep(300 * time.Millisecond)
if rand.Float32() < 0.15 {
return fmt.Errorf("node not ready during rolling restart of %q", key)
}
return nil
}
When this fires, the reconcile loop catches the error, logs it, and requeues with backoff. Run the demo a few times — you'll see it happen, and you'll see the controller recover automatically.
Section 4: Requeue to Verify
if needsRequeue {
return ReconcileResult{Requeue: true, RequeueAfter: 500 * time.Millisecond}, nil
}
// Desired == Actual. Nothing to do.
return ReconcileResult{}, nil
After taking an action, requeue with a short delay to verify it took effect. Your scaling action might succeed at the API level but take a few seconds to propagate — the requeue lets you confirm before declaring victory.
In controller-runtime: ctrl.Result{RequeueAfter: 30 * time.Second}. The duration depends on how long your actions take to propagate.
Running All Six Scenes
git checkout blog-02
go run ./cmd/control-loop
Each scene maps to something you'll encounter in production:
Scene 1 — Resource creation. Watch fires ADDED, controller provisions from zero.
Scene 2 — Scale up. User changes replicas: 3 to replicas: 5. Controller detects the drift and acts.
Scene 3 — Version upgrade. Controller notices version mismatch, runs rolling upgrade, requeuees to verify. If the upgrade fails (15% chance), it retries with backoff.
Scene 4 — Rapid updates. 10 changes in 200ms. Watch fires 10 events, but Reconcile runs only once or twice against final state. This is the deduplication property in action.
Scene 5 — Second resource. A second cluster is created. The controller manages both keys independently, same pattern.
Scene 6 — Deletion. DELETED event arrives. Reconcile finds the resource gone, cleans up internal state. (Finalizers, covered in Blog 9, let you run cleanup logic before deletion completes.)
The summary at the end shows total reconciliation count. On a clean run with no upgrade failures, you'll see around 12–15 reconciliations across all six scenes — even though many more events fired.
The Three Properties That Make This Work
After running the demo, these three properties are worth naming explicitly. Every production operator relies on them.
1. Idempotency. Reconcile() must be safe to call multiple times with the same state. If you call it twice with replicas=3 and the cluster already has 3 replicas, the second call should be a no-op. This isn't just nice to have — Kubernetes guarantees at-least-once delivery of reconcile calls, not exactly-once. Your operator will get called multiple times for the same state.
2. Level-triggered. Work from current state, never from event payloads. If you try to reason about "what changed" from an event, you'll eventually process a stale event, miss one due to a restart, or receive them out of order. Fetch fresh state in every Reconcile() call and let the comparison logic figure out what needs to change.
3. Error = requeue. Returning an error from Reconcile() is not a failure state — it's a signal to try again. Transient failures (the target system isn't ready, a network call timed out, a resource hasn't propagated yet) should all return errors. The controller runtime handles backoff, logging, and retries. Your operator's responsibility ends at returning the error; recovery is automatic.
These three properties are why operators can run unattended. They don't assume success — they verify it. They don't assume events are complete — they read state. They don't give up on transient failures — they retry.
How This Maps to Real Kubernetes
The components we built map directly to the real Kubernetes machinery:
| Our code | Real Kubernetes equivalent |
|---|---|
watcher.Store | etcd + kube-apiserver |
store.Watch() | cache.Informer with ListWatch |
watchLoop | Informer event handlers (AddFunc, etc.) |
workQueue | workqueue.RateLimitingInterface |
reconcileLoop | controller-runtime worker goroutine pool |
Reconcile() | reconcile.Reconciler interface |
ReconcileResult | ctrl.Result |
The real versions add persistence, distribution, authentication, rate limiting, leader election, and a lot more production hardening. But the pattern is identical. Understanding our simplified version means understanding the real one.
What Kubebuilder Actually Gives You
Now that you've seen every piece, here's what Kubebuilder generates when you scaffold a controller:
- The Informer setup (our
store.Watch()+watchLoop) - The work queue and rate limiting (our
workQueue+enqueue) - The reconcile loop goroutines (our
reconcileLoop) - The
Managerthat wires all of this together and handles leader election
What it doesn't generate — what you write — is Reconcile().
That's the contract. Kubebuilder handles all the machinery. You handle the business logic: what desired state looks like, how to observe actual state, and what actions close the gap between them.
In Blog 3 we install Kubebuilder, run kubebuilder init, and look at every generated file with this mental model in hand. Nothing will be surprising.
This is Blog 2 of the series "From Zero to Kubernetes Operators with Kubebuilder (KubeAgent Journey)." The series covers Kubernetes foundations, real operator development with Kubebuilder, and evolving into intelligent KubeAgent-style systems.