Your First Reconciliation Loop — Line by Line

Part of the series: From Zero to Kubernetes Operators with Kubebuilder (KubeAgent Journey)

→ Full code for this post: github.com/madmmas/kubeoperator-journey/tree/blog-04 → Controller: internal/controller/databasecluster_controller.go

Blog 3 ended with this:

func (r *DatabaseClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    // TODO(user): your logic here

    return ctrl.Result{}, nil
}

That TODO is the entire surface area of your operator. Everything else — the watch loop, the work queue, the retry logic, the leader election — is handled by controller-runtime. Your only job is to implement this function correctly.

In this post we replace that stub with a complete, working Reconcile() and explain every line. By the end you'll have an operator that:

• Provisions a PostgreSQL StatefulSet from a DatabaseCluster resource
• Updates Status.Phase and Status.ReadyReplicas from observed facts
• Sets standard Conditions for monitoring and alerting
• Handles creation, updates, deletion, and transient failures correctly

The Four-Step Contract

Every Reconcile() function — regardless of what it manages — follows the same four steps. Memorise this pattern. It's the template for every operator you'll ever write.

Step 1 — Fetch desired state       r.Get(ctx, req.NamespacedName, &cluster)
Step 2 — Observe actual state      r.Get(ctx, ..., &statefulSet)
Step 3 — Act to close the gap      controllerutil.CreateOrUpdate(...)
Step 4 — Update status             r.Status().Patch(ctx, &cluster, patch)

And three return paths that determine what the controller does next:

return ctrl.Result{}, nil                          // done — don't requeue
return ctrl.Result{}, err                          // error — requeue with backoff
return ctrl.Result{RequeueAfter: 10 * time.Second} // requeue after fixed delay

Everything in the rest of this post is an application of these four steps and three return paths.

Step 1: Fetching Desired State

var cluster databasesv1alpha1.DatabaseCluster
if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
    if errors.IsNotFound(err) {
        log.Info("DatabaseCluster not found — likely deleted", "key", req.NamespacedName)
        return ctrl.Result{}, nil
    }
    return ctrl.Result{}, fmt.Errorf("fetching DatabaseCluster: %w", err)
}

r.Get() is store.Get() from Blog 2 — but talking to the kube-apiserver over HTTPS instead of our in-memory map. req.NamespacedName is the namespace/name key we enqueued in the watch loop.

The errors.IsNotFound(err) branch is the deletion case. A DatabaseCluster was deleted between the watch event firing and Reconcile() running. This is completely normal — events are delivered at-least-once, and the resource may no longer exist by the time we process the event. Return nil (not err) — there's nothing to do and nothing to retry.

Any other error (network blip, auth failure, kube-apiserver temporarily unavailable) gets returned unwrapped with %w so the controller-runtime logs it with context and requeues with exponential backoff.

Step 2: Handling First Creation

if cluster.Status.Phase == "" {
    log.Info("New DatabaseCluster detected — setting initial phase")
    cluster.Status.Phase = databasesv1alpha1.PhaseProvisioning
    if err := r.Status().Update(ctx, &cluster); err != nil {
        return ctrl.Result{}, fmt.Errorf("setting initial phase: %w", err)
    }
    return ctrl.Result{}, nil
}

When a DatabaseCluster is first created, its Status is empty — no phase, no conditions, no ready replica count. Users who kubectl get databasecluster immediately after applying would see blank columns.

We set Phase = Provisioning immediately so the status is meaningful from the first second. Then we return without doing any further work. The status update triggers another watch event → another Reconcile() call → the controller continues from where it left off with the phase already set.

This separation — "set initial status" as its own reconciliation pass — keeps each pass focused and debuggable. You can see in the logs exactly what happened in each call.

Notice we use r.Status().Update() here, not r.Update(). The status subresource (enabled by //+kubebuilder:subresource:status in your types file) has its own API endpoint. This means:

• Users can update Spec without accidentally overwriting Status
• Your controller can update Status without triggering a resource version conflict with concurrent Spec updates

Never mix them up. r.Update() is for Spec. r.Status().Update() is for Status.

Step 3: Provisioning the StatefulSet

sts := r.buildStatefulSet(&cluster)

if err := controllerutil.SetControllerReference(&cluster, sts, r.Scheme); err != nil {
    return ctrl.Result{}, fmt.Errorf("setting owner reference: %w", err)
}

result, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
    sts.Spec.Replicas = &cluster.Spec.Replicas
    sts.Spec.Template.Spec.Containers[0].Image = postgresImage(cluster.Spec.Version)
    return nil
})
if err != nil {
    return ctrl.Result{}, fmt.Errorf("reconciling StatefulSet: %w", err)
}

Three things happening here that are worth understanding individually.

buildStatefulSet — Pure Function, No Side Effects

func (r *DatabaseClusterReconciler) buildStatefulSet(
    cluster *databasesv1alpha1.DatabaseCluster,
) *appsv1.StatefulSet {
    labels := map[string]string{
        "app.kubernetes.io/name":       "databasecluster",
        "app.kubernetes.io/instance":   cluster.Name,
        "app.kubernetes.io/managed-by": "kubeoperator-journey",
    }
    // ... returns a fully specified StatefulSet
}

buildStatefulSet constructs the desired StatefulSet in memory — no API calls, no side effects. This matters because it makes the function trivially unit-testable. You can call it in a test with a DatabaseCluster struct and assert on the returned StatefulSet without any cluster or mock client.

The StatefulSet includes a readiness probe:

ReadinessProbe: &corev1.Probe{
    ProbeHandler: corev1.ProbeHandler{
        Exec: &corev1.ExecAction{
            Command: []string{"pg_isready", "-U", "postgres", "-d", "postgres"},
        },
    },
    InitialDelaySeconds: 10,
    PeriodSeconds:       5,
},

This is crucial. ReadyReplicas in the StatefulSet status only counts pods that pass their readiness probe. Without this probe, Kubernetes marks pods ready immediately after they start — before PostgreSQL has finished initialising. Your Status.ReadyReplicas would lie, and your Phase calculation would be wrong.

SetControllerReference — Automatic Garbage Collection

controllerutil.SetControllerReference(&cluster, sts, r.Scheme)

This sets the DatabaseCluster as the owner of the StatefulSet. The consequence: when you kubectl delete databasecluster production-postgres, Kubernetes automatically garbage-collects the owned StatefulSet and all its pods. You get clean deletion for free, without writing a single line of deletion logic in Reconcile().

This is the correct approach for resources that should not outlive their parent. In Blog 9 (Finalizers) we'll cover the case where you do need cleanup logic to run before deletion — for external resources like S3 buckets that Kubernetes can't garbage-collect automatically.

CreateOrUpdate — The Idempotent Apply

result, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
    sts.Spec.Replicas = &cluster.Spec.Replicas
    sts.Spec.Template.Spec.Containers[0].Image = postgresImage(cluster.Spec.Version)
    return nil
})

CreateOrUpdate is the correct primitive for managing child resources. It:

1. Tries to fetch the existing StatefulSet by name
2. If it doesn't exist: creates it using the full object you provided
3. If it exists: calls your mutate function with the current server-side version, then patches only what changed

The mutate function is where you set the fields you own. Notice we only set Replicas and Image — not the entire Spec. Fields like updateStrategy, podManagementPolicy, or annotations that users or other systems may have set are left untouched. This is how you avoid accidentally overwriting things you don't own.

result tells you what happened: controllerutil.OperationResultCreated, OperationResultUpdated, or OperationResultNone. Logging this is useful:

INFO  StatefulSet reconciled  result=created  statefulset=production-postgres
INFO  StatefulSet reconciled  result=none     statefulset=production-postgres

The second line is the steady state — Reconcile ran, compared, found nothing to change. This is the output you want to see most of the time.

Step 4: Observing Actual State

var currentSts appsv1.StatefulSet
if err := r.Get(ctx, types.NamespacedName{
    Name:      sts.Name,
    Namespace: sts.Namespace,
}, &currentSts); err != nil {
    return ctrl.Result{}, fmt.Errorf("fetching StatefulSet status: %w", err)
}

readyReplicas := currentSts.Status.ReadyReplicas

After CreateOrUpdate, we fetch the StatefulSet back from the API to read its current status. We can't use the sts object we just wrote — it contains only what we sent, not what the StatefulSet controller has observed about the running pods.

Status.ReadyReplicas is set by the StatefulSet controller, not by us. It reflects how many pods have passed their readiness probe. This is the authoritative source of truth for whether the cluster is healthy.

Step 5: Updating Status

patch := client.MergeFrom(cluster.DeepCopy())

cluster.Status.ReadyReplicas  = readyReplicas
cluster.Status.CurrentVersion = cluster.Spec.Version

switch {
case readyReplicas == 0:
    cluster.Status.Phase = databasesv1alpha1.PhaseProvisioning
case readyReplicas < desiredReplicas:
    cluster.Status.Phase = databasesv1alpha1.PhaseDegraded
default:
    cluster.Status.Phase = databasesv1alpha1.PhaseRunning
}

setReadyCondition(&cluster, readyReplicas, desiredReplicas)

if err := r.Status().Patch(ctx, &cluster, patch); err != nil {
    return ctrl.Result{}, fmt.Errorf("updating status: %w", err)
}

We use r.Status().Patch() here instead of r.Status().Update(). The difference:

• Update() sends the entire status object and requires a matching resourceVersion. If another reconcile ran between our Get and our Update, the resourceVersion won't match and the update fails with a conflict error.
• Patch() sends only the diff (via client.MergeFrom) and is resilient to concurrent updates. Prefer Patch for status updates.

client.MergeFrom(cluster.DeepCopy()) captures the state before our changes. The patch is computed as the diff between the snapshot and the modified struct.

The setReadyCondition Function

func setReadyCondition(
    cluster *databasesv1alpha1.DatabaseCluster,
    readyReplicas, desiredReplicas int32,
) {
    condition := metav1.Condition{
        Type:               databasesv1alpha1.ConditionReady,
        ObservedGeneration: cluster.Generation,
    }

    if readyReplicas >= desiredReplicas {
        condition.Status  = metav1.ConditionTrue
        condition.Reason  = "AllReplicasReady"
        condition.Message = fmt.Sprintf("All %d replicas are ready", desiredReplicas)
    } else {
        condition.Status  = metav1.ConditionFalse
        condition.Reason  = "ReplicasNotReady"
        condition.Message = fmt.Sprintf("%d/%d replicas ready", readyReplicas, desiredReplicas)
    }

    metav1.SetStatusCondition(&cluster.Status.Conditions, condition)
}

metav1.SetStatusCondition is the correct way to set conditions. It handles three things you'd otherwise get wrong manually:

1. Deduplication — if a condition with this Type already exists, it updates it in place rather than appending a duplicate
2. LastTransitionTime — only updates the timestamp when Status actually changes (True → False), not on every reconcile. This makes condition timestamps meaningful — they tell you when something changed, not just when it was last observed
3. ObservedGeneration — records which version of the Spec this condition reflects, so tooling can tell whether the condition is stale

The Reason field must be a CamelCase string with no spaces. It's the machine-readable part — used in kubectl describe and in alerting rules. Message is the human-readable explanation.

After this update, kubectl describe databasecluster production-postgres shows:

Status:
  Phase:           Running
  Ready Replicas:  3
  Current Version: 15.4
  Conditions:
    Last Transition Time:  2026-07-07T09:23:00Z
    Message:               All 3 replicas are ready
    Reason:                AllReplicasReady
    Status:                True
    Type:                  Ready

Step 6: The Return Decision

if readyReplicas < desiredReplicas {
    log.Info("Waiting for replicas — requeuing",
        "ready", readyReplicas,
        "desired", desiredReplicas,
    )
    return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
}

return ctrl.Result{}, nil

If we're still waiting for replicas to become ready, requeue after 10 seconds. The StatefulSet controller will also fire watch events as pods pass their readiness probe — so in practice we'll reconcile before the 10s is up. The requeue is a safety net for cases where watch events are missed (controller restart, network blip).

If everything is healthy: return nil, don't requeue. The next reconcile will be triggered by the next real change — a user editing the Spec, a pod failing its readiness probe, or the StatefulSet controller updating its status.

SetupWithManager — Watching Owned Resources

func (r *DatabaseClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&databasesv1alpha1.DatabaseCluster{}).
        Owns(&appsv1.StatefulSet{}).
        Complete(r)
}

Owns(&appsv1.StatefulSet{}) is what makes the operator reactive to pod readiness changes without polling. When a StatefulSet we own changes — a pod becomes ready, a pod crashes, the replica count changes — Reconcile() is triggered for the owning DatabaseCluster.

Without .Owns(), we'd only reconcile when the DatabaseCluster itself changes. We'd never know when a pod became ready and Status needed updating — unless we polled with a fixed RequeueAfter.

With .Owns(), the control loop is event-driven end to end. The operator responds to real changes immediately, not on a timer.

Running It Against a Local Cluster

# Create a kind cluster
kind create cluster --name kubeoperator-dev

# Install the CRDs
make generate manifests install

# Run the operator locally (connects to kind cluster via kubeconfig)
make run

In a second terminal:

# Apply a DatabaseCluster
kubectl apply -f - <<EOF
apiVersion: databases.madmmas.dev/v1alpha1
kind: DatabaseCluster
metadata:
  name: production-postgres
  namespace: default
spec:
  replicas: 3
  version: "15.4"
  storageSize: "1Gi"
EOF

# Watch the status update in real time
kubectl get databasecluster production-postgres -w

You'll see the Phase column progress:

NAME                   PHASE          READY   VERSION   AGE
production-postgres    Provisioning   0       15.4      2s
production-postgres    Provisioning   0       15.4      8s
production-postgres    Provisioning   1       15.4      15s
production-postgres    Provisioning   2       15.4      22s
production-postgres    Running        3       15.4      29s

And in the operator terminal, the structured log output shows each reconcile pass:

INFO  Reconciling DatabaseCluster  name=production-postgres  replicas=3  version=15.4  phase=Provisioning
INFO  StatefulSet reconciled       result=created            statefulset=production-postgres
INFO  Observed StatefulSet state   ready=0                   desired=3
INFO  Waiting for replicas — requeuing  ready=0  desired=3

INFO  Reconciling DatabaseCluster  name=production-postgres  replicas=3  version=15.4  phase=Provisioning
INFO  StatefulSet reconciled       result=none               statefulset=production-postgres
INFO  Observed StatefulSet state   ready=1                   desired=3
INFO  Waiting for replicas — requeuing  ready=1  desired=3

INFO  Reconciling DatabaseCluster  name=production-postgres  replicas=3  version=15.4  phase=Running
INFO  StatefulSet reconciled       result=none               statefulset=production-postgres
INFO  Observed StatefulSet state   ready=3                   desired=3
INFO  DatabaseCluster is in desired state ✓  phase=Running  ready=3

Notice result=none after the first pass — the StatefulSet already exists and matches desired state, so CreateOrUpdate does nothing. This is idempotency in action.

Testing Scale-Up

# Scale from 3 to 5 replicas
kubectl patch databasecluster production-postgres \
  --type merge -p '{"spec":{"replicas":5}}'

# Watch the operator respond
kubectl get databasecluster production-postgres -w

The operator detects the spec change, updates the StatefulSet, watches new pods come ready, and transitions from Degraded (2 of 5 ready) to Running (5 of 5). No manual steps. No runbook.

Testing Deletion

kubectl delete databasecluster production-postgres

# The StatefulSet is garbage-collected automatically
kubectl get statefulset production-postgres
# Error from server (NotFound): statefulsets "production-postgres" not found

The owner reference set by SetControllerReference does the work. No explicit deletion logic in Reconcile().

The Complete Return Path Map

Before moving on, it helps to have a clear map of every Reconcile() return path and what it means:

The most common mistake: returning (ctrl.Result{}, nil) when an error occurred, because you handled it "internally." If something went wrong and you didn't return the error, the controller won't retry. The next reconcile only happens when something else changes. Silent failures are the hardest to debug in operators.

What We Haven't Handled Yet

This is a solid first Reconcile() — it provisions, observes, updates status, and handles the deletion case. But production operators need more:

Version upgrades — if Spec.Version changes from 15.4 to 15.5, our current CreateOrUpdate updates the container image but doesn't track that an upgrade is in progress. Status would flip to Degraded during the rolling restart. We need to set Phase = Upgrading and a specific condition. That's Blog 6.

Backup scheduling — Spec.BackupSchedule is defined but ignored. A real operator would create a CronJob resource and manage it alongside the StatefulSet. That's also Blog 6 territory.

External resource cleanup — if our operator created an S3 bucket or a DNS entry, SetControllerReference can't garbage-collect those. We need a Finalizer. That's Blog 7.

When Reconcile() doesn't run — there's an entire class of failure modes where the controller is running but Reconcile() isn't called for a resource you'd expect. Wrong RBAC, missed watches, status update loops, resource version conflicts. That's all of Blog 8.

For now: you have a working operator. It provisions real Kubernetes resources, tracks real cluster state, and handles the full lifecycle correctly.

This is Blog 4 of the series "From Zero to Kubernetes Operators with Kubebuilder (KubeAgent Journey)." The series covers Kubernetes foundations, real operator development with Kubebuilder, and evolving into intelligent KubeAgent-style systems.

Discuss on Twitter • View on GitHub