Repository navigation
fix(primitives): stop stale DaemonSets and ReplicaSets reporting healthy - #223
Merged
Merged
Conversation
…caSet The DaemonSet and ReplicaSet grace handlers did not check observedGeneration when pods were desired, so a stale generation with all pods ready gave Healthy for grace while convergence reported Updating. The DaemonSet handlers also did not read updatedNumberScheduled. With maxSurge the controller keeps the old pod on a node until the new pod is ready and counts only the oldest pod, so a rollout whose new pods never become ready reported Healthy. Both kinds now use the rule of the Deployment and StatefulSet handlers. The controller must observe the current generation and the ready count must equal the desired count. For a DaemonSet, updatedNumberScheduled must reach desiredNumberScheduled, except with the OnDelete strategy, because the controller does not replace those pods. A DaemonSet has no scale-down to wait for. For a ReplicaSet, status.replicas must not be more than the desired count. The grace handlers report Down when pods are desired and none are ready, Healthy only when convergence is Healthy, and Degraded otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The health predicates, strategy exceptions, tests, and synchronized documentation are consistent and complete.
Review effort: Balanced
Findings: None
What changed in this PR
Aligns DaemonSet and ReplicaSet health reporting with other workload primitives, preventing stale or incomplete states from reporting healthy.
Changes:
- Adds generation, readiness, rollout, and scale-down health checks.
- Adds regression coverage for stale status and incomplete convergence.
- Updates GoDoc and synchronized user/plugin documentation.
| File | Description |
|---|---|
pkg/primitives/daemonset/handlers.go |
Strengthens DaemonSet health checks. |
pkg/primitives/daemonset/handlers_test.go |
Tests stale and unfinished rollouts. |
pkg/primitives/daemonset/builder.go |
Updates builder GoDoc. |
pkg/primitives/daemonset/resource.go |
Updates resource GoDoc. |
pkg/primitives/replicaset/handlers.go |
Adds scale-down and generation checks. |
pkg/primitives/replicaset/handlers_test.go |
Tests stale and scaling states. |
pkg/primitives/replicaset/builder.go |
Updates builder GoDoc. |
pkg/primitives/replicaset/resource.go |
Updates resource GoDoc. |
docs/primitives/daemonset.md |
Documents DaemonSet status handling. |
docs/primitives/replicaset.md |
Documents ReplicaSet status handling. |
plugin/skills/using-primitives/references/primitives/daemonset.md |
Synchronizes DaemonSet plugin reference. |
plugin/skills/using-primitives/references/primitives/replicaset.md |
Synchronizes ReplicaSet plugin reference. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A DaemonSet or ReplicaSet whose controller had not observed the current spec could report
Healthyafter the graceperiod, and a DaemonSet rollout whose new pods never become ready reported
Healthy. This PR brings the DaemonSet andReplicaSet status handlers in line with the rule from #220: the converging and grace handlers agree, and
Healthyneeds the current generation observed, the ready count matched, and the rollout or scale-down finished. A DaemonSet with
the
OnDeletestrategy still reportsHealthyonce its pods are ready, because the controller does not replace them.Changes
DefaultConvergingStatusHandlerandDefaultGraceStatusHandlerrequirenumberReadyto equaldesiredNumberScheduled, andupdatedNumberScheduledto reach it. A ready but unfinished rollout reportsUpdating(
Creatingright after create) withWaiting for rollout: N/M pods updated.OnDeleteskips the rollout check. Adesired count of zero is still
Healthyonce the generation is observed.status.replicasnot to be more than the desired count, and reportWaiting for scale-down: N/M replicaswhile it is higher.Downwhen pods are desired and none are ready,Healthyonly when the converging handlerreports
Healthy, andDegradedfor all other states, including a staleobservedGeneration.numberReadyto equal the desired count, not to be at least it. Thegrace handler already reported
Degradedfor more ready pods than desired, so the two handlers disagreed there. Theupstream controller never reports that state.
Waiting for DaemonSet controller to observe latest spec, thesame text as the converging handler.
docs/primitives/daemonset.md(rewritten) anddocs/primitives/replicaset.md(new),synced to the plugin references. GoDoc on the handlers, builders and resources matches.
Challenges
The DaemonSet controller counts only the oldest pod on each node when it computes
numberReadyandupdatedNumberScheduled(updateDaemonSetStatusin Kubernetes v1.34.1). WithmaxSurge, the old pod stays until thenew pod is ready, so
numberReadycan equal the desired count whileupdatedNumberScheduledstays below it. This iswhy the rollout check is needed. A DaemonSet has no scale-down analogue: the controller counts at most one pod for each
node that must run one, and
kubectl rollout statusdoes not wait fornumberMisscheduled.kubectl rollout statusalso waits for
numberAvailable. The handlers usenumberReady, like the other workload kinds.Related
MessageQueuegrace handler example indocs/custom-resource.mdhas the same missing generationcheck.
e2e/primitives/deployment_test.gowritesuseUpdatedImagefrom the testgoroutine while the reconcile goroutine reads it.
Testing
TestDefaultHandlers_UnfinishedRollout(DaemonSet) andTestDefaultHandlers_NotConverged(ReplicaSet) call theconverging handler (what the component reports before the grace period expires) and the grace handler (what it reports
after) on the same object. Both tables cover a stale generation with all pods ready and with none ready. The DaemonSet table
adds a surge rollout with all old pods ready and an unfinished rollout just after create. The ReplicaSet table adds a
scale-down with one pod above the desired count.
TestDefaultHandlers_OnDeletechecks that anOnDeleteDaemonSet withold pods stays
Healthy. Every new case failed before the fix except two that pin existing results: the stalegeneration with none ready (
Down) andOnDelete(Healthy). The healthyDaemonSet fixtures now set
updatedNumberScheduled, and two existing DaemonSet tests changed with the contract (moreready than desired, and the grace reason for a stale generation).
make allpasses, andmake e2e-primitivespasses forPRIMITIVE=daemonsetandPRIMITIVE=replicaset.🤖 Generated with Claude Code