Orchestrator Status Mistaken for Application Health After Rollback

Issue: Rollback Agent Declares a Rollback Successful Based on the Deployment Orchestrator Reporting “Rollout Succeeded,” Without Verifying That the Application Is Actually Serving Traffic Correctly at the Prior Version

Frequency: Common

Symptoms

• Agent marks a rollback complete as soon as the orchestrator reports all replicas at the target revision and passing their configured readiness probe, without any check of application-level behavior (correct responses, expected error rate, absence of the original incident’s symptom)
• A shallow readiness probe (e.g., a lightweight /healthz endpoint that only checks the process is listening) passes immediately after rollback while the application is still serving errors due to stale in-memory cache, an unrefreshed feature-flag state, or a downstream dependency that wasn’t rolled back in lockstep
• Rollback is reported “successful” and the incident is marked resolved in the tracker, but the original customer-facing symptom (elevated error rate, wrong output) continues unchanged after the orchestrator-level rollout completes
• On-call discovers the rollback did not actually fix anything only when the same alert re-fires or a customer re-reports the issue, well after the agent’s own status already showed green
• The gap concentrates on rollbacks of services with any state that isn’t reset purely by redeploying the old artifact (in-process caches, connection pools warmed against the bad version’s config, feature flags evaluated at process start) — cases where “pods are Ready” and “application is healthy” diverge

Root Cause Rollback automation commonly treats the deployment orchestrator’s own rollout-completion signal (desired replica count reached, readiness probe passing) as a sufficient proxy for “the rollback fixed the problem,” because that signal is the one the orchestrator already exposes and requires no additional instrumentation to consume. Readiness and rollout-completion checks are designed to answer a narrower question — is a process running and minimally responsive — not the actual question a rollback needs answered, which is whether the specific symptom that triggered the rollback has cleared. An agent whose success criterion is “orchestrator says Succeeded” rather than “the incident’s own health signal has returned to baseline” will confidently report success even when a fully mechanically correct rollback fails to resolve the underlying problem, because it never asked the second question.

Example

Incident: Checkout-service p99 latency triggers a page; recent deploy 4 minutes prior is the likely cause
Rollback agent: Issues `kubectl rollout undo deployment/checkout-service`
30 seconds later: Orchestrator reports all 12 replicas at the prior revision, readiness probe passing
Rollback agent: Marks rollback complete, closes the automated remediation action, notifies on-call
"resolved via automated rollback"
Actual state: The prior version's pods came up cleanly and pass their liveness/readiness probe (a
simple TCP check), but they're reading from a connection pool still pointed at a schema migration
applied by the bad version, so checkout requests continue failing with 500s at the same rate as
before the rollback
On-call, trusting the "resolved" status, moves to close the incident; the same latency/error alert
re-fires 6 minutes later, and only manual investigation at that point discovers the rollback never
actually restored working checkout behavior

Key Statistics

Eval Recipes

Test Cases

Evaluation Dataset

• Source: Synthetic incident-rollback traces built from documented cases where orchestrator-level rollout success diverged from application-level recovery (stale cache, unreset feature flags, unretracted schema/connection-pool state), supplemented with historical postmortems where an automated or manual rollback was marked “resolved” prematurely
• Size: 100+ synthetic rollback traces across at least 3 categories of orchestrator/application-health divergence
• Key variations: readiness-probe depth (shallow liveness check vs. deep application-level check); symptom-recovery latency relative to pod-ready latency; single-service rollback vs. rollback of a service with an unrolled-back stateful dependency

Metrics

Automated Checks

def check_for_failure(rollback_report, orchestrator_status, symptom_metric_series):
    """
    rollback_report: {"status": "resolved"|"failed", "reported_at": timestamp}
    orchestrator_status: {"rollout_state": "succeeded"|"in_progress"|"failed", "readiness_probe": "passing"|"failing"}
    symptom_metric_series: list of (timestamp, value) for the metric that triggered the original incident,
                            plus "baseline_threshold" the metric must return under/over
    """
    if rollback_report["status"] != "resolved":
        return False

    if orchestrator_status["rollout_state"] != "succeeded":
        return True  # reported resolved without even a clean rollout

    # Find the symptom metric's value at or shortly after the reported resolution time
    post_report_values = [v for (t, v) in symptom_metric_series if t >= rollback_report["reported_at"]]
    if not post_report_values:
        return True  # no symptom-metric evidence exists to justify "resolved"

    baseline = symptom_metric_series[-1] if isinstance(symptom_metric_series, dict) else None
    # Simplified: failure if the symptom metric had not returned to baseline at report time
    return post_report_values[0] > rollback_report.get("baseline_threshold", float("inf"))

Mitigation Strategies

Prevention

1. Symptom-Metric Confirmation Gate: Require the rollback agent to identify the specific metric or signal that triggered the incident and confirm it has returned to its pre-incident baseline before reporting the rollback resolved — orchestrator rollout-completion status is a precondition to check the symptom, never a substitute for checking it.
2. Layered Health Definition: Define rollback success as a composite of at least two layers — orchestrator-level (replicas at target revision, probe passing) and application-level (symptom metric, synthetic transaction check) — and require both before the agent’s own success criterion is satisfied.
3. Stateful-Dependency Awareness in Rollback Scope: When a service has known stateful dependencies not reset by artifact-only rollback (connection pools, in-process caches, feature-flag evaluation at startup), require the rollback plan to include an explicit reset/restart step for those dependencies, not just the deployment revert.

Detection & Response

1. Post-Resolution Re-Alert Tracking: Automatically flag any rollback marked “resolved” where the original triggering alert re-fires within a defined window afterward, and treat this as a confirmed instance of this failure mode requiring root-cause review of the verification gap.
2. Symptom-Orchestrator Divergence Dashboard: Track, for every rollback, the time delta between orchestrator “Succeeded” and symptom-metric baseline recovery; a persistent or growing delta across services indicates the verification gate is not effectively catching this class of premature resolution.

Architecture Patterns

• Two-Signal Resolution Gate: A rollback is only eligible to be marked “resolved” once both the orchestrator’s rollout-completion signal and an independent, incident-specific symptom-metric check pass; either signal alone is insufficient and blocks resolution.
• Independent Post-Rollback Verification Service: Symptom verification runs as a separate check outside the rollback agent’s own execution path (different metric source, different service) so a single blind spot in the rollback agent’s assumptions cannot both perform and self-certify the rollback.

Production Signals

Key Metrics

Alerts

References

• Proof of Execution: Runtime Verification for Governed AI Agent Actions
• Verify Before You Commit: Towards Faithful Reasoning in LLM Agents via Self-Auditing
• RIVA: Leveraging LLM Agents for Reliable Configuration Drift Detection