Multi-Agent Handoff Drops Override Flag Between Deploy and Rollback Agent

Issue: A Deploy Agent Notes in Its Free-Text Reasoning That a Just-Deployed Change Includes a Manual Hotfix Applied to Address a Separate, Concurrent Incident, but This Note Is Never Written to a Structured Field the Downstream Rollback Agent Reads, So an Automated Rollback Triggered by an Unrelated Regression Reverts the Hotfix Along With the Bad Change

Frequency: Occasional

Symptoms

• A regression completely unrelated to a recent hotfix triggers an automated rollback, and the rollback silently takes the hotfix down with it because nothing in the rollback agent’s view of deploy history distinguishes that revision from any other
• The commentary attached to the hotfix names both its purpose and the fact that it should survive an unrelated rollback, in language a person would immediately understand, sitting one field over from the ones the rollback agent actually reads
• Re-running the identical rollback decision with a structured do-not-revert marker added to that one revision produces the hotfix-preserving outcome, confirming the rollback agent’s decision logic itself is not at fault
• The previously resolved incident reappears the moment the rollback completes, so the team ends up re-diagnosing and re-fixing a problem it had already solved once
• The same blind spot applies to any revision applied outside the normal release cadence, not just this hotfix, since none of them have a field to be protected in

Root Cause Deploy-history entries are schematized around the normal release path, where every revision goes through the same review process and none needs to be marked differently from any other. An out-of-band hotfix, applied ahead of the scheduled release specifically to firefight a separate incident, is by definition an exception to that normal path, but the deploy-history schema has no way to record that a given entry is an exception – the only place that distinction gets written down is the commentary field a human would read, not a field the rollback agent’s automated decision logic was built to check, because that automation was designed to handle the normal case, where no entry ever needs special protection.

Example

Deploy agent applies a hotfix patch to address a separate, ongoing incident, and notes in its deploy commentary: "Hotfix for INC-4821, applied ahead of the scheduled release -- do not revert independently of a fix for INC-4821"
Deploy agent's structured deploy-history entry for this revision contains only standard fields (version, timestamp, author); the do-not-revert status exists only in the free-text commentary field, which the rollback agent's decision logic does not read
A few hours later, an unrelated regression in the same release triggers an automated rollback
Rollback agent reverts to the prior revision, which removes the hotfix along with the unrelated regression, because its structured deploy-history view shows no protected-change marker on the hotfix revision
INC-4821 resurfaces immediately after the rollback completes, requiring a second hotfix and a second incident response

Key Statistics

Contributing Factors

• Deploy agent’s output contract permits a protected-change note to exist only in free-text commentary without requiring a corresponding structured “do-not-revert” field
• Rollback agent is designed to act on the structured deploy-history record only, with no step that re-scans recent commit commentary for protection language before reverting
• No automated reconciliation compares protection-relevant language in deploy commentary against the structured deploy-history fields actually consulted by the rollback agent

Mitigation Strategies

1. Mandatory Structured Protection Field for Out-of-Band Hotfixes: Require the deploy agent to populate a structured protected_change field (with reason and related incident ID) for any hotfix applied outside the normal release cycle, and block the deploy from completing without it
2. Commentary-to-Field Reconciliation Check: Run an automated scan of deploy commentary for protection-relevant keywords (“hotfix,” “do not revert”) and verify a corresponding structured field exists before the deploy is marked complete, flagging any mismatch for human review
3. Rollback Agent Checks Protection Field Before Reverting: Require the rollback agent’s decision logic to check the structured protection field on every revision in the rollback range and explicitly halt for human confirmation if any revision is marked protected
4. Selective Rollback for Protected Ranges: When a protected revision exists within the rollback range, require the rollback agent to compute a selective revert that excludes the protected change rather than defaulting to a full revert to the prior revision

Metrics

• Rate of automated rollbacks that revert a revision later found to have had protection-relevant commentary but no structured protection field
• Percentage of out-of-band hotfixes deployed with a structured protected_change field populated versus commentary-only
• Time between a protected change being re-introduced by rollback and the related incident resurfacing

Alerts

• Automated rollback reverts a revision with a structured protected_change field set → P1
• Reconciliation check finds a deploy with protection-relevant commentary and no structured protection field, and a rollback occurs before resolution → P1
• A deploy or rollback workflow is modified without a mandatory structured protection-field check → P3

References

• Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows
• Why Do Multi-Agent LLM Systems Fail? (MAST)
• Automated Cloud Infrastructure-as-Code Reconciliation with AI Agents