Multi-Agent Handoff Drops Flagged Risk Between Review and Summary Agent

Issue: A Document-Review Agent in a Multi-Stage Due-Diligence Pipeline Identifies a Material Risk (a Change-of-Control Clause, an Undisclosed Litigation Reference, a Non-Standard Indemnification Carve-Out) Only in Its Own Free-Text Annotation of the Reviewed Document, and a Downstream Summary Agent That Generates the Diligence Memo from a Structured Findings List Never Sees It

Frequency: Occasional

Symptoms

• Final diligence memo omits a risk that the document-review stage’s own annotation or commentary on the underlying document explicitly identified, discoverable only by re-reading the review stage’s raw output rather than the structured findings it passed downstream
• The structured findings list consumed by the summary agent shows no entry corresponding to a risk the review agent’s free-text commentary on the same document clearly flagged
• Documents reviewed early in a large diligence batch are disproportionately affected, since review-stage output volume peaks there and a risk annotation is more likely to get buried in lengthy commentary the structured-findings extraction step has to condense
• Re-running the same document’s review output through the summary agent with the full annotation explicitly included in context (rather than only the structured findings list) correctly surfaces the risk in the memo, isolating the handoff – not the summary agent’s reasoning capability – as the failure point
• Post-closing disputes or indemnification claims occasionally trace back to a risk that diligence review had, in fact, identified internally but that never appeared in the memo that informed the deal decision

Root Cause The summary agent was built to synthesize a memo from the structured findings list, not to re-read every document’s full annotation – so a risk that exists only as free-text commentary is invisible to it by construction, not by oversight. This becomes a problem specifically because the review agent’s output contract treats structured findings-list entries and free-text commentary as independent channels: nothing requires a risk noted in commentary to also produce a corresponding structured entry, so the two channels drift apart whenever the reviewer’s phrasing doesn’t map cleanly onto a predefined findings category.

Example

Document-review agent reviews a target company's material contract and notes in its commentary: "Note: this contract contains a change-of-control clause that could trigger termination rights for the counterparty upon this transaction's closing -- worth flagging to deal team"
Review agent's structured output for this document lists only the contract's standard metadata (parties, term, renewal date) in the findings-list fields the summary agent consumes; the change-of-control flag exists only in the free-text commentary field, which is not part of the structured findings schema
Summary agent generates the diligence memo from the structured findings list across all reviewed documents, and the change-of-control risk -- never present in any structured field -- does not appear in the memo
Deal proceeds without the counterparty-termination risk having been raised to the deal team; it surfaces only after closing when the counterparty exercises the termination right the original review had already identified internally

Key Statistics

Contributing Factors

• Review agent’s output contract permits a material-risk note to exist only in free-text commentary without requiring a corresponding structured findings-list entry
• Summary agent is designed to synthesize the structured findings list only, with no step that re-scans review-stage commentary for risk language not captured in structured fields
• No automated reconciliation compares risk-relevant language in review-stage commentary against the structured findings list actually passed to the summary stage

Mitigation Strategies

Prevention

1. Mandatory structured risk-flag interface with schema enforcement: Redesign review agent’s output contract to require: (a) for any document where review-stage commentary identifies a material risk, explicit population of structured fields {risk_category, risk_description, severity_level, recommended_action}, (b) validation gate: block review record from completion unless all risk_category items are populated when commentary mentions risk keywords, (c) schema enforcement via JSON Schema or Protocol Buffer validation, (d) pipeline fails open: if risk detected but structured fields missing, escalate to human reviewer for manual categorization before document proceeds to summary stage. Root cause: Prevents undocumented risks by enforcing structured risk fields as mandatory exit condition.

2. Commentary-to-structured-findings reconciliation with automated gap detection: Implement post-review audit: (a) parse review-stage free-text commentary for risk-keyword patterns (“risk”, “concern”, “flag”, “material”, “undisclosed”, “change-of-control”, “termination trigger”), (b) cross-check against structured findings list for corresponding risk entries, (c) if risk keyword found with no matching structured field, auto-escalate to reviewer: “Risk keyword detected in commentary but not in structured findings. Please confirm: is this intentional, or should it be added?” (d) log all gap instances in audit trail. Root cause: Detects and forces closure of commentary-to-structured-findings gaps before handoff.

3. Multi-stage handoff contract validation with invariant checking: Define handoff contract between review and summary agents: required fields (risk_flags[], material_risks[], change_of_control_flag, undisclosed_liabilities_flag), optional fields (commentary). Before passing review output to summary agent, validate: (a) all mandatory fields populated, (b) semantic invariant: if risk keywords in commentary, at least one structured risk_flag entry exists, (c) count invariant: number of risk_category entries >= count of risk keywords in commentary, (d) fail if invariants violated. Root cause: Prevents handoff when contract violated.

Detection & Response

1. Handoff audit logging with commentary-findings reconciliation: For each document reviewed, log: {document_id, review_date, risk_keywords_found_in_commentary: [list], structured_risk_flags_populated: [list], mismatch_count, reconciliation_status (PASS/FAIL/ESCALATED), reviewer_action_if_gap}. Run daily audit: sample 10% of reviews from past 5 days, auto-scan commentary for risk keywords, verify structured fields match. Alert if mismatch rate >2%. Generate weekly report: “Documents with commentary-findings gaps by category” for trend analysis.

2. Memo finalization gate with explicit risk-flag reconciliation: Before diligence memo published to deal team, trigger automated reconciliation: (a) extract all risk_flags from all reviewed documents, (b) scan memo text for mentions of each flagged risk, (c) flag risks not mentioned in memo for explicit deal-team review, (d) require deal team to sign off: “We acknowledge the following risks were identified during review but not included in this memo: [list]”. Prevents un-reviewed deal progression.

Architecture Patterns

1. Handoff Contract Engine with Schema Enforcement: Review agent output → JSON Schema validation (enforces risk_flags[], commentary reconciliation) → Semantic invariant checker (if risk keywords in commentary, risk_flag entry required) → Structured findings list → Summary agent input. Validation fails open: unvalidated reviews escalated to human.

2. Risk-Keyword Detector with Gap Reconciliation: NLP pipeline detects risk patterns in commentary (trained on legal risk language). On each review: (a) identify risk keywords, (b) cross-check against structured_risk_flags for matching entries, (c) report gaps to reviewer for confirmation/closure before handoff. Maintains risk taxonomy: {keyword_pattern, required_structure_field, severity_if_missed}.

3. Deal-Memo Reconciliation Gate: Before memo publication: (a) enumerate all risk_flags from document reviews, (b) full-text search memo for each flagged risk, (c) unmapped risks tagged for deal-team acknowledgment, (d) sign-off gate: memo release blocked until deal team explicitly reviews unmapped risks. Tracks acknowledgment audit trail.

Key Metrics

Alerts & Escalation

References

• Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows
• Why Do Multi-Agent LLM Systems Fail? (MAST)
• Large Language Models Meet Legal Artificial Intelligence: A Survey