Ambiguous Goal Interpretation

Issue: Agent optimizes for a different meaning of the user’s/business goal.

Frequency: Common

Symptoms

• Clarification avoided; final output solves adjacent problem.
• Agent silently picks one of several equally plausible readings of an underspecified term (e.g., “archive” vs. “delete”) and proceeds without flagging the alternative.
• The delivered output is internally consistent and well-executed, but addresses a narrower or different scope than the requester had in mind.
• Different stakeholders reviewing the same output disagree about whether it satisfies the original request, revealing the request itself was underdetermined.
• Agent’s stated confidence in its chosen interpretation is high even though the request contains genuinely ambiguous phrasing.

Root Cause Requests are frequently phrased with overloaded, colloquial verbs (“close,” “clean up,” “handle”) that map cleanly onto several distinct system operations, and because intake isn’t structured into an enumerated action list, nothing forces the requester to disambiguate before the agent starts working. Latency and throughput incentives push the agent to auto-execute rather than round-trip a clarifying question, and without access to a domain glossary of previously-resolved terms it has no grounded way to prefer one reading over another. The result is that requester and agent silently operate on different implicit conventions, and the gap only surfaces after an interpretation with irreversible consequences has already been acted on.

Example

A support agent at a B2B SaaS company receives the ticket: "Please close out the Acme Corp
account, they're not renewing." "Close out" is ambiguous between (a) marking the account as
churned/inactive in the CRM while preserving historical data, and (b) fully deleting the
account and its data per a data-retention offboarding flow. The agent infers the second
reading, executes the deletion workflow, and reports success. Three days later the account
team discovers Acme's historical usage data and support history -- needed for a win-back
campaign -- is gone. The agent never surfaced that "close out" had two materially different,
equally plausible executions with irreversible consequences for one of them.

Contributing Factors

• Overloaded or colloquial verbs in the request (e.g., “close,” “clean up,” “handle”) that map to multiple distinct system operations.
• No structured intake schema forcing the requester to pick from an enumerated action list.
• High pressure to auto-execute without a clarification round-trip (latency or throughput incentives).
• Domain glossary/interpretation history not available to the agent at inference time.
• Requester and agent operate with different implicit context (requester assumes a shared convention the agent was never given).

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Structured Goal Restatement Gate: Before execution, the agent must produce a paraphrase of the request plus explicit, checkable acceptance criteria. Execution proceeds only if the user (or an automated confidence check) confirms the restatement, or if a calibrated confidence score clears a threshold. This forces the ambiguity to surface as a question instead of a silent default reading.
2. Ambiguity Scoring via Interpretation Enumeration: Before planning, sample multiple candidate interpretations of the request (e.g., via multi-sample generation or a lightweight parser) and score how much they diverge. If more than one materially different, plausible interpretation exists, block auto-execution and route to a clarifying question instead of picking the most likely one silently.
3. Interpretation-Locked Context Injection: Maintain an org/domain glossary of previously-resolved ambiguous terms and examples (“when a user says X in context Y, they mean Z”). Retrieve and inject the relevant entries before execution so default readings are grounded in prior resolutions rather than the model’s unconstrained guess.

Detection & Response

1. Divergence-from-Candidate-Interpretations Check: After generation, compare the final output’s content/keywords against each candidate interpretation identified pre-execution. Flag and hold for review any output that aligns with a non-primary (adjacent) interpretation rather than the one the user most likely meant.
2. Clarification-Avoidance Rate Monitoring: Track, per session, the fraction of ambiguity-flagged requests where the agent proceeded without asking a clarifying question. A rising trend indicates the clarification gate is being bypassed or under-triggering, and should page the owning team.
3. Post-Delivery Correction Mining: Scan follow-up user turns for correction phrases (“I meant,” “no, I wanted,” “that’s not right”) tied to an earlier ambiguous request. Feed confirmed misinterpretations back into the ambiguity classifier and the interpretation glossary so the same phrasing is caught next time.

Architecture Patterns

1. Clarification Middleware: A pre-execution service sits between the input parser and the planner. It runs the ambiguity classifier on every incoming request and either forwards it to the executor or returns a clarifying-question response, so ambiguity handling is enforced outside the model’s own discretion.
2. Multi-Sample Consensus Check: Sample N candidate plans from the model at a relaxed decoding setting; if the plans diverge substantially (different target entities, different scope), route to a human-in-the-loop or clarifying question rather than silently taking the majority-vote plan.
3. Goal Contract Object: Generate a structured object (task, scope, explicit exclusions, acceptance criteria) once at request intake and pass it through the entire execution pipeline as the single source of truth for “what the goal means,” versioned so later steps can’t silently reinterpret it.

Metrics

1. ambiguity_flagged_rate_percent: Target: 10-20% of genuinely ambiguous request types; Alert threshold: < 5% (classifier likely under-triggering)
2. clarification_ask_rate_on_flagged_percent: Target: > 90%; Alert threshold: < 70%
3. post_delivery_reinterpretation_rate_percent: Target: < 5%; Alert threshold: > 15%
4. interpretation_divergence_score_avg: Target: < 0.3; Alert threshold: > 0.5

Alerts

1. Silent Reinterpretation Spike (P1 - Critical): Condition - post_delivery_reinterpretation_rate exceeds 15% over a rolling 24h window. Action: force clarification-first mode for the affected request category, pause unattended auto-execution.
2. Ambiguity Classifier Drift (P2 - Warning): Condition - ambiguity_flagged_rate falls below 5% for a sustained week despite historical baseline being higher. Action: recalibrate/retrain the classifier, audit recent unflagged sessions for missed ambiguity.
3. Repeated Correction on Same Task Template (P3 - Info): Condition - 3+ user corrections logged against the same task template within 7 days. Action: review and update the task template wording or glossary entry to close the interpretation gap.

Production Signals

Key Metrics

Alerts

References

• MS-Agentic-Failure-Taxonomy
• Note: Agentic AI failure modes; safety/security; memory poisoning; tool use; multi-agent risks.