Business-Policy Mismatch

Issue: Agent completes a technical action that violates company policy.

Frequency: Common

Symptoms

• Compliance review flags policy violation.
• Agent approves or executes a technically valid transaction that falls outside written policy limits (e.g., spend cap, vendor allowlist).
• Action succeeds against the underlying API/system even though a policy rule would have blocked it if checked.
• Policy violation is only caught after the fact by a human auditor, not at execution time.
• Agent cites a plausible-sounding but non-existent or outdated policy justification for the action it took.

Root Cause Policy rules live in documents that are never retrieved or checked at the moment of action, while the tools the agent calls expose permissions far broader than what policy actually allows — the underlying API will happily process an amount or vendor that the written rules forbid. Because there is no pre-action gate that distinguishes “technically possible” from “permitted,” and the agent’s working knowledge of policy thresholds can be stale relative to recent changes, a technically valid action sails through with no check against the rule that should have stopped it. Compliance sampling that happens after the fact, on a lag, means these gaps are discovered well after execution rather than prevented at the point of decision.

Example

A finance-ops agent handles employee expense reimbursements. An employee submits a $2,400
client-dinner receipt. Written policy caps client entertainment at $150/head and requires
pre-approval above $500, but the agent only has access to the generic "reimburse valid
business expense" tool and no structured policy lookup. The receipt looks legitimate
(itemized, business-related), so the agent approves and reimburses the full amount,
reasoning that the expense is "clearly business-related." The action is technically
achievable and the receipt is genuine, but it violates the company's spend-cap and
pre-approval policy. The violation surfaces two weeks later during a routine quarterly
audit, by which point the reimbursement has already been paid out.

Contributing Factors

• Policy rules live in a separate wiki/PDF that isn’t retrieved or checked at the moment of action.
• Tool/API permissions are broader than what policy actually allows (the agent can approve any amount even though policy caps it).
• Policy changes frequently and the agent’s cached or trained knowledge of the rules is stale.
• No pre-action gate distinguishing “technically possible” from “permitted.”
• Reviewer/compliance sampling happens post-hoc on a lag, so violations aren’t caught until well after execution.

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Policy-as-Code Pre-Action Gate: Every action call is evaluated by a policy engine (e.g., OPA/Rego rules) against the current policy set before execution, using structured inputs (actor, action type, target, context). Actions that fail the check are blocked outright, decoupling “technically possible” from “permitted,” which is the specific gap this failure exploits.
2. Policy Retrieval-Augmented Planning: Before drafting a plan, the agent retrieves the relevant policy clauses (via RAG over a versioned policy corpus) tied to the action’s domain and type, so plan generation is policy-aware from the start rather than only checked as an afterthought once the plan is formed.
3. Segregation of Capability and Authority: Tool wrappers expose a narrower permission surface than the underlying system supports (e.g., a refund tool enforces dollar caps and category exclusions independent of what the raw payment API could technically do), so violating policy requires an explicit, logged override path rather than being reachable by default.

Detection & Response

1. Real-Time Policy Re-Evaluation: Every completed action is logged and asynchronously re-evaluated against the latest policy version, catching cases where policy changed between plan-time and audit-time or where the gate was misconfigured at execution time.
2. Compliance Review Sampling: A statistically sampled slice of transcripts is routed to compliance reviewers; violation rate is tracked by action type and team, and confirmed violations feed back into policy engine rule updates rather than being handled as one-offs.
3. Justification-to-Policy Consistency Check: Automatically compare the agent’s stated rationale for an action against the policy clause it should be citing; a mismatch (rationale doesn’t reference or align with any applicable policy) is flagged even when the action itself wasn’t explicitly blocked.

Architecture Patterns

1. Policy Engine Sidecar: A dedicated policy service (e.g., OPA) is queried synchronously before every policy-relevant action and returns allow/deny/require_approval. It is deployed and versioned independently of the agent so policy updates take effect without redeploying the agent.
2. Policy Version Pinning + Change Feed: Policy documents are versioned; the agent subscribes to a change feed and invalidates any cached policy context on update, preventing execution against a stale policy snapshot.
3. Immutable Action-Policy Audit Log: Every action is recorded together with the exact policy version and clause it was evaluated against, stored immutably to support compliance audits and regulatory reporting.

Metrics

1. policy_check_coverage_percent: Target: 100% of policy-relevant actions gated; Alert threshold: < 100%
2. policy_violation_rate_percent: Target: < 0.1%; Alert threshold: > 0.5%
3. stale_policy_context_incidents: Target: 0 per month; Alert threshold: > 0
4. compliance_review_flag_rate_percent: Target: < 1%; Alert threshold: > 3%

Alerts

1. Policy Gate Bypass Detected (P1 - Critical): Condition - a policy-relevant action executed without a recorded policy-engine evaluation. Action: halt action-executing agents in the affected domain, page on-call, run incident review.
2. Confirmed Policy Violation (P1 - Critical): Condition - compliance review confirms a completed action violated policy. Action: reverse or remediate the action where possible, notify the affected party, root-cause why the gate missed it.
3. Policy Staleness Warning (P3 - Info): Condition - agent’s cached policy context exceeds its TTL while an upstream policy change is pending. Action: force a cache refresh and audit any actions taken during the stale window.

Production Signals

Key Metrics

Alerts

References

• NIST-AI-RMF
• Note: Govern, map, measure, manage framework for AI risk.