Bad Refusal

Issue: Agent refuses safe requests or gives unsafe help.

Frequency: Common

Symptoms

• False positive/negative safety classifications.
• Agent refuses a benign request because it superficially resembles a disallowed one (e.g., refusing to explain a billing dispute process because it mentions “chargeback”).
• Agent provides step-by-step help with an action it should have declined or routed to a human (e.g., walking a user through disabling account security features) because the request was phrased innocuously.

Root Cause Safety classification here is largely keyword-driven, flagging surface terms like “chargeback” or “password” rather than evaluating the actual intent behind the request, and it operates on each message in isolation, missing conversational context that would either confirm legitimate intent or reveal a social-engineering attempt. Because the classifier only has a binary allow/refuse decision to work with — no graded tier for a caveated or partial answer — it forces every borderline case to one extreme or the other, and sparse or stale eval coverage of near-miss safe/unsafe pairs means both kinds of misfire can regress silently for a long time before anyone notices.

Example

User: "My toddler somehow charged $200 on in-app purchases, can you help me dispute this?"
Agent: "I'm sorry, I can't assist with disputing financial charges or discussing chargebacks."

[Separately, in the same deployment:]
User: "I lost access to my email, can you just remove 2FA from my account so I can get back in?"
Agent: "Sure, here's how to disable two-factor authentication on your account..."

Contributing Factors

• Keyword-based safety filtering flags surface terms (“chargeback,” “dispute,” “password”) without evaluating actual intent or context.
• No graded response tiers, so the classifier only has a binary allow/refuse decision even when a caveated or partial answer would be appropriate.
• Safety classifier evaluates each message in isolation, missing conversational context that would clarify legitimate intent (or reveal a social-engineering attempt).
• Sparse or stale eval coverage of near-miss safe/unsafe pairs lets both false positives and false negatives regress unnoticed.

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Calibrated refusal policy with graded responses: replace binary allow/refuse with graded responses (full help, help-with-caveat, partial help, refuse) tied to specific policy categories, since the root failure is a binary classifier producing both false positives (over-refusal) and false negatives (unsafe help) rather than a calibrated risk response. Trade-off: graded policies are harder to specify precisely and increase prompt/policy complexity.
2. Refusal eval suite covering adjacent safe/unsafe pairs: build eval sets of near-miss pairs (a safe request that resembles an unsafe one, and vice versa) and regression-test every policy/prompt change against them. Trade-off: building and maintaining a high-quality near-miss eval set requires ongoing human red-teaming effort.
3. Context-aware safety classification: feed the safety classifier the full conversation context and stated business purpose, not just the isolated message, since many false-positive refusals come from single-message classification missing legitimate context. Trade-off: richer context increases the chance of a jailbreak using fabricated context to justify an unsafe request.

Detection & Response

1. Refusal-rate anomaly detection: track refusal rate per intent category; a rate that jumps or sits far above historical baseline signals a false-positive spike. Response: sample refused transcripts from the anomalous category and re-classify by hand.
2. Post-hoc unsafe-help audit sampling: randomly sample “helped” conversations in risk-relevant categories (financial, medical, account security) for human safety review, since false negatives don’t self-report the way user complaints do for over-refusal. Response: any confirmed unsafe-help case triggers immediate policy patch and eval-set addition.
3. User pushback/appeal detection: detect user language indicating a refusal was wrong (“I’m just asking about…”, explicit complaints) and route to a refusal-appeals queue. Response: humans triage and correct the underlying policy or few-shot examples.

Architecture Patterns

1. Two-stage classify-then-respond pipeline: separate the safety classification step from response generation so refusal decisions are auditable and independently testable, rather than an entangled judgment made inline during generation.
2. Human-in-the-loop gate for borderline scores: route requests near the classifier’s decision boundary to a lightweight human or secondary-model review rather than forcing a binary auto-decision where most false positives/negatives cluster.
3. Policy-as-config versioning: keep refusal policy thresholds and category definitions in versioned config separate from the model prompt, so refusal behavior can be tuned and rolled back independently of broader prompt changes, with each change gated by the eval suite.

Metrics

1. false_positive_refusal_rate: Target: <2% of safe requests; Alert on >4% weekly
2. false_negative_unsafe_help_rate: Target: <0.5% of risk-category conversations (from audit sampling); Alert on any confirmed case
3. refusal_rate_by_category: Target: within +/-20% of 90-day baseline per category; Alert on >50% deviation week over week
4. refusal_appeal_rate: Target: <3% of refusals appealed; Alert on >6%

Alerts

1. Unsafe Help Confirmed (P1): Condition - human audit confirms an unsafe response was given. Action: immediate policy patch, add case to eval regression set, notify safety/compliance lead within 1 hour.
2. Refusal Rate Spike (P2): Condition - refusal rate in any intent category exceeds baseline by 50% over 24h. Action: pause the triggering prompt/policy change if recent, sample transcripts, roll back if false positives confirmed.
3. Appeal Queue Backlog (P3): Condition - refusal-appeals queue exceeds SLA (unresolved >48h). Action: reprioritize human review staffing.

Production Signals

Key Metrics

Alerts

References

• NIST-GenAI-Profile
• Note: Generative AI risks including confabulation, data privacy, information integrity, human-AI configuration, security, value chain.