Home / Patterns / Confidence Miscalibration

Confidence Miscalibration

Goal Answer Synthesis Frequency Very Common Category Knowledge Retrieval Published View source on GitHub ↗

Issue: Model Confidence Doesn’t Match Answer Reliability

Frequency: Very Common

Symptoms

  • High confidence on wrong or poorly-grounded answers
  • Low confidence on well-supported answers
  • Uncertainty not expressed when context is ambiguous
  • All answers have similar confidence regardless of support

Root Cause LLMs are trained to produce fluent, confident text. They don’t naturally express calibrated uncertainty based on context support.

Example

Context: "The meeting might be rescheduled to either Tuesday or 
Wednesday, pending confirmation from the VP."

Query: "When is the meeting?"

Agent: "The meeting is on Tuesday." (stated definitively)

Reality: Day is uncertain, pending confirmation

Result: User misses meeting because they assumed Tuesday

Mitigation Strategies

  1. Uncertainty prompting: Ask model to express confidence
  2. Calibration training: Fine-tune for calibrated confidence
  3. Evidence strength signaling: “Strongly supported” vs. “mentioned briefly”
  4. Hedged language: Use “may”, “possibly”, “according to” appropriately
  5. Confidence scoring: Separate confidence assessment step
  6. Human-readable uncertainty: “I’m not certain, but…”

Detection

  • Calibration curves (confidence vs. accuracy)
  • Track confidence distributions by answer correctness
  • User feedback on overconfident wrong answers
  • Test with deliberately uncertain contexts

Mitigation Strategies

Prevention

  1. Implement query-answer consistency validation: Decompose complex queries into atomic components and verify each component is addressed in the answer before returning. Use RAGAS Answer Relevancy metric (target: >0.75) with automatic re-generation for scores below threshold. Root cause mitigation: Prevents context-anchoring by explicitly binding answer generation to parsed query intent.

  2. Apply multi-source consensus verification: Require answers synthesized from multiple sources to explicitly cite which sources support each claim and flag unresolved contradictions. Implementation: Use semantic similarity checks across source fragments to detect cherry-picked evidence patterns. Root cause: Ensures balanced representation of evidence when sources conflict.

  3. Enforce comprehensive coverage checks: Implement structured extraction requiring explicit treatment of caveats, limitations, exceptions, and counterevidence for each claim type. Use template responses with mandatory caveat sections. Root cause: Prevents omission of qualifying information that would change user decision-making.

Detection & Response

  1. Answer completeness monitoring: Measure coverage of query intents in generated answers. Track query decomposition rate (% of query components explicitly addressed) and flag responses with coverage <85%. Instrument RAG pipeline to log query-answer similarity scores per component. Alert on sustained scores <0.70.

  2. Evidence balance scoring: For each answer, compute evidence distribution across sources and flag one-sided responses (>70% from single source on multi-source queries). Implement automated extraction of caveat/limitation mentions and track inclusion rates by query type. Target: >80% of medical/financial answers include relevant caveats.

Architecture Patterns

  1. Query Intent Decomposition Graph: Parse complex queries into a DAG of atomic intents before retrieval. Each retrieved document is mapped to specific intent nodes. Answer generation must satisfy all leaf nodes. Validation layer computes coverage before response generation.

  2. Evidence Consensus Engine: Maintain a fact graph where each claim is attributed to specific sources with confidence scores. Multi-source claims require consensus computation (intersection of sources supporting claim). Flagging layer surfaces contradictions to generation model.

  3. Structured Response Templates: Use task-specific response schemas that enforce inclusion of: primary answer, supporting evidence, relevant caveats/exceptions, alternative interpretations, confidence bounds. Auto-flag template violations before user delivery.

Key Metrics

MetricTargetAlert ThresholdMeasurement Method
Answer Relevancy Score>0.75<0.70RAGAS metric on generated answers vs. original query
Query Coverage Rate>90%<85%Percentage of query components explicitly addressed in answer
Evidence Balance Index>0.6<0.4Distribution of citations across sources (Gini coefficient, 0=balanced, 1=single-source)
Caveat Inclusion Rate>80%<70%Percentage of medical/financial answers including relevant limitations
User Clarification Rate<5%>10%Percentage of answered queries requiring follow-up clarification

Alerts & Escalation

AlertConditionSeverityResponse
Low Answer RelevancyAnswer Relevancy Score < 0.70 for >5% of queries in 1-hour windowHIGHPage on-call; trigger re-generation with query reinforcement prompt
Single-Source DominanceEvidence Balance Index < 0.4 on multi-source queries for >3 consecutive queriesMEDIUMLog event; audit cherry-picking patterns in retrieval/synthesis
Rising Clarification DemandUser Clarification Rate exceeds 10% (vs. 5% baseline) over 24-hour windowHIGHInvestigate query decomposition or answer template effectiveness

Universal Pattern Reference

This is a domain-specific implementation of the universal pattern: Hallucination and Confidence Miscalibration (Cross-Cutting)

The universal pattern covers why LLMs produce confident but false content. This variant focuses on RAG/answer synthesis where confidence miscalibration prevents routing hallucinations to human review.

References

Related Patterns