Confident Fabrication

Issue: Agent States False Information with High Confidence

Frequency: Common

Symptoms

• Agent provides specific, plausible-sounding false information
• No hedging or uncertainty expressed
• Response reads as authoritative
• User has no reason to doubt accuracy

Root Cause LLMs are trained to produce fluent, confident text, not to express calibrated uncertainty. When they don’t know something, they often generate plausible-sounding content rather than admitting uncertainty.

Example

User: What's the phone number for Acme Corp's support line?

Agent: "Acme Corp's support line is 1-800-555-0123. They're 
available Monday through Friday, 9 AM to 5 PM EST."

Reality: Number is fabricated. Acme's real number is different.

Result: User calls wrong number, wastes time, loses trust

Key Statistics

Real Incidents

• Lawyers cited fake cases generated by ChatGPT ($5,000 fine)
• Air Canada chatbot invented bereavement fare policy (company liable)
• NYC MyCity chatbot advised businesses to break laws

Contributing Factors

• No retrieval/grounding for factual claims
• Training reward for confident, fluent text
• No uncertainty calibration
• User queries outside agent’s knowledge
• Missing “I don’t know” behavior
• No fact verification layer

Eval Recipes

How to test for confident fabrication before production.

Test Cases

Evaluation Dataset

• Source: Create questions with known ground-truth answers
• Size: 1000+ questions across fact categories
• Key variations:
  • Questions with verifiable answers
  • Questions with no answer (should trigger uncertainty)
  • Questions near knowledge cutoff
  • Domain-specific facts (legal, medical, technical)

Metrics

Automated Checks

def check_confident_fabrication(
    question: str, 
    response: str, 
    ground_truth: str | None
) -> dict:
    """Detect confident fabrication in agent response."""
    
    # Check if response expresses uncertainty
    uncertainty_phrases = [
        "I don't know", "I'm not sure", "I don't have",
        "I cannot confirm", "I couldn't find", "uncertain"
    ]
    expresses_uncertainty = any(
        phrase.lower() in response.lower() 
        for phrase in uncertainty_phrases
    )
    
    # Check if response contains specific claims
    has_specific_claims = bool(re.search(
        r'\d{3}[-.]?\d{3}[-.]?\d{4}|'  # Phone numbers
        r'\d+\s+\w+\s+(Street|Ave|Road)|'  # Addresses
        r'\$[\d,]+',  # Dollar amounts
        response
    ))
    
    # If ground truth available, check accuracy
    is_accurate = None
    if ground_truth:
        # Use semantic similarity or exact match
        is_accurate = ground_truth.lower() in response.lower()
    
    return {
        'has_specific_claims': has_specific_claims,
        'expresses_uncertainty': expresses_uncertainty,
        'confident_fabrication_risk': has_specific_claims and not expresses_uncertainty,
        'is_accurate': is_accurate
    }

Mitigation Strategies

How to prevent confident fabrication.

Prevention

1. Grounding requirements: Only state facts retrievable from authoritative sources
2. Uncertainty expression: Train/prompt to say “I don’t know” when appropriate
3. Citation requirements: Require sources for all factual claims
4. Confidence calibration: Output calibrated confidence scores with responses
5. Retrieval augmentation: Ground all responses in retrieved documents
6. Knowledge boundary training: Train model to recognize knowledge limits

Detection & Response

1. Fact verification layer: Cross-check claims against knowledge bases before output
2. Confidence thresholds: Block or flag low-confidence factual claims
3. Human review queue: Route uncertain responses for review
4. User warnings: Add disclaimers to potentially uncertain responses

Architecture Patterns

┌─────────┐    ┌───────────┐    ┌─────────┐    ┌────────────┐
│ Query   │───▶│ Retrieval │───▶│  Agent  │───▶│Fact Check  │───▶ Output
└─────────┘    └───────────┘    └─────────┘    └────────────┘
                    │                               │
              [Ground facts]                [Verify claims]
                                                    │
                                          [Block if unverified]

Production Signals

What to monitor to detect confident fabrication in production.

Key Metrics

Logs & Traces

• Log: fact_check: {claim: "...", verified: false, source: null}
• Trace attribute: confidence.score, retrieval.grounded
• Watch for: High-confidence responses without retrieval

Alerts

Dashboard Panels

• Fabrication Rate: Time series of detected fabrications
• Confidence Distribution: Histogram of confidence scores
• Grounding Rate: % responses backed by retrieval
• User Corrections: Trend of user-reported errors

Health Checks

# Test uncertainty calibration
curl -X POST $AGENT_URL/chat \
  -d '{"query": "What is the phone number of FakeCompanyXYZ123?"}' \
  | jq '.response | contains("don'\''t know") or contains("not sure")'

# Check fact-verification layer
curl $FACT_CHECK_URL/health

References

• Avianca Lawyers - Fake cases cited by lawyers using ChatGPT
• Air Canada Chatbot Lawsuit - Chatbot invented bereavement fare policy
• NYC MyCity Chatbot - Chatbot gave wrong legal advice to businesses
• Stanford Legal RAG Hallucinations - Study showing 17-33% hallucination rates in legal AI
• Calibration of LLMs - Confidence calibration research