Object Hallucination

Issue: Vision Model Detects Objects Not Present in Image

Frequency: Occasional

Symptoms

• Model confidently identifies objects that aren’t in the image
• Agent takes action based on hallucinated object (e.g., tries to pick up non-existent item)
• False detections more common in low-resolution or cluttered images
• Hallucination rate increases with model temperature/sampling

Root Cause Vision models are trained on diverse visual data and learn statistical associations between visual features. When presented with ambiguous or low-confidence regions, the model’s language prior about “what objects typically appear in this context” overrides what’s actually visible. Salient features (bright colors, edges) can trigger false object predictions, and the model defaults to high-confidence predictions even when visual evidence is weak. This is compounded by imbalanced training data where certain objects are overrepresented.

Example

Scenario: Warehouse inventory agent using vision model to count items

Image: Shelf with mostly empty space, one blue box, and some packaging debris

Model output: 
- Blue box (correct)
- Red box (hallucinated - debris interpreted as red box)
- Green bottle (hallucinated - lighting artifact)
- Plastic bag (correct)

Agent action: Attempts to pick up green bottle → gripper hits empty space → collision detected

Impact: Inventory count incorrect (3 items vs. 2), robot malfunction from unexpected collision

Key Statistics From Vision Hallucination Survey (arXiv 2404.18930):

• 15-25% of vision models produce hallucinated objects in cluttered scenes
• Hallucination rate higher in low-resolution images (below 256×256)
• Temperature/sampling increases hallucination: 0.5°C = 5% false positives, 1.0°C = 18% false positives
• Rare objects (bottom 10% of training frequency) hallucinated 40% more often than common objects

Contributing Factors

• Training data imbalance (certain objects overrepresented)
• Low image resolution or compression artifacts
• Cluttered backgrounds that create ambiguous features
• High model temperature or sampling diversity
• Context priming (if model expects certain objects in a scene type)
• Partial occlusion creating ambiguous shapes

Eval Recipes

Test Cases

Evaluation Dataset

• Source: COCO dataset subset (cluttered scenes) + synthetic noise images + low-resolution variants
• Size: 1,000 images (500 with ground truth labels, 500 deliberately empty/noisy)
• Key variations:
  • Resolution: 64×64, 128×128, 256×256, 512×512
  • Scene complexity: empty, simple (1-2 objects), cluttered (5+ objects)
  • Object types: common (chair, person), rare (telescope, microscope)

Metrics

Automated Checks

def evaluate_hallucination(model, test_images, gt_annotations):
    """
    Detect hallucination patterns in vision model outputs.
    """
    fps = []  # false positives per image
    confidences = []
    
    for img, gt_objects in zip(test_images, gt_annotations):
        detections = model.detect(img, confidence_threshold=0.3)
        
        # False positives: detections not in ground truth
        false_pos = [d for d in detections 
                     if d.class_id not in gt_objects]
        fps.append(len(false_pos) / max(len(detections), 1))
        
        # Confidence stats for hallucinated objects
        for fp in false_pos:
            confidences.append(fp.confidence)
    
    # Hallucination metrics
    fpr = sum(fps) / len(fps)  # False positive rate
    mean_halluc_confidence = mean(confidences)
    
    # Flag issues
    assert fpr < 0.05, f"FPR too high: {fpr:.2%}"
    assert mean_halluc_confidence < 0.5, \
        f"Hallucinations too confident: {mean_halluc_confidence:.2f}"
    
    return {
        "false_positive_rate": fpr,
        "hallucination_confidence": mean_halluc_confidence,
        "passed": fpr < 0.05 and mean_halluc_confidence < 0.5
    }

Mitigation Strategies

Prevention

1. Confidence Thresholding: Only accept detections with >70% confidence in production; monitor calibration curve (actual accuracy vs. predicted confidence)
2. Ground Truth Cross-Check: Run object detection twice with different model checkpoints; only accept objects detected by both models
3. Negative Sampling in Training: Include deliberately empty/noisy images in training data to teach model to reject ambiguous detections
4. Resolution Preprocessing: Ensure minimum input resolution (256×256); upscale smaller images instead of downscaling
5. Semantic Consistency: Validate detected objects make sense in context (e.g., “watermelon on ceiling” is implausible; flag for review)

Detection & Response

1. Confidence Monitoring: Log all detections with confidence < 60%; alert if hallucination rate spikes above 10%
2. Disagreement Detection: Compare outputs across model ensembles; disagreement indicates uncertainty
3. Action Validation: Before gripper/robot action, verify object with second sensor (e.g., depth camera confirms object exists)
4. Post-Action Feedback: Log gripper failures (attempted pickup with empty result) as hallucination incidents

Architecture Patterns

1. Human-in-the-Loop: Route all low-confidence (<60%) detections to human review before action
2. Conservative Thresholding: Start with high threshold (>80%) in production; gradually lower as model proves reliable
3. Ensemble Voting: Use 3+ vision models; require agreement (2+ models) before accepting detection
4. Degradation Mode: If hallucination rate exceeds threshold, fall back to supervised labeling instead of autonomous detection

Production Signals

Key Metrics

Logs & Traces

• Log all detections with confidence <60% in separate “uncertain_detections” stream
• Include image metadata: resolution, lighting, scene type
• Trace gripper contact state post-grasp: successful grip vs. empty grasp
• Include model checkpoint version in detection logs (for rollback correlation)

Alerts

Dashboard Panels

• Panel 1: False positive rate over time (24h rolling window)
• Panel 2: Confidence distribution for hallucinated vs. correct detections (histogram)
• Panel 3: Detection rate by scene complexity (empty, simple, cluttered)
• Panel 4: Gripper grasp success rate by detection confidence bins
• Panel 5: Model ensemble agreement matrix (heatmap of pairwise disagreement)

Health Checks

-- Daily hallucination audit
SELECT 
  DATE(timestamp) as date,
  COUNT(*) as total_detections,
  SUM(CASE WHEN is_hallucinated THEN 1 ELSE 0 END) as hallucinated_count,
  AVG(CASE WHEN is_hallucinated THEN confidence ELSE NULL END) as halluc_avg_confidence,
  SUM(CASE WHEN is_hallucinated AND confidence > 0.7 THEN 1 ELSE 0 END) as high_conf_halluc
FROM vision.detections
WHERE timestamp > NOW() - INTERVAL 1 DAY
GROUP BY DATE(timestamp)
HAVING hallucinated_count / total_detections > 0.1
  THEN RAISE ALERT "Hallucination rate >10%"

Universal Pattern Reference

This is a domain-specific implementation of the universal pattern: Hallucination: Objects (Cross-Cutting)

The universal pattern covers why models hallucinate objects/fields. This variant focuses on vision-based tasks where models confidently detect objects that don’t exist in the image (especially in cluttered or empty scenes).

Related Domain Variants

• Document Processing: Object Hallucination — Hallucinated fields in document extraction

Related Base Pattern

• Hallucination: Base Mechanism — Universal root cause of all hallucinations

References

• Hallucination of Multimodal LLMs Survey - Comprehensive taxonomy of object, attribute, and spatial hallucinations in vision models
• Mitigating OCR Hallucinations in MLLMs - Phantom element detection techniques
• Evaluating Multimodal LLMs for Production - Grounding validation and confidence calibration
• Confidence Calibration in Vision Models - Analysis of confidence-accuracy gaps in vision transformers