Cost Anomaly Blindness

Issue: Unusual Cost Patterns Go Undetected

Frequency: Common

Symptoms

• Runaway costs discovered only at month-end
• Gradual cost creep unnoticed
• Sudden spikes not alerted
• No baseline for “normal” costs
• Anomalies buried in aggregate metrics

Root Cause Without anomaly detection, cost problems are only discovered when invoices arrive or budgets exhaust. Gradual increases (prompt bloat, new features) go unnoticed. Sudden spikes (loops, attacks) aren’t caught in real-time. Organizations need both baseline tracking and anomaly detection.

Example

Cost timeline (undetected):

Week 1: $500/day (baseline)
Week 2: $520/day (prompt got longer - unnoticed)
Week 3: $580/day (new feature - unnoticed)
Week 4: $4,200/day (agent loop - UNNOTICED until Friday)

Total unexpected spend: $25,000

With anomaly detection:
Week 2: Alert - 4% above baseline (investigate)
Week 3: Alert - 16% above baseline (confirm new feature)
Week 4: Alert - 840% above baseline (STOP IMMEDIATELY)

Anomaly detection would have caught the loop on Day 1

Contributing Factors

• No cost baseline established
• Aggregate-only metrics
• Alerting only on absolute thresholds
• No velocity/rate-of-change tracking
• Manual review only
• Alerts go to unmonitored channels

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Establish baselines: Track normal cost patterns
2. Rate-of-change alerts: Alert on velocity, not just totals
3. Statistical detection: Use stddev-based thresholds
4. Per-user/agent baselines: Granular anomaly detection
5. Real-time monitoring: Sub-minute cost tracking
6. Actionable alerts: Route to on-call, not email

Detection Methods

Simple: Alert if cost > 2x daily average
Better: Alert if cost > baseline + 3*stddev
Best:   ML-based anomaly detection with seasonality

Production Signals

Key Metrics

Alerts

References

• Arize: Cost Monitoring
• MindStudio: Token Budget Management