Response Time SLA Breach

Issue: System Exceeds Latency Commitments

Frequency: Common

Symptoms

• User-facing latency exceeds promised SLA
• Customer complaints about slow responses
• Timeout errors in downstream systems
• Degraded user experience scores

Root Cause System design doesn’t account for real-world latency distribution. P50 may be acceptable but P99 exceeds SLA. Load spikes, model complexity, or infrastructure issues push response times beyond commitments.

Example

SLA: 95% of requests under 2 seconds

Actual performance:
- P50: 800ms ✓
- P90: 1.8s ✓
- P95: 3.2s ✗ (SLA breach)
- P99: 8.5s ✗✗

Root causes:
- Complex queries hit P99: long context, multi-tool
- Peak hours: 3x normal latency
- Cold starts after scaling events

Contributing Factors

• No latency budgeting across components
• Testing only happy path scenarios
• Not accounting for tail latency
• Missing load testing at scale
• Infrastructure not right-sized

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Latency budgeting: Allocate time budget per component
2. Tail latency design: Design for P99, not P50
3. Load testing: Regular tests at 2-3x expected load
4. Graceful degradation: Fallback to faster, simpler responses
5. Right-sizing: Match infrastructure to latency requirements

Detection

• Real-time percentile monitoring
• SLA compliance dashboards
• Anomaly detection on latency distribution

Recovery

• Auto-scaling triggers
• Circuit breakers for slow dependencies
• Request shedding under extreme load

Production Signals

Key Metrics

Alerts

References

• Google SRE: Latency SLOs
• Tail Latency at Scale