Network Latency Blindness

Issue: System Doesn’t Account for Network Round-Trip Time

Frequency: Occasional

Symptoms

• Latency much higher in production than local testing
• Cross-region deployments have poor performance
• API calls slower than expected
• Timeout calculations wrong

Root Cause Development and testing often ignore network latency. Production deployments span regions, cross network boundaries, and involve multiple hops. Each hop adds 10-200ms that isn’t accounted for in design.

Example

Local testing: LLM API call = 500ms

Production reality:
- Client to edge: 20ms
- Edge to app server: 15ms
- App to LLM API (cross-region): 80ms
- LLM processing: 500ms
- LLM to app: 80ms
- App to edge: 15ms
- Edge to client: 20ms
Total: 730ms (46% overhead)

Multi-tool scenario (3 API calls):
Local: 1500ms
Production: 1500 + (3 × 160) = 1980ms (32% overhead)

Contributing Factors

• Testing only in local/same-region
• Not measuring network latency separately
• Cross-region API calls
• No CDN/edge deployment
• DNS resolution time ignored

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Measure network separately: Instrument network time
2. Co-locate services: Minimize network hops
3. Edge deployment: Process closer to users
4. Connection pooling: Reuse connections
5. DNS caching: Avoid repeated lookups

Network-Aware Architecture

class NetworkAwareClient:
    def __init__(self):
        self.latency_by_region = {}
        self.connection_pool = {}
    
    def call_api(self, endpoint, region=None):
        start = time.time()
        
        # Reuse connection
        conn = self.get_pooled_connection(endpoint)
        
        # Measure network vs processing
        network_start = time.time()
        response = conn.request(endpoint)
        network_time = time.time() - network_start
        
        total_time = time.time() - start
        processing_time = total_time - network_time
        
        # Track for routing decisions
        self.latency_by_region[region] = network_time
        
        return response, {
            'network_ms': network_time * 1000,
            'processing_ms': processing_time * 1000,
        }

Production Signals

Key Metrics

Alerts

References

• Global Latency Considerations
• Network Performance Monitoring