Performance Investigation Runbook

1. Identify Slow Operations

1
2
3
4
5
6
SELECT service_name, operation, avg(duration_ms) as avg_latency
FROM traces
WHERE _timestamp > now() - interval '1 hour'
GROUP BY service_name, operation
ORDER BY avg_latency DESC
LIMIT 20

2. Analyze Latency Distribution

• Check P50, P95, P99 in histograms
• Look for bimodal distributions

3. Trace Slow Requests

• Filter traces by latency > threshold
• Identify bottleneck spans

4. Root Cause

• Database queries
• External API calls
• Resource contention

5. Optimize

• Add caching
• Optimize queries
• Scale resources