Incident Response Runbook

1. Triage

• Check service health dashboards
• Identify affected domain/concept
• Get correlation_id from reported issue

2. Investigate

1
2
3
4
5
6
SELECT * FROM logs 
WHERE sea_domain = 'affected-domain'
  AND level = 'error'
  AND _timestamp > now() - interval '1 hour'
ORDER BY _timestamp DESC
LIMIT 100

3. Trace Analysis

• Find traces with errors
• Analyze span waterfall
• Identify root cause service

4. Mitigate

• Apply fix or rollback
• Monitor recovery metrics

5. Post-Mortem

• Document timeline
• Update runbooks
• Create action items