Workbench BFF Incidents

Symptoms

• Auth failures (401/403)
• API timeouts
• High error rate
• UI not loading

Immediate Mitigation

1
2
3
4
5
6
7
8
# Check BFF health
curl http://localhost:8000/health

# Check auth provider status
curl http://localhost:8000/auth/status

# Restart BFF
just restart workbench-bff

Verification Steps

1. Health endpoint OK
2. Auth working (test with valid token)
3. API responding
4. Error rate <1%

Rollback

1
2
3
# Revert last deployment
git revert <commit>
just deploy workbench-bff

Escalation

• P1: Auth completely broken → Check OIDC provider
• P2: Partial degradation → Check BFF logs