StaffLoans

app

Incident Response Runbook

Step-by-step procedures for critical incidents

Emergency Escalation

On-Call Rotation

Slack: @on-call

PagerDuty: [link]

Critical Contacts

CTO: +1-555-XXXX (24/7)

Security: security@company.com

War Room

Zoom: [link] (always open)

Slack: #incidents (real-time)

Database Down

Owner: DBA Team

critical

Service Level Objective (SLO)

RTO: 4 hours | RPO: 1 hour

Incident Response Steps

1. Declare incident in Slack #incidents channel

2. Page on-call DBA immediately

3. Check database logs for errors

4. Attempt automatic failover to replica

5. If failover fails, initiate restore from latest backup

6. Notify stakeholders of ETA

7. Post-incident: RCA within 24 hours

Stakeholder Communication

Initial Alert (within 5 min):
"🚨 INCIDENT: Database Down
Severity: CRITICAL
Status: INVESTIGATING
Updates every 10 minutes"

Update (every 10 min):
"Root cause: [description]
Action: [mitigation in progress]
ETA to resolution: [time]"

Universal Incident Workflow

< 1 min

Detection

Alert fires, on-call paged

< 5 min

Response

Team assembled, incident declared

5-30 min

Investigation

Root cause identified

30-120 min

Mitigation

Fix implemented or rollback

120-240 min

Recovery

System restored to normal

< 24 hours

Post-Incident

RCA completed, lessons learned documented

Incident Commander Checklist

Declare severity level (P1/P2/P3)

Activate war room (Zoom + Slack channel)

Page appropriate team members via PagerDuty

Assign roles: Commander, Incident Lead, Communicator

Set 10-minute update cadence

Create ticket for RCA (due 24 hours)

Once resolved: Schedule post-mortem

Publish timeline and lessons learned