A pager that goes off at three in the morning for something that could have waited until nine is not reliability. It is a slow way to lose your best engineers. Humane on-call is a design problem, not a rota problem.
Alert on symptoms, not causes
Page a human only when a user is actually affected and only they can fix it. Everything else is a ticket. Most alert fatigue comes from paging on internal noise that self-heals before anyone even reads it.
- Every page maps to a real user impact
- Every page has a runbook
- Anything without both becomes a dashboard, not an alarm
Protect the recovery, not just the response
Whoever gets woken up gets the next morning back. On-call is a cost the whole team shares, not a tax on one person, and the schedule has to reflect that in the open.
If the fix is always the same, the fix belongs in the code, not the runbook.
Feed incidents back into the build
Every incident is a message about the system. The blameless review that follows should end with a change to code or process, so the same page never wakes anyone twice.
