An incident runbook is a short, pre-agreed document that answers three questions under pressure: how bad is this, who is doing what, and what do we do in the next fifteen minutes. If it takes longer than a minute to find or read, nobody will use it.
Below is the template we give small engineering teams. Copy it into your wiki or repo, replace the bracketed values, and link it from every alert. After that, we cover the two things that decide whether it works in practice: a sustainable on-call rotation and alerts that only page for things users can actually feel.
The runbook template
Paste this as-is and edit it. The numbers in it (response times, update intervals) are sensible starting points, not standards; change them to match what you promise customers.
INCIDENT RESPONSE RUNBOOK - [Company]
Owner: [name] Last reviewed: [date]
Incident channel prefix: #inc- Status page: [URL]
1. SEVERITY LEVELS (if unsure, pick the higher one; downgrade later)
SEV1 Product down or unusable for most customers, data loss,
or confirmed security breach. Page 24/7. All hands.
SEV2 Core feature broken or badly degraded for many customers,
no workaround. Page 24/7.
SEV3 Partial or minor impact, workaround exists, single customer,
or internal tooling. Business hours only.
2. ROLES
Incident commander (IC) Owns the incident. Coordinates, decides,
keeps time. Does not debug.
Responder(s) Investigate and fix. Report to the IC.
Comms lead Status page, customer emails, support team,
management. Uses templates in section 4.
Scribe Timeline in the incident channel (UTC).
Small team rule: for SEV3 one person can hold every role. For SEV1
the IC is never the person typing commands into production.
3. FIRST 15 MINUTES
[ ] Acknowledge the page.
[ ] Open #inc-[yyyymmdd]-[short-name], post the alert and this runbook.
[ ] Declare: "I am IC. This is SEV[n]. Impact so far: [...]"
[ ] What changed? Last deploys, feature flags, config, Terraform
applies, cron jobs, AWS Health Dashboard, third-party status pages.
[ ] How big is it? Error rate, latency, traffic dashboards, support inbox.
[ ] Decide: roll back / mitigate / keep investigating (section 6).
[ ] SEV1-2: assign a comms lead, first status page update.
[ ] Security suspected? Preserve evidence (snapshot, don't terminate),
revoke exposed credentials, go to section 8.
4. COMMUNICATION
Internal update, every 30 min (SEV1) or 60 min (SEV2):
Status: investigating | identified | monitoring | resolved
Impact: [who and what is affected]
Actions: [what we are doing now]
Next update: [HH:MM UTC]
Customer update - investigating:
"We are investigating [elevated errors / slow responses] affecting
[feature]. Some customers may be unable to [action]. We will post
an update by [HH:MM UTC]."
Customer update - identified:
"We have identified the cause of [issue] and are [rolling back a
recent change / applying a fix]. [Feature] may remain [degraded]
until then. Next update by [HH:MM UTC]."
Customer update - resolved:
"[Feature] has been working normally since [HH:MM UTC]. Between
[start] and [end], [describe impact]. We are reviewing what
happened and will share a summary by [date]."
5. ESCALATION
No acknowledgement in [5] min -> page secondary
No acknowledgement in [15] min -> call [engineering lead], [phone]
Escalate to [CTO / founder] when: SEV1, suspected data loss or breach,
contractual SLA at risk, or no mitigation after [60] min.
Vendors: AWS Support (case severity per your support plan),
[payment provider], [DNS / CDN], [other] - contacts: [link]
6. ROLLBACK DECISION
Roll back first, debug later, when:
- the incident started within [2] hours of a deploy, flag or
config change, AND
- the rollback is known to be safe (no irreversible migration).
Do not roll back blindly when:
- a database migration is not backward compatible -> forward fix
or restore plan, IC decides
- the cause is external (AWS, a vendor) -> mitigate: fail over,
disable the feature, degrade gracefully
Time-box investigation to [15] min before choosing a mitigation.
7. RESOLUTION
[ ] Impact gone and stable for [30] min.
[ ] Final internal and customer updates sent.
[ ] Post-incident review created, owner assigned, due in [5]
working days.
[ ] Summary pinned in the channel, channel archived.
8. SECURITY AND REGULATORY
[ ] Personal data affected? -> [DPO]. GDPR: notify the supervisory
authority within 72 hours of becoming aware, where feasible.
[ ] In scope for NIS2 and the incident may be significant?
-> [named person] decides. Early warning within 24 hours,
incident notification within 72 hours, final report within
one month.
[ ] Customer contracts with notification clauses? -> [owner]
9. CONTACTS
Role Name Phone Backup
Engineering lead [...] [...] [...]
CTO / founder [...] [...] [...]
DPO / legal [...] [...] [...]Severity levels: decide on impact, not on cause
Severity should describe what customers experience, so anyone on the team can pick it in seconds without knowing the root cause. Three levels are enough for a small team; a fourth level usually becomes a place to hide things.
Severity drives three things: whether someone is woken up, how often you communicate, and who gets told. Keep it that simple. If an incident needs a debate about severity, pick the higher one and move on.
Roles: why the incident commander doesn't debug
The incident commander coordinates and decides; the responders fix. Separating the two is the single most effective habit we see in small teams, because the person deep in a stack trace cannot also track time, notice that nobody has updated the status page, or call for help.
The roles follow the incident command model popularised by PagerDuty's open incident response documentation, which is worth reading in full. In a team of five, the IC is often the engineering lead or whoever is on secondary. The roles are hats, not job titles: the same person can be scribe one week and IC the next.
The rollback rule
If something broke shortly after a change and the rollback is safe, roll back first and investigate afterwards. Most production incidents in small teams follow a deploy, a config change or a feature flag, and restoring service is worth more than understanding the bug ten minutes sooner.
That only works if rollback is boring. Immutable image tags, one-click redeploys of the previous version and backward-compatible migrations are prerequisites; we wrote about the first one in stop shipping the latest tag to Kubernetes and about pipeline habits that make rollback hard in CI/CD anti-patterns.
Post-incident review template (blameless)
A blameless review asks what in the system allowed the failure, not who caused it. People who expect blame hide details, and the details are what you need to stop it happening again. Hold the review within a week, while memories are fresh.
POST-INCIDENT REVIEW - [title]
Date: [yyyy-mm-dd] Severity: SEV[n] IC: [name] Author: [name]
Status: draft | reviewed | actions complete
SUMMARY (3 sentences)
What happened, who was affected, how it was resolved.
IMPACT
Duration (start -> detected -> mitigated -> resolved, UTC):
Customers / requests affected:
Data loss or exposure: yes / no (details)
SLA or regulatory implications:
TIMELINE (UTC)
[hh:mm] [event - from the incident channel]
CONTRIBUTING FACTORS
Technical: [what failed and why it was possible]
Detection: [how we found out; could we have known sooner?]
Process: [what slowed the response]
(Describe systems and decisions, never individuals' mistakes.)
WHAT WENT WELL
[...]
WHERE WE GOT LUCKY
[...]
ACTION ITEMS
Action Owner Ticket Due
[prevent recurrence] [...] [...] [...]
[detect faster] [...] [...] [...]
[respond faster / runbook fix] [...] [...] [...]Two rules make reviews useful. Every action item gets an owner and a ticket, or it doesn't go in. And "be more careful" is never an action item; if a human could make the mistake, the fix is a guardrail, a test or an alert.
On-call for a 3 to 10 person team
With a small team, sustainable on-call comes down to two levers: share the load evenly, and keep the number of out-of-hours pages low. Rotation design helps with the first; alert quality is the only fix for the second.
- Weekly rotation, handover mid-week. A Tuesday or Wednesday handover means nobody starts their shift on a Friday evening, and the handover happens when everyone is around.
- With 3 or 4 engineers: one primary, and the engineering lead or CTO as the escalation point. Page out of hours only for SEV1 and SEV2.
- From about 5 engineers: add a secondary who takes the page if the primary doesn't acknowledge, and who becomes IC for bigger incidents.
- Compensate it. Pay an on-call allowance or give time off, and after a night page the on-call engineer starts late the next day. Check what local employment law and contracts require; in Greece and much of the EU, standby and overtime rules apply.
- Shadow before solo. New engineers shadow one rotation, with the primary, before they carry the pager alone.
- Every alert links to a runbook entry. The person paged should never have to guess what an alert means.
- Review pages weekly. Go through every page from the past week: was it real, was it actionable, did it need a human at that hour? Delete, demote or fix anything that fails.
Treat out-of-hours pages as a metric. If one person had a rough week, the next sprint should include fixing the causes, not just thanking them. Google's SRE book chapter on being on-call goes deeper on balancing load and quality.
Alerting that pages only on user-visible symptoms
Page on symptoms users feel, like errors, latency and failed key journeys, and send everything else to a ticket or a dashboard. High CPU on one node at 3am is not worth waking anyone if users aren't affected; a rising error rate is, whatever the cause.
This is the approach in the monitoring chapter of Google's SRE book: monitor the four golden signals (latency, traffic, errors, saturation) and keep pages for problems that are urgent, user-visible and actionable. In practice for a typical AWS web service:
- Page: 5xx error ratio above threshold for several minutes, p99 latency above threshold, synthetic check of login or checkout failing from more than one location, queue age growing past what users would notice.
- Ticket, not page: disk filling slowly, certificate expiring in two weeks, a single instance unhealthy behind a load balancer, high CPU without user impact.
Here is a CloudWatch alarm on the ALB 5xx ratio rather than raw error counts, so it scales with traffic and ignores quiet periods. Thresholds are examples; tune them against your own baseline.
resource "aws_cloudwatch_metric_alarm" "api_5xx_ratio" {
alarm_name = "api-5xx-ratio-high"
alarm_description = "SEV2: API 5xx ratio above 2%. Runbook: [wiki link]"
comparison_operator = "GreaterThanThreshold"
threshold = 2
evaluation_periods = 5
datapoints_to_alarm = 3
treat_missing_data = "notBreaching"
alarm_actions = [aws_sns_topic.pager.arn]
ok_actions = [aws_sns_topic.pager.arn]
metric_query {
id = "ratio"
expression = "IF(requests > 50, 100 * FILL(errors, 0) / requests, 0)"
label = "5xx %"
return_data = true
}
metric_query {
id = "errors"
metric {
namespace = "AWS/ApplicationELB"
metric_name = "HTTPCode_Target_5XX_Count"
period = 60
stat = "Sum"
dimensions = { LoadBalancer = aws_lb.api.arn_suffix }
}
}
metric_query {
id = "requests"
metric {
namespace = "AWS/ApplicationELB"
metric_name = "RequestCount"
period = 60
stat = "Sum"
dimensions = { LoadBalancer = aws_lb.api.arn_suffix }
}
}
}Route the SNS topic to whatever paging tool you use (PagerDuty, incident.io and similar tools all accept it). Put the runbook link in the alarm description so it arrives with the page.
NIS2 and incident reporting
If your company is in scope for NIS2, a significant incident triggers reporting deadlines: an early warning within 24 hours, an incident notification within 72 hours and a final report within one month, as set out in Article 23 of the directive. Section 8 of the template exists so that a named person makes that call during the incident, not days later. Our NIS2 checklist for teams on AWS covers what that report needs and how Greek Law 5160/2024 applies it. The same incident log and reviews double as evidence for a SOC 2 audit.
Make it real with a drill
A runbook nobody has used is a draft. Once a quarter, run a 30-minute game day: someone breaks something in staging, the on-call engineer follows the runbook, and the team fixes whatever was confusing. It is the cheapest way to find out that the status page password lives in a former colleague's head.
If your team is too small to carry a pager on top of shipping features, or the alerts are so noisy that nobody trusts them, that is exactly the kind of work our managed DevOps service takes on: cleaning up alerting, writing the runbooks and sharing the on-call load.
Frequently asked questions
What should an incident response runbook include?
At minimum: severity levels defined by customer impact, roles (incident commander, responders, comms lead, scribe), a first-15-minutes checklist, communication templates for internal and customer updates, escalation paths with contacts, rules for when to roll back, and a post-incident review template. For regulated companies, add a step for GDPR and NIS2 reporting decisions.
How do you set up on-call for a small team?
Use a weekly rotation with a mid-week handover, one primary and, from about five engineers, a secondary. Page out of hours only for serious, user-facing incidents. Compensate on-call with pay or time off, let people start late after a night page, and review every page weekly to remove noisy or non-actionable alerts.
What should trigger a page versus a ticket?
Page only for user-visible, urgent and actionable symptoms: rising error rates, high latency, failing synthetic checks on key journeys. Causes without user impact, such as high CPU on one node, slowly filling disks or a certificate expiring next week, should create a ticket or show on a dashboard instead of waking someone up.
What is a blameless postmortem?
A blameless postmortem, or post-incident review, examines what in the system and process allowed an incident to happen, rather than who made a mistake. It records the timeline, impact, contributing factors and action items with owners. Removing blame makes people share the details that actually prevent the incident from happening again.