SRE On-Call Escalation and Runbook Design

Question 1

What is the 'primary and secondary on-call' model, and what is the role of the secondary on-call engineer?

Accepted Answer

The primary on-call responds first to all alerts; the secondary serves as backup if the primary cannot be reached within a defined timeout, and may assist on high-severity incidents requiring multiple responders

Answer

The secondary on-call monitors all alerts and decides which ones to forward to the primary, acting as a filter

Answer

The secondary on-call handles alerts during business hours while the primary handles after-hours alerts only

Answer

The secondary on-call is responsible only for change management approvals, not incident response

Question 2

What information should ALWAYS be included in a runbook's 'prerequisites' section?

Accepted Answer

Required access (AWS account, Kubernetes cluster, database), credentials locations, tools to install, and any context needed before the first step (e.g., which monitoring dashboard to check)

Answer

The author's name, creation date, and last review date for version control purposes

Answer

A glossary of all technical terms used in the runbook for team members unfamiliar with the system

Answer

A list of all services that depend on the service described in the runbook

Question 3

An on-call engineer receives a P1 alert at 2 AM but cannot determine the cause after 20 minutes of investigation. What is the CORRECT action?

Accepted Answer

Escalate to the secondary on-call or the designated expert as defined in the escalation policy, without waiting longer — unresolved P1 incidents require additional resources

Answer

Continue investigating independently for another hour before escalating to avoid waking others unnecessarily

Answer

Resolve the alert as 'no issue found' and create a low-priority ticket for investigation during business hours

Answer

Attempt to mitigate by restarting all services in the affected stack, then monitor to see if the alert clears

Question 4

What is the 'severity matrix' in incident management, and why must it be defined BEFORE incidents occur?

Accepted Answer

A severity matrix defines criteria for each severity level (P1-P4) based on user impact, service criticality, and business risk — predefined so engineers make consistent, non-emotional severity decisions during high-pressure incidents

Answer

A severity matrix is a post-incident tool that retrospectively rates the severity of past incidents for trend analysis

Answer

A severity matrix is a tool for estimating the engineering effort required to fix a known issue

Answer

A severity matrix defines how many on-call engineers should be on the rotation based on team size

Question 5

What is 'on-call shadowing' and why is it a recommended practice for new team members?

Accepted Answer

On-call shadowing pairs new team members with experienced on-call engineers, allowing them to observe incident response and use runbooks in real situations before taking primary on-call responsibility

Answer

On-call shadowing is a security audit practice where senior engineers secretly monitor on-call engineers' actions

Answer

On-call shadowing requires new team members to work back-to-back on-call shifts to rapidly build experience

Answer

On-call shadowing means the on-call engineer should always have a senior engineer available on video call during their shift

Question 6

Which metric is MOST useful for identifying that the on-call rotation is unsustainable for the team?

Accepted Answer

Number of pages per on-call shift exceeding the team's policy limit (e.g., more than 5 pages per night shift), indicating that on-call load is too high to allow adequate rest

Answer

The percentage of incidents that require escalation to secondary on-call

Answer

The average time from alert to acknowledgment across all incidents

Answer

The ratio of false positive alerts to true positive alerts