SRE Alerting Strategy and Noise Reduction 1 — Questions and Answers
Question 1: What is 'alert fatigue' in the context of SRE?
- A state where on-call engineers become desensitized to alerts due to excessive or low-quality notifications (Correct answer)
- A monitoring system failure caused by too many simultaneous metrics being tracked
- The gradual degradation of alert threshold accuracy over time as services scale
- An architectural pattern where alerts are batched to reduce notification frequency
Correct answer: A state where on-call engineers become desensitized to alerts due to excessive or low-quality notifications
Alert fatigue occurs when engineers receive so many alerts—especially false positives—that they begin ignoring or dismissing them, increasing the risk of missing real incidents.
Question 2: Which alerting philosophy does Google SRE recommend as a best practice for page-worthy alerts?
- Alert on causes so engineers understand the root issue immediately
- Alert on symptoms that directly indicate user-visible impact (Correct answer)
- Alert on any metric that deviates more than two standard deviations from baseline
- Alert on resource saturation before symptoms appear to enable proactive response
Correct answer: Alert on symptoms that directly indicate user-visible impact
SRE best practice is to alert on symptoms (user-visible impact) rather than causes, since causes can be diagnosed after the alert fires and symptom-based alerts have lower false-positive rates.
Question 3: What is a 'dead man's switch' (watchdog) alert in SRE monitoring?
- An alert that fires when a critical service exceeds its error budget
- An alert that fires when it STOPS receiving expected periodic signals from a system (Correct answer)
- An automated rollback trigger activated when a deployment health check fails
- An alert generated when a primary monitoring system loses connection to a secondary
Correct answer: An alert that fires when it STOPS receiving expected periodic signals from a system
A dead man's switch alert fires when it stops receiving a regular heartbeat signal, ensuring that alerting system failures or complete service outages are detected even when no explicit error is generated.
Question 4: In SRE alerting, what does 'precision' of an alert measure?
- The percentage of real incidents that triggered at least one alert
- The average time between when an alert fires and when an engineer acknowledges it
- The proportion of alert firings that correspond to actual incidents requiring action (Correct answer)
- The accuracy of the alert threshold relative to the SLO violation boundary
Correct answer: The proportion of alert firings that correspond to actual incidents requiring action
Alert precision measures the fraction of alert firings that are true positives (real incidents), with low precision indicating many false positives that contribute to alert noise.
Question 5: What is the primary distinction between a 'page' alert and a 'ticket' alert in SRE practice?
- Page alerts go to the primary on-call engineer; ticket alerts go to the team lead
- Page alerts require immediate human action; ticket alerts can be addressed during business hours (Correct answer)
- Page alerts are triggered by SLO violations; ticket alerts are triggered by infrastructure metrics
- Page alerts use push notifications; ticket alerts use email-based delivery systems
Correct answer: Page alerts require immediate human action; ticket alerts can be addressed during business hours
Page alerts indicate situations requiring immediate human response (urgent, action-required now), while ticket alerts represent issues that can wait for the next business day without significant user impact.
Question 6: What is 'alert flapping' and why is it problematic?
- When an alert fires simultaneously across multiple monitoring regions, creating duplicate notifications
- When an alert repeatedly transitions between firing and resolved states in a short time window (Correct answer)
- When alert thresholds are automatically adjusted by ML systems without human approval
- When an alert's notification channel is misconfigured and messages fail to deliver
Correct answer: When an alert repeatedly transitions between firing and resolved states in a short time window
Alert flapping occurs when a metric oscillates around a threshold, causing rapid on/off transitions that generate excessive notifications without indicating a stable incident requiring intervention.
Question 7: Which of the following is a key indicator that an alert should be removed or tuned down in severity?
- The alert fires more than once per quarter during business hours
- The alert consistently fires and resolves without any engineer taking action (Correct answer)
- The alert requires more than 15 minutes of investigation before resolution
- The alert fires during deployments but not during steady-state operations
Correct answer: The alert consistently fires and resolves without any engineer taking action
If an alert regularly fires and auto-resolves without requiring engineer action, it is a false positive that adds noise without value and should be tuned or removed to reduce alert fatigue.
SRE Alerting Strategy and Noise Reduction 1
What is 'alert fatigue' in the context of SRE?