SRE Performance Testing and Load Management

Question 1

What is the difference between a 'load test' and a 'stress test,' and when should each be used?

Accepted Answer

A load test validates performance under expected production traffic levels to verify SLOs are met; a stress test pushes beyond expected levels to find breaking points and understand failure modes

Answer

A load test runs for a short duration (minutes); a stress test runs for a long duration (hours or days) to detect memory leaks and gradual degradation

Answer

A load test uses synthetic traffic; a stress test uses replayed production traffic to simulate realistic user behavior

Answer

A load test applies constant traffic; a stress test ramps traffic exponentially to simulate viral growth scenarios

Question 2

What is 'throughput' vs. 'latency' in performance testing, and what is their typical trade-off relationship?

Accepted Answer

Throughput is the number of requests processed per second; latency is the time to process a single request. At low load, both can be optimized simultaneously, but at high load, higher throughput often comes with increased latency as queuing occurs

Answer

Throughput and latency are inversely proportional at all load levels — any increase in throughput always causes a proportional increase in latency

Answer

Throughput measures front-end performance; latency measures back-end database performance — they are independent metrics

Answer

Throughput and latency are the same metric expressed in different units — higher throughput always means lower latency

Question 3

What is a 'p99 latency' measurement, and why do SREs prefer it over average (mean) latency for SLO monitoring?

Accepted Answer

p99 is the 99th percentile latency — 99% of requests complete faster than this value; it captures the experience of the worst-served 1% of users, which average latency hides by diluting outliers in the mean

Answer

p99 latency is calculated as 99% of the maximum observed latency, providing a conservative upper bound for capacity planning

Answer

p99 latency measures latency for premium (99th tier) users who receive priority service, reflecting the best-case performance

Answer

p99 latency is the average latency measured over 99% of uptime, excluding outlier periods during incidents

Question 4

What is 'autoscaling,' and what metric is MOST appropriate to trigger autoscaling for a web API service?

Accepted Answer

Autoscaling adjusts the number of running instances based on demand; for a web API, scaling based on requests per second (RPS) or concurrent connections is more appropriate than CPU utilization alone, as some APIs are I/O-bound rather than CPU-bound

Answer

Autoscaling should always use CPU utilization as the trigger because it is the most accurate proxy for user-facing load across all service types

Answer

Autoscaling based on memory utilization is most appropriate because memory is the limiting resource for most web applications

Answer

Autoscaling is only appropriate for stateless services; stateful services must be manually scaled to avoid data consistency issues

Question 5

What is the 'golden signals' framework and which four metrics does it include?

Accepted Answer

Latency, traffic (throughput), errors, and saturation — the four signals that best describe the health of any service from a user and capacity perspective

Answer

CPU, memory, disk I/O, and network bandwidth — the four system-level metrics that determine service performance

Answer

Availability, reliability, performance, and security — the four dimensions of service quality per the NIST framework

Answer

Mean time to detect, mean time to respond, mean time to recover, and mean time between failures — the four incident response quality metrics

Question 6

What is 'capacity planning' in SRE, and what data is MOST critical for generating accurate capacity forecasts?

Accepted Answer

Capacity planning ensures service resources are provisioned to handle current and projected future load; the most critical inputs are historical traffic growth trends, SLO performance at various utilization levels from load tests, and upcoming business events that could cause traffic spikes

Answer

Capacity planning is purely a cost optimization exercise focused on minimizing infrastructure spend

Answer

Capacity planning only requires knowing the current peak traffic level and doubling it for a conservative estimate

Answer

Capacity planning is the responsibility of finance teams, not SREs — SREs only respond to capacity problems after they cause incidents