Monitoring, Alerting & Troubleshooting Flashcards

What is the purpose of the `scrape_timeout` setting in a Prometheus scrape configuration?

Answer: Maximum duration allowed for a single scrape request before it is aborted

scrape_timeout defines how long Prometheus will wait for a target's HTTP response before canceling the scrape and marking it as a failure.

A PromQL query returns `No data` for `rate(http_requests_total[5m])`. The metric exists in Prometheus. What is the most common reason?

Answer: The selected time window is shorter than two scrape intervals

rate() requires at least two data points within the range; if the range window is shorter than two scrape intervals, there may be insufficient data.

Which Prometheus component is responsible for discovering and maintaining the list of scrape targets?

Answer: Service discovery / relabeling pipeline

Prometheus uses service discovery mechanisms (e.g., file_sd, kubernetes_sd) combined with the relabeling pipeline to discover and configure scrape targets.

An inhibition rule in Alertmanager has `source_match: {severity: 'critical'}` and `target_match: {severity: 'warning'}`. What happens when a critical alert fires?

Answer: Warning alerts matching the target are suppressed

Inhibition rules suppress (mute) target alerts when a matching source alert is firing, reducing redundant notifications.

What does `increase(errors_total[1h])` calculate in PromQL?

Answer: The total increase in the counter value over the last hour

increase() calculates the total increase in a counter's value over the specified time range, accounting for resets.

In a Prometheus recording rule, what is the primary benefit of pre-computing an expensive query?

Answer: It speeds up dashboard and alerting queries by storing results as new time series

Recording rules evaluate and store complex expressions as new time series, so dashboards and alerts query the precomputed result instead of recomputing it each time.

Which HTTP endpoint should you check on a Prometheus server to verify that a specific target is being scraped successfully?

Answer: /api/v1/targets

/api/v1/targets returns the current state of all scrape targets including their health status and last scrape error.