Generate alerting rules for error rate, latency, burn rate, saturation and pod restarts, each with a for duration, a severity and a real description.
2 alerts · 35 lines
The rule inspector finds no errors in this, which is the intended result.
What it cannot tell you: A ratio says nothing about volume: one failed request out of two is 50% and usually not worth waking anyone.
What it cannot tell you: `up == 0` means the scrape failed, which can be a network problem rather than a down service.
A runbook_url annotation is what turns a page into a procedure. It matters most for the alerts that fire rarely, which are exactly the ones nobody remembers how to handle. Leave it blank and the annotation is omitted rather than emitted empty.
# Generated alerting rules — edit them.
#
# Every alert has a `for` duration, a severity label and a description,
# because an alert without those three pages someone with nothing to act on.
#
# Thresholds here are starting points, not recommendations: the right error
# ratio or latency bound depends on what the service promises. The metric
# names assume the common exporters (http_requests_total, node_exporter,
# kube-state-metrics) and will need changing if yours differ.
#
# Nothing here has been evaluated against a Prometheus server.
groups:
- name: service-alerts
rules:
- alert: HighErrorRate
expr: "(\n sum(rate(http_requests_total{job=\"my-service\", status=~\"5..\"}[5m]))\n /\n sum(rate(http_requests_total{job=\"my-service\"}[5m]))\n ) > 0.05\nand\n sum(rate(http_requests_total{job=\"my-service\"}[5m])) > 1"
for: 5m
annotations:
summary: Over 5% of requests are failing
description: "The 5xx ratio over the last 5 minutes is above 5% while the service is handling more than 1 request per second. The second condition is deliberate: without it, one failure on an idle service reads as a 100% error rate."
runbook_url: https://runbooks.example.com/high-error-rate
labels:
severity: critical
service: my-service
- alert: TargetDown
expr: up{job="my-service"} == 0
for: 5m
annotations:
summary: A scrape target is not responding
description: up == 0 means the scrape itself failed. That can be the service being down, but it can equally be a network path, a changed port or a TLS problem — check whether the service is serving before assuming it is not.
runbook_url: https://runbooks.example.com/target-down
labels:
severity: critical
service: my-serviceWhat the rule inspector says
HighErrorRate aggregates without by() or without()
The result is a single series with no labels, so the alert cannot say which instance, job or route is affected. One alert fires for the whole fleet and the notification has nothing to point at.
HighErrorRate
The thresholds here are starting points, not recommendations — the right error ratio or latency bound depends on what your service promises, and only you know that. The metric names assume the common exporters (http_requests_total, node_exporter, kube-state-metrics) and will need changing if yours differ. Nothing has been evaluated against a Prometheus server, so an alert referencing a metric you do not produce will simply never fire, which is the quietest way for an alert to be wrong.
Prometheus Alert Rule Generator runs entirely on your device — no Prometheus server is contacted and nothing is evaluated.
You’re on the Free plan. Compare plans
The error-rate alert has a volume guard, which is the single most important thing about it. A ratio says nothing about volume: one failed request out of two is a 50% error rate and almost never worth waking anyone. The generated rule requires more than one request per second alongside the ratio, and the description says why — because a reader who does not know will delete the condition that stops the false pages.
The burn-rate alert uses the multi-window form. Both a fast window and a slow one must be breaching, which trades a little detection speed for far fewer false pages: the short window detects quickly and the long one stops a two-minute blip from paging. The 14.4 multiplier is the rate that consumes 2% of a 30-day error budget in one hour, and the thresholds scale with the SLO target you set.
The latency alert gets the aggregation order right: `rate()` over the buckets first, `histogram_quantile` last, with `le` surviving the aggregation. Taking the quantile per-series and averaging afterwards produces a number that is not a quantile of anything, and it is the most common histogram mistake there is.
Each template also states what it cannot tell you. `up == 0` means the scrape failed, which can be a network path rather than a down service. High CPU is only a fault if it is causing queuing. A pod restarting repeatedly looks identical whether the cause is a crash or an over-aggressive liveness probe.
Every generated rule is run through the rule inspector on the page, which was written separately. Thresholds are starting points rather than recommendations, and nothing has been evaluated against a Prometheus server.
Because a ratio ignores volume. On an idle service a single failed request is a 100% error rate, and an alert on the ratio alone pages someone for it. The rule requires more than one request per second as well, which is the condition that makes the alert survivable.
It is the rate at which you consume 2% of a 30-day error budget in one hour. Alerting on it catches a genuine fast burn quickly while ignoring blips, and the multi-window form — both a 5-minute and a 1-hour window breaching — is what keeps the false-page rate low.
Because the quantile must be computed from bucket rates, not averaged after the fact. Taking a quantile per series and averaging the results produces a number that is not a quantile of anything — it looks plausible on a graph, which is why it ships so often.
Almost certainly not as-is, and the generated file says so at the top. The right error ratio or latency bound depends on what your service promises. These are starting points with the structure correct; the numbers are yours to decide.