AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Error Budgets That Drive Decisions: Build Burn-Rate Alerts Without Blind Spots
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Connect error budgets to release authority and actionable pages. Build paired Prometheus burn-rate alerts, preserve event semantics, and diagnose missing telemetry.
TL;DR: Useful error budgets connect customer outcomes to release decisions and incident response. This article defines an eligible-event contract and provides a complete Prometheus rule file for paired fast and sustained burn alerts. It explains aggregation, counter resets, label matching, sparse traffic, and missing telemetry, then diagnoses the Boolean-comparison mistake that can page healthy services. A practical policy separates active burn from budget exhaustion and assigns decision authority. Thresholds are illustrative policy choices; validation concerns alert logic, not measured production reliability or the accuracy of application instrumentation.
An error budget should change an engineering decision
A checkout service meets its monthly availability objective, yet the on-call engineer gets paged every night. Another service stays quiet during a rolling outage because its alert requires an uninterrupted hour above a threshold. Both teams have an SLO dashboard. Neither has connected budget consumption to an actionable response.
Build the alert around the customer outcome and the team's intervention time. Then connect it to a policy: who can slow releases, which changes remain permissible, and what evidence ends the restriction. An error budget without an owner is just another chart.
Specify the event contract first
For this example, availability means a valid terminal checkout response before the product-defined deadline. The instrumentation exports slo_events_total with service, environment, and result labels. Eligible outcomes have exactly one result: good or bad. Internal retries do not create additional customer requests in the denominator.
Initialize both result series to zero, even before the first failure. A missing bad series is different from a present counter that has not increased. Record deadline violations as bad even when an eventual HTTP response is successful. Decide which validation errors and client cancellations are eligible before implementing the counter.
The SLO implementation guidance explains why the indicator must represent user experience. Here, service-level aggregation pools eligible requests across replicas. It can hide a small tenant's outage; add a separate objective for critical journeys or cohorts when business impact demands it, rather than attaching unbounded customer identifiers to every metric.
Turn the objective into an alert threshold
For an illustrative 99.9% objective, the allowed bad-event fraction is 0.001. Burn rate is the observed bad-event fraction divided by that allowance. A 1% bad fraction therefore represents 10 times the allowed rate. These are definitions and arithmetic, not measured service performance.
Use two paging paths for the 30-day objective: 14.4 times the allowance over both one hour and five minutes; or six times over both six hours and thirty minutes. The Google SRE alerting chapter describes this multiwindow pattern. The long window establishes significance; the short window checks that the problem remains active. Both must pass within a pair, while either pair can page.
Table. Columns: Path, Long window, Short window, Bad fraction threshold.
Row 1. Path: Faster burn; Long window: 1 hour; Short window: 5 minutes; Bad fraction threshold: Greater than 1.44%.
Row 2. Path: Sustained burn; Long window: 6 hours; Short window: 30 minutes; Bad fraction threshold: Greater than 0.6%.
End of table.
These thresholds are starting policies, not universal severity boundaries. For request-based budgets, actual spend also depends on the share of period traffic represented by the window. Time-based estimates assume representative traffic and do not establish the remaining budget.
Image. Two paired lookback rulers: long-window significance and short-window activity
A complete Prometheus rule file
Save the following as checkout.rules.yml. It assumes the counter contract above and an existing Prometheus scrape path. Use Prometheus/promtool 3.15.0 for the commands below; the version is available in the official release.
Code example. Refer to the written article for the complete code.
Every recording rule keeps the same service and environment identity. The explicit matching clauses make that intent visible during review. Keep the recording rules and alert in one group so the group evaluates them sequentially, as described in the recording-rule documentation.
The rate function handles counter resets. Apply it to each original series before aggregation; summing counters before calculating a rate can obscure a replica reset. The ratio here weights by request volume, rather than averaging per-replica error percentages.
Validate in an isolated, disposable container without changing the running server. Prerequisites are Docker with Linux containers and a POSIX shell:
Code example. Refer to the written article for the complete code.
The command checks rule syntax and exits; --rm removes the container. It does not create a scrape target, send notifications, or prove the application's counter semantics.
For deployment, append the file path to the existing configuration's rule_files list, retaining other entries:
Code example. Refer to the written article for the complete code.
Review the change through the monitoring team's normal rollout, then confirm the loaded rules, evaluation health, and Alertmanager route. Removing this entry and reloading through the same process is the rollback. The example intentionally has no long for duration; adding one introduces additional detection delay. Alerting-rule documentation defines that pending-state behavior.
A quiet dashboard can mean missing evidence
Zero traffic produces no meaningful error fraction. Missing scrape data can remove the entire expression from evaluation. Neither state proves the service is healthy.
Keep telemetry-health alerts separate from budget alerts. Compare expected targets with actual scrape health, monitor rule evaluation failures, and alert on missing SLI coverage using an inventory of expected services. A global absent query cannot enumerate every missing label combination without that inventory.
For sparse services, review single-event impact and expected intervention cost. A blanket minimum-request guard can suppress the only evidence of a real outage. Choose a longer observation policy, synthetic availability check, or explicit escalation for critical transactions; document which failure each signal detects.
Failure case: a Boolean comparison that pages healthy services
Consider a configuration review that changes a threshold comparison to greater-than-bool. In PromQL, that modifier returns zero or one for matching elements instead of filtering out false elements. Alerting rules activate on returned vector elements, not on whether their numerical value is one.
Combining these Boolean vectors with and makes the trap worse: set intersection checks whether matching elements exist. Healthy zero-valued elements can survive. The operator documentation distinguishes comparison filtering from logical set operations.
Diagnose this by evaluating each half of the expression for a healthy service. If zero-valued rows remain after a comparison, inspect bool and label matching. Recover by restoring filtering comparisons, validating the rule, and checking healthy, active-burn, and recovered scenarios before reloading.
Use promtool rule tests for regression cases. Include no-alert behavior below the threshold, each paging path independently, a recovered short window, mismatched environments, and absent telemetry. Test fixtures verify logic; they are not evidence of production reliability.
Make the budget policy operational
The example error-budget policy offers a concrete governance model. Adapt it with the product and service owners: identify the reporting window, authority to restrict releases, permitted reliability work, and exceptions for urgent security fixes.
A page should trigger triage and mitigation, not automatically freeze every deployment. Budget exhaustion and active burn answer different questions. Use the historical budget view for release decisions and the paired windows for immediate response.
Assign the alert to one owning team. Preserve a stable alert identity; put changing diagnostic values in annotations. Route pages to the existing on-call escalation and slower issues to a tracked reliability backlog. Require a runbook with rollback ownership, dependency checks, and criteria for recovery.
FAQ
Does a burn rate of one mean an incident? It means consumption matches the allowed rate. Whether to notify depends on the observation window and response policy.
Should I add an hour-long for clause? That adds an hour of pending time even during severe failures. Review the detection cost instead of using duration as a substitute for lookback windows.
Can missing metrics count as successful requests? No. Missing telemetry is an evidence gap; handle it with coverage and scrape-health monitoring.