AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
SRE Interviews: Connect SLOs, Error Budgets and Incident Decisions
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Work through reliability contracts, window accounting, burn-rate urgency and accountable incident decisions with tested Prometheus alert logic.
TL;DR: SRE interview answers should connect a customer-facing reliability contract to request-weighted budget accounting and current incident evidence. Remaining budget and recent burn rate answer different questions, especially under variable traffic and rolling windows. This article works through explicit SLI eligibility, a fast-burn AND rule, missing telemetry and low-traffic limits, incident ownership and release-policy exceptions. A complete Prometheus rule and local regression fixtures make the signal behavior reviewable without treating test inputs as operational measurements. Recovery still requires verified customer outcomes, trustworthy instrumentation and an accountable service owner.
"Our monthly SLO is still green, but checkout is failing now. Do you page someone, stop releases, or wait?" This SRE interview question tests whether you can connect a reliability contract, operational urgency and incident ownership without confusing their timescales.
The worked answers below use an illustrative checkout service with a 99.9% request-success SLO over a rolling 30-day window. The threshold is a design choice, not a claim about a real service. A complete Prometheus rule and local regression fixtures make the alert behavior reviewable; they do not measure production reliability.
Question one: what exactly does the SLO promise?
Worked answer: Define the user journey, eligible population, good outcome, observation point, window and owner before quoting a percentage. For checkout, count each eligible logical attempt once at the agreed boundary, classify its terminal outcome, and explicitly account for timeouts. State whether invalid client submissions are excluded and whether retries represent another attempt. Internal RPC success is a poor substitute when the user cannot complete checkout. Implementing SLOs
A practical contract might exclude structurally invalid submissions but include server errors and valid requests that time out. An application-level decline could be an expected business outcome; a failed payment dependency could be a reliability failure. Product and service owners must agree on those distinctions rather than letting an HTTP status regex decide them accidentally.
Specify latency separately if "successful but unusably slow" is a meaningful failure. Do not average instance success percentages: an idle replica and a busy replica should not receive equal weight. Aggregate bad and total event counts for the same service and environment, then form the ratio. Retain cohort views so a global SLO cannot conceal an important customer segment. SLI principles
Follow-up: "Can you just choose 100%?" A target must support a decision about customer needs and engineering cost. A zero error allowance also removes the useful budget trade-off. Discuss the product's consequences and feasible operating model rather than presenting another company's target as universal.
Question two: calculate remaining budget, then separate urgency
Worked answer: For 10,000,000 eligible requests in the current window, a 99.9% objective permits 10,000 bad events. If 7,500 were bad, 2,500 remain: 25% of that window's allowance. These are illustrative arithmetic inputs, not observations. In a rolling window, requests enter and old requests expire; remaining allowance is not a bank account that only decreases.
Burn rate answers a different question: recent bad-event ratio divided by the permitted bad-event ratio. With a 0.1% allowance, a recent 2% bad ratio burns at 20 times the target rate. It can justify urgent response even while the full-window SLO remains satisfied. Conversely, exhausted historical budget does not prove an incident is still active. Burn-rate alerting
A threshold of 14.4 over one hour corresponds approximately to 2% of the full 30-day allowance when traffic is uniform: 14.4 × 1 / 720 = 0.02. Request-weighted budget consumption can differ under bursty traffic. Do not describe that approximation as an exact event ledger or a forecast of the remaining budget's exhaustion time.
Image. Separate window accounting, current burn evidence and incident/release decisions
The diagram separates the accounting question from the response question. Neither a remaining percentage nor an alert independently owns a release decision.
Question three: show an alert that also notices recovery
Worked answer: Combine a long-window condition with a shorter-window condition using AND. The long window establishes sustained budget pressure; the short window checks whether it is still happening. A long for duration is not a substitute: it can delay a severe outage and reset during intermittent failures. The complete example below implements one fast-burn paging branch, not an entire service's alert portfolio.
Prerequisites: Prometheus/promtool 3.15.0, two monotonically increasing counters exported by the agreed observer, and an existing scrape and notification path. checkout_eligible_total counts eligible terminal attempts; checkout_bad_total counts their bad subset. Initialize the bad counter to zero even before the first error. Keep identical service and environment labels and avoid duplicate observation or HA double counting. Prometheus's rate handles resets per source series; sum afterward. Release, counter functions
Save as checkout.rules.yml:
Code example. Refer to the written article for the complete code.
Comparisons deliberately omit bool. False comparisons remove matching series; and intersects the surviving label sets. Turning false values into zero-valued series with bool would still leave them present, changing this logic. No for is configured here, so an eligible evaluation can become firing immediately; notification delivery is a separate Alertmanager concern. PromQL operators, alert state
Follow-up: "Does no alert mean healthy?" No. Zero traffic produces an undefined ratio, and absent telemetry produces no series. Both avoid a false fast-burn page here, but neither establishes success. Monitor scrape health, observer coverage and expected traffic independently. Low-volume services may need a different paging policy or a carefully scoped synthetic availability signal. Do not manufacture requests merely to make a percentage statistically impressive.
Verify the rule's decisions without inventing an incident
Save this complete checkout.test.yml beside the rule. The fixtures test sustained failure, a recovered short window, zero traffic and missing counters. Counter increments are test inputs, not throughput measurements. Promtool rule tests
Code example. Refer to the written article for the complete code.
Run promtool check rules checkout.rules.yml, then promtool test rules checkout.test.yml using the stated version. The local syntax check and all four fixture groups passed with promtool 3.15.0. These checks validate rule syntax and specified behavior, not production instrumentation or notification delivery. No cloud infrastructure or monitoring configuration was changed. Retain the files as regression tests; removing them does not disable a remotely deployed rule.
Question four: the page fires; what do you do first?
Worked answer: Establish impact, affected journey and time window, declare an incident when the agreed criteria apply, and assign incident command, operations and communication ownership. Record evidence, hypotheses, mitigations and outcomes in one working timeline. The incident commander coordinates decisions; the operations lead executes bounded recovery; the communications lead maintains stakeholder updates. One person can initially hold several roles, but handoffs must be explicit. Incident response
Suppose a release introduced payment timeouts. Restore customer service through the authorized owner using a compatible rollback, feature disablement or scoped traffic change. Check schema and dependency compatibility first. Do not wait for a final root cause before an evidence-supported mitigation, and do not treat temporal correlation as proof that rollback is safe.
If the short-window signal recovers while the long window stays high, the AND rule can clear. That validates one signal's recovery, not the entire incident. Verify representative checkout outcomes, telemetry coverage and notification state before declaring recovery. Preserve evidence and follow up on the failure mechanism and ownership gaps.
Question five: must every release stop when budget is exhausted?
Worked answer: Apply the agreed error-budget policy, not an improvised veto. Define who can pause promotion, what reliability work takes priority, which urgent security or recovery changes are exceptions, who approves them and what evidence reopens routine releases. Error-budget policy supports a shared engineering decision; it is not punishment for the on-call team. Policy example
A product can still meet an aggregate SLO while a critical cohort is failing, and corruption or security impact can warrant incident response independently of availability budget. State the scope and uncertainty explicitly. A senior answer connects the contract, recent evidence, accountable recovery and the conditions under which the decision should change.