AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Applying SRE Error Budgets to Services Running on EKS: SLIs, Burn Rates, and Release Decisions
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Build error budgets around user requests, not healthy pods. Implement Prometheus burn-rate alerts, diagnose EKS failure modes, and turn reliability evidence into release decisions.
TL;DR
Error budgets turn a service reliability objective into an explicit allowance for unsuccessful user events. For EKS workloads, measure requests at a defined user boundary, calculate good and total events consistently, and distinguish application failures from load balancer failures and missing telemetry. This guide implements request-based SLIs, rolling budget calculations, and multiwindow burn-rate alerts with Prometheus. It also explains latency budgets, low-traffic edge cases, Kubernetes disruption controls, safe release gates, and operational validation. Use the resulting evidence to prioritize mitigation and reliability work without treating autoscaling as a universal response to errors.
Healthy Kubernetes objects can still deliver an unreliable service
An EKS deployment can have ready replicas while checkout requests fail because a database rejects connections. It can also have restarting pods while users see no failures because other replicas absorb the traffic. These situations require different responses, and a pod-health dashboard cannot distinguish them by itself.
SRE error budgets on EKS measure the reliability users receive and turn that evidence into operational decisions. The implementation has four connected steps: define the service objective, measure eligible user events, detect dangerous consumption, and agree what changes are appropriate when reliability deteriorates.
An error budget is an allowance for bad events, not a pool of CPU or replicas. Infrastructure capacity can protect that allowance, but adding capacity does not repair invalid releases, broken credentials, or dependency failures.
The examples below use a hypothetical checkout API with a 99.9% availability objective over a rolling 30-day window. The traffic volumes and incident calculations are illustrative arithmetic, not measured production results. Start by reviewing the EKS reliability guidance against your actual application dependencies and failure boundaries.
Image. User requests pass through an ALB to checkout pods; boundary metrics feed SLI calculations, burn alerts, and release policy while infrastructure metrics support diagnosis.
Measure the user boundary first. Application and infrastructure telemetry explain failures, but they cannot replace requests that never reached the pods.
Step 1: Define a contract before defining a dashboard
Write an SLO document that answers five questions: which user action, which environment, which eligible events, which success conditions, and which reporting window? Avoid a definition such as “the cluster is available,” which leaves business behavior unspecified.
Code example. Refer to the written article for the complete code.
The latency objective here is separate from availability. A slow success can satisfy the availability objective and fail latency. A failed request also fails this particular latency definition because it requires success. Document that overlap; do not add the two error counts together and call the sum a unique incident count.
For HTTP APIs, status codes require interpretation. A malformed customer request can be excluded or counted as an expected response, depending on the contract. A 429 generated by an unexpectedly small service quota may represent a service failure. A 200 carrying a failed business transaction may be bad. Consistent classification matters more than a blanket “all 4xx are good” rule.
Choose whether retries represent separate attempts or one user operation. Attempt metrics diagnose servers; operation metrics describe whether the user eventually completed checkout. If you use both, give them separate names and denominators. Never count good user operations above bad server attempts.
The arithmetic: allowed events, spent events, and remaining fraction
Let S be the SLO objective, N eligible events, and E bad events during the same reporting window:
Code example. Refer to the written article for the complete code.
If N = 20,000,000 and S = 0.999, the allowance is 20,000 bad events. If 8,000 events are bad, 12,000 remain, or 60% of the allowance. Negative remaining budget communicates an actual SLO miss; hiding it with a display clamp loses useful information.
The familiar 43.2-minute allowance comes from 30 × 24 × 60 × 0.001. That is a time-based 99.9% calculation. It is not interchangeable with a request-based budget when traffic varies. Five minutes of failures during a sale can cost more user events than an hour overnight.
A rolling window also changes continuously: older failures and older successes leave the calculation, while new traffic changes N. Budget remaining is a retrospective ratio, not a fixed bank balance guaranteed to grow monotonically. A calendar-month objective has different boundary behavior and should use calendar-aware reporting.
Step 2: Capture the failures your pods cannot see
Place primary accounting at a boundary that sees the complete eligible journey. Instrumentation inside the handler cannot observe TLS failures, requests rejected upstream, or a load balancer with no usable targets. Use edge telemetry and external probes to cover those blind spots, while retaining application counters for diagnosis.
An ALB exposes separate metrics such as HTTPCode_ELB_5XX_Count, HTTPCode_Target_5XX_Count, and RequestCount. Their definitions and dimensions differ; RequestCount only increments when the ALB can choose a target. Adding every 5xx counter and dividing by that metric does not automatically produce a coherent user-facing SLI. Reconcile populations and aggregation periods first. AWS documents ALB metric semantics and statistics.
In the following implementation, a trusted accounting component exports custom counters. These names are examples, not metrics automatically supplied by EKS:
Code example. Refer to the written article for the complete code.
Initialize both outcome series at zero when the component starts, increment exactly one outcome per eligible event, and keep the labeling contract uniform. A missing bad series should not make the numerator disappear during healthy traffic. An unavailable exporter should not be fabricated into a zero-valued healthy series.
Use bounded labels such as service, environment, and a reviewed route template. User IDs, authorization headers, full URLs, and transaction IDs belong neither in these counters nor in public dashboards. Such labels create unbounded cardinality and may expose sensitive information. Prometheus instrumentation guidance explains why instrumentation and label design need deliberate limits.
Latency: count qualifying events instead of averaging percentiles
For a classic histogram whose observations cover the eligible population, configure a bucket exactly at the contractual threshold. This example assumes histogram observations contain only successful eligible requests, while the denominator counts all eligible requests:
Code example. Refer to the written article for the complete code.
The resulting fraction is successful eligible requests completed within 500 ms. Verify that both metrics represent the same ingress population and service labels. Comparing an internal-handler duration with a browser journey objective would understate network and queueing latency.
A histogram bucket supports an event-based threshold directly. Averaging pod-level p99 values does not yield a service-wide p99, and p99 itself is not an error count. For this classic-histogram example, using an exact bucket boundary also avoids interpolation around the objective. Prometheus explains histogram aggregation and threshold fractions.
Step 3: Build recording rules with explicit missing-data behavior
For every alert window, calculate the sum of per-series counter rates before calculating the ratio. This retains counter-reset handling when individual pods or exporters restart. A service-wide average of pod error percentages gives equal weight to a pod receiving one request and a pod receiving thousands.
The following generator produces the full four-window rule file used later. Install PyYAML in an isolated environment before running it:
Code example. Refer to the written article for the complete code.
Code example. Refer to the written article for the complete code.
Run python generate_slo_rules.py, then promtool check rules slo-recording.yaml. The service name stays in the output labels; ensure scrape relabeling supplies canonical labels rather than letting individual exporters invent them. These are ordinary Prometheus rule files. A Prometheus Operator installation requires them under spec.groups in a PrometheusRule, plus selectors that actually load the object. Recording-rule configuration is documented here.
The code deliberately leaves zero traffic as 0 / 0, which produces NaN, rather than declaring the service perfect. Missing series remain missing. This distinction is valuable:
Table. Columns: Observation, Interpretation, Follow-up.
Row 1. Observation: Positive total, zero bad; Interpretation: Observed requests succeeded; Follow-up: Continue normal reporting.
Row 2. Observation: Total and bad both zero; Interpretation: No observed eligible requests; Follow-up: Evaluate expected traffic and external probes.
Row 3. Observation: Counter absent; Interpretation: Accounting unavailable or label contract broken; Follow-up: Investigate telemetry.
Row 4. Observation: Positive bad, zero total; Interpretation: Inconsistent counters or reset/population issue; Follow-up: Reject the SLI until corrected.
End of table.
Use a separate telemetry alert for this named service:
Code example. Refer to the written article for the complete code.
This alert detects an entirely absent outcome population, not partial loss when one exporter disappears but others remain. Monitor scrape health, exporter inventory, edge request reconciliation, and freshness as additional coverage. Do not use or vector(0) as a universal cure for absent metrics. Prometheus defines rate, increase, and absence functions.
Step 4: Alert on burn rate, then decide how to respond
Burn rate normalizes the recent bad-event fraction by the permitted fraction:
Code example. Refer to the written article for the complete code.
With a 99.9% objective, 0.1% bad events means burn rate 1; 1% means burn rate 10; 5% means burn rate 50. A threshold of 5% has no special meaning unless it follows from the objective and response policy.
Under a constant request-rate approximation, the fraction of a 30-day allowance consumed over h hours at burn rate b is b × h / 720. Choosing 2% consumption over one hour gives b = 0.02 × 720 / 1 = 14.4. Choosing 5% over six hours gives b = 0.05 × 720 / 6 = 6. Traffic seasonality can make actual event-budget consumption differ, so display event counts alongside these signals.
Image. For a 99.9 percent objective, error ratios of 0.1, 0.6, 1.44, and 5 percent correspond to burn rates of 1, 6, 14.4, and 50. Paired short and long windows determine whether to page.
Burn rate connects error fraction to the objective. The 14.4 and 6 thresholds shown here are starting points for a 30-day window, with explicit traffic assumptions.
Use a short window to confirm that the long-window problem is still active:
Code example. Refer to the written article for the complete code.
Each pair uses and; the pairs use or. All ratio rules retain identical service/environment labels. A long arbitrary for duration would postpone severe outages; evaluate notification noise using historical traffic before adding delays. Add slower ticket conditions when your team can act on sustained lower burn. Google's SLO alerting chapter provides the multiwindow strategy and its limitations.
Low traffic needs a separate policy. One failed operation among ten is a 10% fraction regardless of whether ten requests arrive in a second or a day. Do not blindly suppress high burn with a request-count floor if every transaction matters. Longer windows, business-critical individual-event alerts, and synthetic journey checks serve different needs. Keep synthetic requests identifiable so successful probes cannot dilute actual customer failures.
Calculate rolling budget remaining independently of paging
This query returns the remaining fraction for the availability objective:
Code example. Refer to the written article for the complete code.
Require retention covering the full window, historical label continuity, and accounting coverage checks. Do not compare a newly installed seven-day metric history with a 30-day contract and call the result complete. For large fleets, precompute aggregate counts or use an appropriately retained backend instead of repeatedly scanning every raw series over 30 days. Document reset handling, sampling gaps, and duplicate ingestion: this is operational event estimation, not a financial ledger.
EKS mitigation: follow the failure mechanism
Once a page fires, establish user impact and stop dangerous changes. Then choose a mitigation that fits the failure:
Table. Columns: Evidence, Likely mechanism to investigate, Candidate mitigation.
Row 1. Evidence: Errors begin with a release; old revision succeeds; Likely mechanism to investigate: Application or configuration regression; Candidate mitigation: Roll back after checking schema compatibility.
Row 2. Evidence: CPU and queueing rise; dependency has headroom; Likely mechanism to investigate: Application saturation; Candidate mitigation: Add validated pod/node capacity or shed optional work.
Row 3. Evidence: Database latency and connections spike; Likely mechanism to investigate: Shared dependency bottleneck; Candidate mitigation: Limit concurrency, reduce retries, restore dependency capacity.
Row 4. Evidence: ALB failures rise while app request counts drop; Likely mechanism to investigate: Target availability or edge failure; Candidate mitigation: Check target health, readiness, routing, and recent infrastructure changes.
Row 5. Evidence: One zone disproportionately fails; Likely mechanism to investigate: Failure-domain placement or dependency; Candidate mitigation: Shift traffic only after verifying surviving capacity.
End of table.
HPA computes desired replicas from configured resource or custom metrics; it does not come with a universal “error budget allocator.” If error ratio feeds an autoscaler, prove that more replicas improve the causal bottleneck and impose a safe limit. Kubernetes documents HPA behavior and custom metrics.
Retries deserve the same scrutiny. If every pod retries database timeouts several times, doubling pods can multiply database pressure. Bound attempts, use jitter, cap concurrent work, and require idempotency for side-effecting operations. A successful retry can protect a user-operation SLI while increasing backend-attempt failures; both views are useful.
Kubernetes controls protect different failure paths
For an existing replicated stateless service, this partial Deployment configuration shows deliberately conservative rollout settings:
Code example. Refer to the written article for the complete code.
This strategy needs spare resources for the surge replica and a readiness probe that reflects ability to serve. Progress can stall when capacity is unavailable. The Deployment controller does not guarantee that ready pods are delivering correct business responses. Deployment documentation explains rollout controls.
For supported voluntary evictions, a separate disruption budget can constrain maintenance:
Code example. Refer to the written article for the complete code.
This object depends on the selector matching the intended replicated workload. It does not prevent involuntary node failure, and workload rollout behavior needs its own strategy. A restrictive budget can also block draining during maintenance. Kubernetes details PDB scope and operational behavior.
Distribute replicas across failure domains when capacity exists. An example scheduling fragment is:
Code example. Refer to the written article for the complete code.
This constrains placement across eligible topology domains; it does not provision nodes or guarantee three-zone placement by itself. Verify node labels, taints, affinity, and actual eligible zones. Hard constraints trade placement consistency against pending pods when capacity disappears. Topology spread documentation describes these scheduling rules.
Turn the budget into a release policy with an accountable owner
Burn alerts protect the service during incidents. A budget policy determines what engineering work takes priority afterward. Agree it with product and service owners before a breach, including exceptions for urgent security work and reliability repairs. Google's example error budget policy illustrates using reliability results to govern change.
The following is a suggested local policy, not an industry mandate:
Table. Columns: State, Release decision, Required action.
Row 1. State: Valid measurement, budget healthy, no active dangerous burn; Release decision: Normal reviewed delivery; Required action: Observe service and canary signals.
Row 2. State: Budget declining rapidly; Release decision: Require an explicit risk review; Required action: Identify top failure contributors.
Row 3. State: Budget exhausted; Release decision: Pause discretionary feature changes; Required action: Prioritize recovery and prevention.
Row 4. State: Telemetry incomplete; Release decision: No automatic reliability approval; Required action: Restore evidence or use a documented human exception.
End of table.
Avoid embedding this decision in a single unchecked Prometheus query. A release check should record the service, environment, objective, query time, telemetry status, result, and policy version. Secure its read access and separate it from deployment permissions. The person or automation approving an exception should leave an auditable reason.
A canary also needs its own outcome comparison. If only 1% of traffic reaches a broken revision, a global error fraction can look acceptable while that revision fails every operation. Compare revision-specific success with the stable baseline and maintain an overall user SLI. Use bounded revision labels and retire old series to control cardinality.
Why this design separates accounting, diagnosis, and policy
The edge accounting answers “what did users receive?” Application traces and EKS metrics answer “where did the failure happen?” The policy answers “what should the team do next?” Keeping these responsibilities explicit prevents a healthy node count from overruling bad user outcomes and prevents a metric gap from granting release permission.
This also makes ownership clearer. The platform team can own collection and access control while the service team owns success classification and mitigation. Shared dependencies need their own reliability objectives, but their budgets do not erase customer-impacting failures from checkout's budget.
Validate the complete loop before relying on it
Use rule checks for syntax and targeted fixtures for behavior. Then perform a controlled nonproduction exercise with real telemetry and notifications. Prometheus supports rule-unit tests so alert behavior can be reviewed with changes.
Code example. Refer to the written article for the complete code.
Build slo-tests.yaml using your actual scrape/evaluation intervals and include these cases:
Table. Columns: Scenario, Expected result.
Row 1. Scenario: Constant 0.05% bad events; Expected result: No page for the shown 99.9% thresholds.
Row 2. Scenario: 2% bad events across both 1h and 5m; Expected result: Fast pair pages.
Row 3. Scenario: 1% bad events across both 6h and 30m; Expected result: Slower pair pages.
Row 4. Scenario: Elevated long window after traffic recovers; Expected result: Short-window condition prevents a lingering page.
Row 5. Scenario: Counter resets during exporter restart; Expected result: Rate calculation preserves service accounting behavior.
Row 6. Scenario: Zero traffic; Expected result: No fabricated healthy percentage; probe/traffic policy handles it.
Row 7. Scenario: Missing good or bad population; Expected result: Telemetry warning fires.
Row 8. Scenario: One exporter missing while others remain; Expected result: Inventory or scrape alert detects partial coverage loss.
End of table.
During the live exercise, record elapsed time from injected failure to observed metric change, alert evaluation, notification, acknowledgment, and mitigation. Those measured delays determine whether the response can protect the budget. Do not infer notification delivery from an alert expression evaluating true.
For production dashboards, show objective and reporting window next to eligible volume, bad volume, remaining fraction, current burn, and measurement coverage. Add deployment annotations and useful dependency panels below that user-facing view. Review the top budget-consuming causes periodically and fund repairs with demonstrated impact.
Frequently asked questions
What is an error budget for an EKS service?
An error budget is the allowed fraction or count of bad user events under an SLO during a defined window. For example, 99.9% availability allows 0.1% bad eligible events. Kubernetes health and infrastructure uptime help explain behavior but do not define the user-facing service result.
Should the HPA scale an EKS application when error rate rises?
Scale when a validated capacity signal shows that replicas can reduce the bottleneck. Error fraction alone does not identify capacity shortage; database exhaustion and configuration regressions need different mitigations. Extra replicas can intensify pressure on a saturated shared dependency.
How do I calculate error budget remaining in Prometheus?
Count total and bad eligible events over the same window, calculate the allowance as total multiplied by one minus the objective, and subtract observed bad events. Divide that remainder by the allowance to express a remaining fraction. Treat incomplete history, missing telemetry, and zero-event windows explicitly before making release decisions.
Do PodDisruptionBudgets protect my service SLO?
A PDB constrains supported voluntary evictions for matching pods. It does not guarantee correct requests, prevent node failures, or replace Deployment rollout settings. Combine it with readiness, placement, capacity, dependency resilience, and user-facing measurements.
Should missing SLI metrics count as success?
Missing metrics indicate an unknown measurement state. Alert on absent or partial accounting and use external probes to retain visibility when the serving path or telemetry fails. Automatic release approval should require valid evidence or an explicit accountable exception.
Put one service through the entire workflow
Choose a single user journey, review its eligibility and success contract, and reconcile boundary counts with application observations. Generate the rules, replay representative incidents, confirm notification delivery, and agree an owner-backed release policy. Then extend the same accounting conventions across the fleet while keeping the success criteria specific to each service.