AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Rate Limiting Under Retry Storms: Buckets, Windows and NGINX Admission
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Choose an overload contract before a limiter algorithm, then configure and diagnose NGINX burst admission, delay, identity scope and replica-local state.
TL;DR: Rate limiting combines an accounting model with an admission policy and an identity scope. Token buckets accumulate burst credits, leaky-bucket shaping paces departures, and sliding windows vary in temporal precision and storage cost. This article provides a complete isolated NGINX configuration with shared worker state, bounded immediate excess, explicit rejection status and diagnostic logs. It explains why rewrite shortcuts can bypass a test, why local zones are not fleet quotas, and how to distinguish limiter rejection from upstream failure. Treat delay, authenticated identity, replica count and limiter outages as parts of the reliability contract.
Start with the overload contract, not the algorithm name
A search endpoint becomes slow during a client retry storm. The gateway accepts bursts immediately, the backend saturates, and monitoring labels rejected requests as upstream 503s. Changing the rate alone will not resolve all three problems: admission, burst handling and observability are separate decisions.
Choose a limiter by the contract it must enforce. Are short bursts acceptable? Should excess work wait or fail quickly? Is the budget per IP, authenticated account or fleet? Which resource does one admitted request consume? A request-rate limit does not cap concurrency when service time rises.
This article compares token buckets, leaky buckets and sliding windows, then provides a complete local NGINX configuration. The configuration receives static review; no performance measurements or predetermined response counts are claimed.
Three accounting models, several admission policies
A token bucket accumulates credits up to capacity B and replenishes them at rate R. Each request spends a token, allowing idle time to fund a later burst. Continuous-refill and periodic-refill implementations differ; inspect the implementation before promising an exact short-interval envelope. RFC 3290 Appendix A describes token-bucket conformance and leaky-bucket shaping in packet networks. Applying these models to unit-cost HTTP requests is an adaptation.
Leaky-bucket shaping spreads departures over time. A limiter can instead track excess debt and reject requests that exceed its allowance. NGINX explicitly documents its request limiter as using a leaky-bucket method; nodelay changes admission delay without changing the configured burst bound. The label “leaky bucket” alone does not tell clients whether they will wait.
An exact sliding log tracks individual admitted timestamps within the rolling window. Expiration, counting and admission must be one atomic decision; otherwise concurrent callers can overspend the quota. It also needs unique event identities and a coherent time source. The Redis tutorial illustrates sorted-set tracking and atomic Lua evaluation.
Segmented or weighted-counter windows trade timestamp detail for smaller state. They need not enforce exactly the same boundary as a timestamp log. Microsoft's limiter documentation describes a segmented sliding window and periodic token replenishment. Define segment width, boundary inclusivity and request cost rather than treating every “sliding window” implementation as identical.
Image. Illustrative token-credit envelope, leaky-bucket shaping and exact rolling-window count
The diagram compares policies schematically. The token envelope assumes continuous refill and unit-cost requests; it is neither an execution trace nor evidence of throughput or fairness.
Separate burst tolerance from backend smoothing
NGINX's limit_req reference defines rate, burst and delay controls. Without nodelay, excessive requests within the allowance wait; requests beyond the burst allowance fail. The default burst allowance is zero. delay=N permits part of the excessive allowance without waiting and delays later excess.
With nodelay, admitted excess requests proceed immediately. Their accounting remains until it drains, so a burst is not permission for an unlimited following stream. The NGINX implementation guide explains that distinction. This policy suits a backend that can absorb a bounded spike; it does not smooth that spike.
Waiting shifts overload into connection occupancy and tail latency. Rejecting shifts recovery to the client. A queued request that expires before reaching useful backend work wastes the caller's deadline. Choose delay against the endpoint's latency budget and cap outstanding work separately.
Complete isolated NGINX example
Prerequisites are an installed NGINX 1.30.5 binary on Linux, Bash, curl and unused loopback ports 8088/8089. The official download page verifies the stable release pin. Use a fresh disposable directory and an ordinary local user; this example does not edit system configuration.
Save the following as nginx.conf. The first listener applies per-client admission and proxies to a deterministic local response on the second listener. The upstream response is a functional fixture, not a representative backend or a benchmark.
Code example. Refer to the written article for the complete code.
Start only this instance:
Code example. Refer to the written article for the complete code.
Confirm the version and that the binary supports the required HTTP modules. The command-line reference documents the alternate prefix, configuration test and signals. Port binding failures or missing modules require correction before testing admission.
The 1 MiB zone holds state for client addresses and is shared by both workers of this instance. 2r/s is a configured rate, not an observed throughput. burst=4 nodelay allows bounded excessive admission immediately; it is not a concurrency limit. 429 explicitly replaces the module's default rejection status of 503.
Keep the verification path out of rewrite shortcuts
The protected location uses proxy_pass. Do not replace it with return 200 and assume the limiter was tested: rewrite-phase handling can finish the request before the limiter's preaccess handler. The pinned module source registers that handler in NGX_HTTP_PREACCESS_PHASE.
The backend fixture can use return because it does not test admission there. Both ports bind only to loopback. A deployed backend must not expose an alternate public path that bypasses the admission layer.
For a bounded local request burst:
Code example. Refer to the written article for the complete code.
Results depend on arrival timing and existing bucket state. Inspect the 8088 records; the 8089 fixture has separate records without frontend limiter decisions. Do not report a guessed count of passes or rejections.
Diagnose limiter failures instead of blaming the upstream
A realistic failure occurs when a rejection retains the default 503 status. Clients retry it as an upstream outage, increasing offered traffic. In this disposable example, change limit_req_status 429 to 503, test and reload, then rerun the bounded burst. This is a reader-executable configuration variation, not a claimed observed incident.
The access log distinguishes limiter=REJECTED from a proxied backend error. Inspect limiter state together with upstream and HTTP status. Restore 429, test and reload:
Code example. Refer to the written article for the complete code.
The named process is selected through this prefix's PID file. Change only the disposable configuration. Clients should use bounded retries and jitter according to API semantics; immediate retry loops defeat overload protection.
For rollout observation, add limit_req_dry_run on; to the protected location. Dry run accounts excess but does not enforce admission. Its status distinguishes REJECTED_DRY_RUN and DELAYED_DRY_RUN. It cannot prove the backend would stay healthy under enforcement.
Scope, identity and failure policy are part of the limiter
This key is the direct client address. Users behind one NAT share a budget; clients with many addresses do not share one. For account quotas, derive the key from verified identity at an appropriate authenticated layer. An empty key is not accounted, so a missing identity must have an explicit reject or fallback policy.
Behind a proxy, do not trust arbitrary forwarded headers. The real-IP module requires an explicit trusted-sender configuration and is not built by default. Test the actual proxy chain before replacing the direct address.
Independent NGINX replicas keep independent zones. A local allowance is not a fleet-wide quota, and scaling replicas changes aggregate admission opportunities. The documented zone synchronization option is commercial functionality. A distributed limiter also needs an explicit outage decision: fail closed, retain a bounded local fallback or accept controlled overspend. It adds coordination latency and failure modes.
Audit inheritance: a location defining its own limit_req does not automatically retain parent limit_req directives. Reapply all intended scopes there. Observe zone pressure too: exhausted state storage can reject admission; a status code alone cannot identify why the request failed.
Stop the example and retain only useful evidence
Code example. Refer to the written article for the complete code.
Wait for this instance to exit before removing its disposable files. Keep any relevant configuration and logs for diagnosis. No system service, cloud resource or external limiter was created by these commands.
Frequently asked questions
Is NGINX nodelay unlimited bursting? No. It removes admission waiting within the configured excess allowance; it does not remove that allowance.
Does a shared zone enforce one quota across replicas? No. Worker sharing inside an instance is different from distributed quota coordination.
Is every sliding window exact? No. Timestamp logs, segmented windows and weighted counters have different temporal precision and state costs.
Can rate limiting alone prevent backend saturation? No. Expensive or slow requests can accumulate concurrently even when arrival rate is limited.