AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Kubernetes Memory QoS: Why a Running Pod Can Stall Before OOM
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Connect requests and limits to effective cgroup controls, distinguish reclaim stalls from OOM, and review Kubernetes 1.37’s conservative Memory QoS defaults.
TL;DR: Requests, limits and Memory QoS control different boundaries: scheduling, CPU execution, memory reclaim and hard-limit failure. Kubernetes 1.37 enables the MemoryQoS feature gate by default, but protection and early throttling still require explicit kubelet policy. This article provides a bounded Burstable web-service example, effective cgroup inspection commands and calculated thresholds with their limits. It explains why an allocated memory budget is not exclusive RAM, how hierarchy affects protection, and how to separate reclaim latency from cgroup OOM, node-wide OOM and eviction before choosing a recovery or rollout policy.
A Pod can get slow before it gets killed
A service's latency rises, but it has no OOMKilled containers and average CPU usage looks ordinary. The response is often to raise its memory limit. With Memory QoS, that can miss the mechanism: the kernel may be reclaiming memory and throttling allocations below the hard limit.
Requests, limits and reclaim protection answer different questions. Requests influence placement. CPU limits constrain execution time. Memory limits establish a hard boundary that can lead to OOM termination when reclaim cannot satisfy allocation. Memory QoS adds protection and throttling controls; it does not turn memory into an unlimited resource or guarantee application latency.
The engineering task is to connect the manifest, kubelet policy, effective cgroup files and application symptoms. A green Pod phase is not enough evidence.
The 1.37 default is more conservative than the feature name
Memory QoS graduated to beta in Kubernetes 1.37, with its feature gate enabled by default. However, memoryReservationPolicy defaults to None and memoryThrottlingFactor defaults to null. Default-on does not mean the kubelet automatically configures request-based protection and early throttling.
This differs from the 1.36 alpha behavior, whose throttling factor default was 0.9. An explicitly configured 0.9 remains an explicit opt-in after upgrading. Inspect effective kubelet configuration, not just the Kubernetes version or feature-gate state.
On cgroup v2, the relevant controls have different consequences:
Table. Columns: Control, Meaning, Operational implication.
Row 1. Control: memory.min; Meaning: Hard reclaim protection within effective hierarchical boundaries; Operational implication: Overprotecting workloads can leave too little reclaimable memory elsewhere.
Row 2. Control: memory.low; Meaning: Best-effort reclaim protection; Operational implication: Memory above the effective boundary remains reclaimable; protection is not an exclusive physical reservation.
Row 3. Control: memory.high; Meaning: Reclaim/throttling threshold; Operational implication: Allocation latency can rise without an OOM kill.
Row 4. Control: memory.max; Meaning: Hard usage limit; Operational implication: If reclaim cannot bring usage within the limit, cgroup OOM can occur.
End of table.
The Linux cgroup v2 documentation defines these semantics, including ancestor constraints. A container's local value is only part of the hierarchy.
Requests are policy inputs, not spare capacity
For the conventional container-level resource model, Kubernetes schedules against requests, not instantaneous usage. A 256 MiB request with a 1 GiB limit still allows the container to consume substantially more than its scheduled request. Setting that gap across many services creates memory overcommit that must be budgeted at the node level.
With TieredReservation, Pod QoS class influences protection: Guaranteed workloads receive memory.min, Burstable workloads receive memory.low, and BestEffort workloads receive neither. In the container-level model, Guaranteed requires matching CPU and memory requests/limits for every container. Matching only memory is insufficient.
These classes do not exempt an application from its limit, node-pressure eviction, or poor sizing. Nor does a toleration create memory capacity. Keep admission defaults, namespace LimitRanges and injected sidecars in the review: the admitted Pod may differ from the manifest a service owner submitted.
Inspect a bounded Burstable service
Use a disposable Kubernetes 1.37.0 test node, matching kubectl, Bash, cgroup v2 and a compatible CRI runtime. Kubernetes recommends kernel 5.9 or higher because earlier kernels can encounter a known livelock issue with memory.high throttling. Use a supported distribution/kernel combination; do not enable this node-wide policy as an unreviewed production experiment.
Have the node administrator merge these fields into the existing kubelet configuration. This is a configuration fragment, not an API object to submit with kubectl. Preserve all other settings and the original values, and restart the kubelet through the distribution's supported procedure. On a dedicated systemd test node, that is typically sudo systemctl restart kubelet after updating its configured file.
Code example. Refer to the written article for the complete code.
Label only the prepared test node so the example cannot drift to an unconfigured node:
Code example. Refer to the written article for the complete code.
Save this as memoryqos-web.yaml. It uses a versioned NGINX image pinned to an immutable image-index digest. The container has a 256 MiB request and a 1 GiB limit, making this a Burstable Pod.
Code example. Refer to the written article for the complete code.
Apply it and inspect the controls from inside the container's cgroup namespace:
Code example. Refer to the written article for the complete code.
The /sys/fs/cgroup root must represent this container's cgroup for that inspection to describe its local controls. If the runtime exposes a different hierarchy, resolve the process's cgroup through /proc/self/cgroup and mount information instead of treating root values as container values. Compare Pod and ancestor cgroups from the node when diagnosing effective protection.
Read the thresholds without inventing a benchmark
With the explicit policy above, the expected local Burstable values are memory.low=268435456, memory.min=0, and memory.max=1073741824. The v1.37.0 kubelet implementation calculates the high threshold from the space between request and limit:
Code example. Refer to the written article for the complete code.
The implementation rounds down to the host page size. With 4 KiB pages, the expected value is 993210368 bytes. This is configuration arithmetic, not a measured operating threshold or a claim that latency changes at exactly that usage.
Image. Annotated memory budget with reclaim protection, early throttling and hard limit
An idle web server is unlikely to approach these boundaries. That is intentional: this example checks configuration and observability without deliberately exhausting a node. memory.current includes more than an application's heap, so correlate it with memory.stat before attributing the entire charge to application objects.
Diagnose reclaim latency separately from OOM
Consider an illustrative cache service that starts stalling after an operator explicitly enables a throttling factor. Its memory.events.local high counter rises and memory pressure increases while OOM counters and restart count remain unchanged. That points toward reclaim/throttling, but it does not prove causality by itself. Compare counter deltas and pressure with latency over the same interval, and rule out CPU quota throttling through cpu.stat.
For an OOM case, inspect termination reason, restart history, oom/oom_kill counters and node events. Cgroup memory-limit OOM, node-wide OOM and kubelet eviction are different mechanisms; node-pressure eviction documentation explains the kubelet's separate response to scarce resources.
Recovery should follow the diagnosed mechanism. For reclaim stalls, reduce cache size or concurrency, review working-set headroom and reassess the explicit high policy. For limit OOM, correct the application's memory budget or resize requests and limits together with a node-capacity review. Raising only the limit can enlarge overcommit and move the failure to a neighbor.
Any policy rollback is node-wide. Restore saved kubelet settings through the normal maintenance procedure, then recreate the isolated test Pod and verify the resulting cgroups. Do not assume a kubelet configuration change instantly rewrites every existing container's controls.
Roll out policy with an exit condition
Define application latency and error-rate acceptance criteria before changing the node pool. Canary the policy, observe reclaim and pressure alongside OOM/eviction evidence, and retain a pool with the previous configuration for recovery. Separate service-owned memory budgets from platform-owned reclaim policy.
Clean up the example and remove only the label you added:
Code example. Refer to the written article for the complete code.
Finally restore the saved kubelet configuration on the dedicated test node using its supported restart procedure. Requests, limits and Memory QoS work together only when the admitted workload and effective node policy are both understood.
Frequently asked questions
Does default-on Memory QoS in 1.37 automatically enable memory.high? No. Early throttling requires an explicitly configured memoryThrottlingFactor; reservation is a separate policy choice.
Is memory.low a dedicated physical-memory reservation? No. It provides best-effort reclaim protection subject to the cgroup hierarchy, not exclusively assigned RAM.
Does memory.high replace the memory limit? No. It adds reclaim/throttling before the independent hard-limit boundary.