AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
EKS Network Failures: Separate DNS, IP Allocation and Conntrack
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Trace failures from one affected pod through address allocation, DNS routing and distinct kernel and EC2 connection-tracking boundaries.
TL;DR: EKS DNS timeouts, stalled pod creation and connection failures can share symptoms while failing at different boundaries. Start with the affected pod and its placement, distinguish scheduling from CNI allocation, then compare its configured resolver, the DNS Service and ready CoreDNS endpoints. Inspect subnet and node capacity separately, and distinguish Linux conntrack occupancy from EC2 connection-tracking allowances. This workflow provides bounded commands for existing approved containers, diagnostic permissions, source-based recovery and verification. No cloud incident or recovery experiment was executed; illustrative failures are clearly identified and configuration changes remain with their owners.
A DNS timeout is a symptom, not a component diagnosis
After a rollout, some EKS pods cannot resolve a dependency. Other replicas work, and newly scheduled pods stall before their containers start. Restarting CoreDNS may temporarily change the affected nodes without explaining either failure. You need to distinguish address allocation, DNS resolution and connection tracking before selecting a recovery action.
Start with one affected pod, its node and the incident window. Compare a healthy replica on the same node and one on another node when available. Preserve that placement: a fresh debug pod can land on healthy infrastructure and produce a convincing but irrelevant success. This article provides a bounded diagnostic workflow for existing Linux EC2 workers using the Amazon VPC CNI and CoreDNS. No cluster incident or recovery experiment was executed for this article.
Separate pod creation from the DNS request path
A pod without an assigned address cannot demonstrate an application DNS failure. Read its phase, conditions and events first. A scheduled pod with FailedCreatePodSandBox and a CNI address-allocation error belongs to the IP-management investigation. A Pending pod without a node may instead have resource, affinity or scheduling constraints. The VPC CNI troubleshooting guide describes address shortages and allocation backoff as distinct causes of startup delay.
For a running pod, inspect its actual /etc/resolv.conf, dnsPolicy and dnsConfig. Follow the configured resolver: NodeLocal DNSCache, a custom resolver and the usual cluster DNS Service have different first hops. For normal cluster DNS, compare the Service address with a ready CoreDNS endpoint. Then distinguish internal cluster names from names CoreDNS forwards upstream. The Kubernetes DNS debugging guide provides the Service, EndpointSlice and configuration checks.
Image. Pod allocation gate, DNS request path and two independent tracking layers
Collect evidence from the affected network namespace
Save the following as collect-dns-path.sh. Prerequisites are Bash, jq, compatible kubectl, and an existing approved affected pod/container that already contains dig. Set all five required inputs explicitly. Use a known permitted DNS name without a trailing dot; the script appends one to avoid search-domain expansion. Set QUERY_TYPE=AAAA for an IPv6 address check; the default is A. It performs four bounded dig invocations with two-second query timeouts and bounded API reads. It creates no pod, node, AWS resource or privileged session.
The full collector requires pod/event reads in the workload namespace, Service/EndpointSlice reads in kube-system, and explicit create permission on pods/exec for the selected container. Have the platform operator run it with those scoped grants; do not expand a workload operator's access merely to satisfy the script. Exec is broader than read access: authorize this command set, not an unrestricted shell for an automated investigator. Handle captured configuration and logs as operational data, not public attachments.
Code example. Refer to the written article for the complete code.
Run it once with operator-selected inputs, for example CONTEXT=approved-context NAMESPACE=payments POD=affected-pod CONTAINER=app DNS_NAME=kubernetes.default.svc.cluster.local bash collect-dns-path.sh > dns-evidence.txt. Replace the domain if your cluster uses another suffix. Repeat with one permitted external name only when forwarding is the suspected boundary. No tool is installed inside the workload; if dig is absent, use an already approved diagnostic path rather than changing the container during collection.
A Service query failing while a ready endpoint query succeeds narrows the Service-routing or policy path; it does not prove kube-proxy is broken. A failing endpoint query can involve policy, routing, CoreDNS health or the selected endpoint. TCP succeeding while UDP fails identifies a transport-dependent difference, not a unique conntrack diagnosis. The UDP probes use +ignore to prevent automatic TCP retry on truncation, as described in the BIND dig manual. Inspect response codes as well as timeouts: NXDOMAIN differs from SERVFAIL. Different CoreDNS replicas may behave differently, so retain the selected endpoint and repeat against another ready replica before generalizing.
Address exhaustion has two capacity boundaries
For an address-allocation failure, the platform operator records installed component versions and configuration before interpreting IPAM evidence. These reads apply to the stated VPC CNI/CoreDNS scope; retain the image tags or digests rather than substituting a presumed latest version.
Code example. Refer to the written article for the complete code.
Then map the node's actual allocation subnet and ENIs. Custom networking or subnet discovery can make assumptions based on the cluster subnet list wrong. AWS explains these choices in its IP-utilization guide. An AWS read-only investigator needs ec2:DescribeSubnets and ec2:DescribeNetworkInterfaces; those actions require Resource: "*" under the EC2 authorization reference, so constrain credentials and scope requests to the approved account and Region.
Code example. Refer to the written article for the complete code.
Subnet free-address count is only one boundary. The instance's ENI/IP limits, allocated warm pool and kubelet pod limit also matter. Prefix mode additionally requires contiguous /28 IPv4 blocks: scattered free addresses can coexist with InsufficientCidrBlocks. AWS's Linux prefix-mode guidance describes reservations and migration considerations. Prefix delegation requires VPC CNI 1.9.0 or later; verify the installed add-on and instance support before proposing it.
Do not minimize warm targets blindly. Too little spare capacity can increase EC2 allocation calls and throttling during bursts. Stop an approved rollout from adding more pressure, preserve healthy replicas and route new capacity only to a verified usable subnet. Long-term subnet expansion, custom networking or IPv6 requires a reviewed network change; flipping CNI settings during the incident is not a substitute for that design.
Distinguish Linux conntrack from EC2 tracking allowances
Linux nf_conntrack_count and nf_conntrack_max describe the host kernel table. EC2 security-group connection tracking is a separate allowance, exposed through ENA metrics such as conntrack_allowance_exceeded and conntrack_allowance_available. Increasing a Linux sysctl cannot raise an EC2 instance allowance. AWS documents the latter in security-group connection tracking.
Use existing node telemetry to correlate kernel occupancy/drops and ENA metrics with affected nodes and the incident window. ENA *_allowance_exceeded metrics are cumulative packet counters since driver reset: look for increases, not a historical nonzero value alone. In contrast, conntrack_allowance_available is a remaining-connection-capacity gauge; it requires ENA 2.8.1 or later on supported Nitro instances. Metric availability depends on driver and instance support. Absence is not zero utilization.
Likewise, linklocal_allowance_exceeded concerns a shared allowance for Amazon DNS, metadata and time-service traffic. It is not a CoreDNS request counter. AWS's network monitoring guide explains why Amazon DNS throttling may be absent from resolver query logs and VPC Flow Logs. Correlate CoreDNS placement, upstream forwarding and ENA drops before increasing replicas.
Recover the demonstrated boundary and prove it stayed healthy
Consider an illustrative incident where cluster names resolve but forwarded names time out only through CoreDNS replicas on one worker, with rising link-local drops there. Distribute the DNS workload across suitable healthy nodes and reduce avoidable query amplification after confirming this path. Re-test the original pod, internal and external names, both transports and application errors. Scaling replicas onto that same worker may not remove the bottleneck.
NodeLocal DNSCache can reduce DNS traffic and Linux UDP conntrack exposure, but needs a compatible deployment and observed cache behavior; it does not create subnet addresses or increase hardware allowances. If a CoreDNS configuration change instead removed ready while the installed add-on probes /ready, restore the supported Corefile through its owner. EKS CoreDNS documentation identifies this readiness requirement.
For Auto Mode, follow its managed troubleshooting path rather than assuming an editable aws-node DaemonSet or SSH access. Record installed component versions, the failing boundary, approved recovery and repeated evidence. The collection example creates nothing to tear down; retain restricted evidence under your incident policy and remove local copies when retention expires.
Questions that prevent the wrong remediation
Does a Pending pod prove IP exhaustion? No. Distinguish unscheduled pods from scheduled sandbox allocation failures using events.
Does a high subnet free-address count rule out allocation failure? No. Prefix fragmentation and node-level limits remain possible.
Will increasing nf_conntrack_max fix an EC2 allowance drop? No. The kernel table and EC2 tracking allowance are different boundaries.
Were these commands executed against a cluster? No. The workflow was source-verified and statically reviewed; the incident examples are illustrative.