AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Deployment Is Green, Customers See 502: Three DevOps Interview Cases
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Work through deployment acceptance, ALB versus target errors and shutdown races using bounded evidence and explicit recovery ownership.
TL;DR: A green pipeline and Running pods do not establish a healthy customer request path. These worked DevOps interview scenarios separate deployment acceptance, ALB-generated versus target-generated errors, and failures during replica termination. Start with revision identity, correlate request and target evidence, and collect existing Kubernetes and ALB state before changing resources. Readiness gates, application shutdown, endpoint conditions and target draining solve different parts of the handoff. Choose a scoped recovery through the deployment owner and verify customer outcomes afterward. All scenarios are illustrative; the evidence script received static review without cloud execution.
The interviewer says: "The pipeline is green, the Deployment rolled out, and customers are getting 502s. What do you do?" A list of commands is an incomplete answer. The engineering decision is whether to restore traffic, stop a rollout, repair a configuration or investigate a dependency without destroying the evidence that separates those choices.
Use one concrete architecture: an Application Load Balancer (ALB), AWS Load Balancer Controller, IP-mode targets and an EKS application serving HTTP/1.1. These worked scenarios are illustrative, not reconstructed production incidents. The evidence collector below was statically reviewed; no cluster commands, cloud requests or failure experiment were executed.
Scenario one: a green rollout is not a traffic acceptance test
Question: "All pods are Running. Why investigate the deployment?"
Worked answer: Running describes pod lifecycle, not customer request success. Compare the intended commit and image digest with actual container imageID, Deployment generation, observed generation, conditions, updated replicas and availability. Check whether the pipeline waited for the intended revision rather than a different rollout. A successful controller rollout does not validate the ALB listener, target registration or a representative customer transaction. Deployment semantics
Readiness should mean the replica can accept its intended traffic, while startup and liveness have different purposes. A health endpoint that answers before route initialization can create false confidence. Conversely, making readiness depend on every optional dependency can withdraw every replica during a shared dependency incident. State which dependencies are essential and why. Probe behavior
In IP mode, a Kubernetes-ready pod and an ALB-healthy target are separate observations. AWS Load Balancer Controller's pod readiness gate can connect target health to pod readiness, but injection depends on the namespace label and matching Service/TargetGroupBinding configuration at pod creation. Inspect actual conditions and the installed controller version; adding a namespace label does not retrofit an existing pod. Readiness gates
The answer should end with an acceptance boundary: "I would gate promotion on the correct revision becoming available and the external request path meeting its error and latency criteria over an agreed observation window." Do not invent a universal five-minute window.
Scenario two: identify who generated the 502
Question: "A client saw 502. Is the application broken?"
Worked answer: Identify the response-producing hop before choosing a fix. In ALB access logs, compare elb_status_code with target_status_code, target address, request path and trace identifier. Check actions_executed: a fixed-response listener rule does not forward to a target. A target status of 502 means an HTTP response was received from a target; that target might itself be a reverse proxy. A dash in the target-status field indicates no target response was recorded, not a proven security-group fault. Access logging is best effort, so absence of one record is not proof that no request arrived. Access-log fields and limits
Compare ALB-generated and target-generated error metrics over the same window and dimensions. Fleet totals can conceal a single target, route or Availability Zone. Use request IDs and application logs to narrow the failing cohort before aggregating latency. ALB metrics
For this architecture, an ALB-generated 502 can result from a target reset, premature connection close, malformed response or backend TLS handshake failure. An empty or wholly unused target group is a documented 503 case; a backend response exceeding the ALB's wait can be 504. The number alone does not select a network, application or deployment owner. ALB troubleshooting
If only one route fails, inspect that route's application/proxy response contract. If one newly introduced replica fails, compare its image, configuration and listening port against healthy replicas. A successful health check validates its configured path and matcher, not every customer request. Also remember that ALB can fail open when all registered targets are unhealthy; "unhealthy" does not automatically mean no customer traffic reaches them. Target health behavior
Collect one bounded evidence package before changing anything
Prerequisites: Bash 4+, jq 1.6+, AWS CLI v2 and kubectl compatible with the actual cluster. The example uses apps/v1 Deployments and discovery.k8s.io/v1 EndpointSlices. Set the variables to existing authorized resources. SELECTOR must match the deployment's actual selector; POD and CONTAINER identify one affected replica, not a newly created debug pod. Obtain the ALB and target-group ARNs from the platform owner.
The collector needs namespace-scoped get/list access to Deployments, ReplicaSets, Pods, Services, EndpointSlices and Events, plus get on pods/log. It does not use pods/exec, create containers or change resources. AWS permissions are elasticloadbalancing:DescribeTargetHealth, elasticloadbalancing:DescribeTargetGroupAttributes and elasticloadbalancing:DescribeLoadBalancerAttributes; their authorization-reference entries require Resource: "*". Use an approved diagnostic role rather than assuming the ARN command argument itself restricts IAM. ELB authorization reference
Code example. Refer to the written article for the complete code.
Save as collect-deployment-502.sh, review the identifiers, then run bash collect-deployment-502.sh with the owner's permission. A failure stops collection and leaves partial evidence; it does not produce a false "all checks passed" result. CLI timeouts bound individual connection/read waits, not the entire investigation. SDK/CLI retries may add time. Target-health command
Store the printed directory privately: pod specifications and logs can contain sensitive configuration. Preserve it with the incident record, then delete that exact directory under your retention policy. Do not upload raw evidence to an interview portfolio.
Interpret imageID, owner references and termination timestamps together. EndpointSlice readiness/termination conditions help explain selection; they do not prove an ALB controller has already deregistered the corresponding IP. Obtain existing ALB access logs and controller evidence separately through their owners. Service debugging, EndpointSlice conditions
Scenario three: errors appear only while old replicas exit
Question: "Rollback fixes it. How do you prove the shutdown hypothesis?"
Worked answer: Correlate failed requests with old pod termination, target draining state and application shutdown logs. Compare the configured ALB idle timeout, application keep-alive behavior and outstanding-request lifetime. A target closing an outstanding connection can produce 502; elapsed deregistration delay can also interrupt a request. Those are hypotheses until the timing and target identity align. Connection and deregistration failures
Image. Traffic handoff timeline separating admission of a new replica from draining an old replica
The qualitative diagram shows required coordination, not measured durations or a guaranteed universal ordering. A new replica needs both application readiness and the relevant traffic-admission state. An old replica must retain enough shutdown budget for in-flight work while discovery and target deregistration converge.
preStop consumes the pod's termination grace period before the normal TERM signal is sent; it is not free extra time. Terminating endpoints remain represented with conditions, including readiness false for ordinary traffic selection. Implement graceful application shutdown and verify its signal handling. Do not add an arbitrary sleep as proof of correct draining. Pod termination, Target deregistration
State the recovery decision and the evidence that would change it
A strong answer separates mitigation from diagnosis. Stop promotion and restore a known compatible release through the deployment owner when customer impact warrants it. Check database/schema compatibility before reverting application code; a rollout undo does not reverse an external migration. Preserve the failed revision and request evidence.
Kubernetes reports a progress deadline failure but does not automatically roll back the Deployment. In a GitOps-managed environment, coordinate desired-state recovery with its owner so reconciliation does not immediately reintroduce the bad revision. Deployment failure behavior
After repair, validate the external route, target cohort and representative request types, including a subsequent controlled rollout. Success means customer errors recover and the hypothesized failure disappears without breaking another route. A healthy target count alone is insufficient.
Should I increase timeouts first? Only after identifying which hop timed out and whether longer waiting is acceptable to the request deadline.
Should I restart every pod? That can erase the failing cohort and increase impact; choose a scoped recovery with its owner.
What distinguishes a senior interview answer? It connects evidence to a reversible decision, names ownership and states what would falsify its hypothesis.