AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Kubernetes, Terraform and AWS Interviews: Prove the Broken Contract
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Diagnose selector drift, Terraform replacement proposals and EKS access failures with worked answers and a provider-free contract test.
TL;DR: Troubleshooting across Kubernetes, Terraform and AWS requires separate evidence for selection, state ownership and authorization. Ready pods can miss a Service selector, an existing cloud object can belong to another Terraform state, and valid AWS credentials do not grant Kubernetes permissions. These worked interview labs pair diagnostic boundaries with owner-controlled recovery and explicit verification. A complete provider-free Terraform example tests a rendered Service selector against declared workload labels, accepting compatible input and rejecting the wrong instance. Its local success validates the configuration contract, not live routing, cloud access or production infrastructure.
The interview lab gives you three facts: pods are Ready, Terraform proposes a replacement, and an operator receives Forbidden. "Fix the deployment" is deliberately underspecified. Each symptom can belong to a different contract, and one successful command cannot establish that the other planes are correct.
Work through the Kubernetes selection contract, Terraform's configuration/state relationship and AWS-to-EKS access identity separately. The scenarios below are illustrative. One isolated Terraform contract test was executed locally; the Kubernetes and AWS commands are reviewable diagnostic examples, not claims of a deployed lab or an observed production incident.
Lab one: Ready pods, empty Service endpoints
Question: "All checkout pods are Ready. Why is the Service failing?"
Worked answer: Confirm the intended cluster and namespace, then compare the Service selector with actual pod labels and inspect its EndpointSlices. Readiness does not make a pod match an unrelated selector. A Service can have no matching pods, matching pods that are not ready, or selected endpoints whose port/path is wrong. Establish which case exists before restarting anything. Service debugging
In the illustrative failure, workload labels contain app.kubernetes.io/instance=prod, while the Service selects stage. The mismatch explains absent selected endpoints if no other pods match. A healthy unrelated deployment does not falsify this diagnosis. Also check owner references: an accidental broad selector can route to another workload rather than simply producing no endpoints.
With namespace-scoped get/list permissions on Services, Pods and EndpointSlices, an approved operator can collect the relevant evidence using a compatible kubectl. Substitute the existing context and resources; these commands do not create diagnostic pods or invoke exec:
Code example. Refer to the written article for the complete code.
Inspect the actual selector, addresses, readiness conditions and ports, rather than assuming the displayed pod list is the Service's selected population. Kubernetes selection uses every selector key/value; extra pod labels are allowed. Named targetPort values must resolve against the selected pods' container-port names. Service selection and ports
Recovery belongs to the workload/platform owner: correct the desired selector or workload labels through the existing delivery workflow, then verify selected endpoints and a representative request. Do not patch live objects behind GitOps without coordinating reconciliation. A selector fix does not establish network-policy, application or load-balancer health.
Image. Evidence contracts across Kubernetes selection, Terraform planning and AWS/EKS authorization
A complete offline guard for the selector contract
Prevent this specific mistake in a module that renders a Service for declared workload labels. The following configuration is intentionally provider-free: it contains no managed resources, data sources, remote modules or backend. Its output precondition blocks an incompatible selector; a nonblocking warning would be insufficient for this contract. Custom conditions
Use a new empty directory and Terraform 1.9.8 for the demonstrated compatibility run. Current documentation was checked separately; 1.9.8 is the tested version, not a claim about the newest release. Save this as main.tf:
Code example. Refer to the written article for the complete code.
Save contract.tftest.hcl alongside it. All runs explicitly use plan; ordinary Terraform tests default to apply. Negative runs expect the output's precondition to fail, so passing means the bad input was rejected. The missing-key/null case catches an important trap: two nulls can compare equal without establishing a real label match. Terraform tests
Code example. Refer to the written article for the complete code.
From that isolated directory, run terraform init -backend=false, terraform validate and terraform test. The validation and all three tests passed locally with Terraform 1.9.8. This verifies declared-label matching and Terraform evaluation; it does not contact Kubernetes or prove actual EndpointSlice selection, readiness, targetPort, networking or controller behavior. The extra version label demonstrates subset matching rather than exact map equality.
For cleanup, remove only the isolated directory after retaining its files and validation log; there is no cloud-resource destroy operation. Integrate the guard with the module's actual workload-label source before using it in delivery. Passing a test on independently supplied label maps does not prove those maps describe the deployed workload.
Lab two: Terraform wants to replace an existing object
Question: "The resource still exists in AWS. Why would Terraform replace or recreate it?"
Worked answer: Verify the account, Region, backend, workspace, resource address and exact proposed action. Existence in AWS is not ownership by the selected state. A changed address, wrong workspace or replacement-triggering configuration difference requires different recovery. Read the relevant plan details; do not infer the cause from a summary count.
If an address changed during a refactor, an owner-reviewed moved block can preserve the state association when appropriate. If an object was never managed in this state, import requires deliberate configuration and ownership reconciliation. Neither mechanism should hide an actual immutable-field change. Moved blocks
An approved speculative plan can clarify proposed actions, but it can read remote APIs and acquire the backend lock. Review configuration and permissions before running it against an existing environment. With -detailed-exitcode, zero means no changes, one means an error and two means a proposed difference. Exit two is not a failed Terraform command. A saved plan may contain sensitive values; protect it as an operational artifact. Plan behavior
Refresh-only planning proposes reconciling Terraform's records with remote changes, not repairing infrastructure. Applying that proposal can write state. Avoid using terraform refresh as a diagnostic shortcut: it implicitly approves a refresh-only state update. Do not disable locking to get past an unexplained active writer. Preserve evidence and resolve ownership with the state owner. Refresh semantics
Recovery check: After correcting the address, configuration or state association through its owner, generate a fresh plan and explain every remaining action. Do not apply the earlier replacement proposal merely because an interview prompt says "fix it."
Lab three: AWS credentials work, kubectl is Forbidden
Question: "STS returns an identity. Should I add administrator access?"
Worked answer: First establish whether AWS credentials are valid and which principal kubectl actually uses. GetCallerIdentity describes the AWS caller; it does not grant Kubernetes permissions. The kubeconfig exec plugin can use another profile or assume another role. Inspect its configuration privately, including the selected context and role, without displaying credential material. Caller identity
An owner-approved identity check, using the same intended profile and Region, is:
Code example. Refer to the written article for the complete code.
A Kubernetes Unauthorized response points toward authentication; Forbidden ordinarily means an authenticated request lacked authorization. Read the actual response and requested verb, resource and namespace before diagnosing the path. EKS cluster authentication mode determines whether access entries, the legacy aws-auth mapping or both apply. Access entries and their policies are managed separately from Terraform's AWS execution permissions. EKS access entries
An access entry's Kubernetes groups do not create RBAC permissions by themselves; appropriate RoleBindings/ClusterRoleBindings must exist. EKS access policies can also provide scoped permissions. Evaluate both grant sources: a narrow access policy does not revoke broader RBAC permissions. Verify the policy association and namespace scope with the cluster-access owner, and remember that spelling a nonexistent group or namespace can produce an ineffective grant. AWS IAM policies and EKS access policies are different authorization systems. Access-policy scope
Recovery check: Request the smallest grant for the intended task through the access owner, then verify that the intended operation succeeds and an unrelated privileged operation remains denied. A broad admin grant destroys the evidence about which boundary was missing.
How to present the result in an interview
State the observed symptom, the contract being tested, the evidence that separates hypotheses, the owner-authorized recovery and the verification that could falsify your explanation. The offline selector guard is an executed check; the live troubleshooting outcomes above remain illustrative. No AWS resources, cluster objects or production state were changed.