AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Incident Response Playbooks for Kubernetes and EKS: Evidence, Containment, and Recovery
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Build Kubernetes and EKS incident playbooks that preserve evidence and contain the actual attack path. Work through pod deletion, compromised workloads, stolen identities, and safe recovery with practical commands.
TL;DR
Effective Kubernetes and EKS incident response connects customer impact, workload state, identity activity, and preserved evidence before responders change production. This guide develops practical playbooks for unexpected pod deletion, compromised containers, stolen service account credentials, and suspected node compromise. It explains scoped evidence collection, Kubernetes audit and CloudTrail correlation, network policy limitations, temporary credential containment, and recovery from trusted artifacts. Use explicit decision gates, separate investigation from destructive actions, and test each containment control in a representative environment. Measure response milestones and recovery quality rather than assuming a successful command proves the incident is resolved.
A playbook should answer the next decision
Deleting a suspicious pod can restore a service while destroying the only remaining evidence of how an attacker entered it. Leaving it running can preserve evidence while allowing continued data theft. Kubernetes and EKS incident response needs an explicit decision about that trade-off, supported by impact, identity, and telemetry.
An incident response playbook is a scenario-specific operating procedure: its trigger, scope, evidence requirements, containment options, decision owner, recovery gates, and escalation path should be clear enough for an on-call engineer to use under pressure. A generic instruction such as “review the deployment and restore the pod” leaves the difficult decisions unanswered.
This guide expands that foundation into four practical scenarios. Examples use Bash, AWS CLI v2, kubectl, and jq on an authorized responder workstation. Substitute your account, region, cluster, namespace, and resource names. Collection examples read state; containment and recovery examples change it and must follow your incident authority and change process. These are reference patterns, not results from an executed production incident.
Start by comparing your environment against the EKS security responsibility guidance: a managed control plane does not make workload configuration, identities, or customer data someone else's responsibility.
Image. Kubernetes and EKS incident response flow showing impact assessment, evidence collection, separate containment decisions, trusted recovery, and verification
Response is a set of decisions with observable completion gates. Active harm can require containment while evidence collection continues.
Define incident scope before you define commands
Keep the original playbook foundations: identify threats, assign roles, build a repeatable template, and exercise it regularly. Make each one operational.
Table. Columns: Field, Example that a responder can act on.
Row 1. Field: Trigger; Example that a responder can act on: Unexpected production workload deletion plus an unfamiliar API identity; or suspicious runtime execution with access to customer data.
Row 2. Field: Scope key; Example that a responder can act on: Account ID, region, cluster ARN, namespace, pod UID, node provider ID, workload owner UID, incident ID.
Row 3. Field: Impact; Example that a responder can act on: Affected user journey, error rate, affected tenant or data class, ongoing exfiltration evidence.
Row 4. Field: Decision owner; Example that a responder can act on: Incident commander chooses availability trade-offs; security lead chooses compromise scope and evidence priorities.
Row 5. Field: Execution owner; Example that a responder can act on: Platform engineer performs scoped Kubernetes/AWS changes; service owner validates application correctness.
Row 6. Field: Evidence owner; Example that a responder can act on: Records collector, command, UTC time, origin, hashes, storage location, and access controls.
Row 7. Field: Containment gate; Example that a responder can act on: Attack path tested and blocked; no new unauthorized activity observed within a stated observation window.
Row 8. Field: Recovery gate; Example that a responder can act on: Fixed cause, trusted deployment, correct data, healthy user journey, and independent verification.
End of table.
Use a severity matrix tied to business impact rather than alert names. An illustrative highest-severity trigger is confirmed unauthorized production data access or widespread critical-service failure. A lower-severity investigation might cover a suspicious event with no established compromise. Your organization must choose actual response targets and escalation timings; none of the examples here are universal SLAs.
Avoid treating “cloud-native application security governance” as a special standardized framework without naming a real standard. The useful idea is concrete: prevention during development, admission and deployment controls, runtime detection, and tested incident procedures should share accountable owners and evidence.
Prepare visibility before an incident happens
For EKS, explicitly configure the control plane log types you need. The following preparation command enables audit and authenticator exports; consider the other control plane types based on investigative requirements. It changes cluster logging, incurs storage/ingestion costs, and should be configured before an incident. Enabling it later cannot recreate missing historical records. See EKS control plane log export.
Code example. Refer to the written article for the complete code.
Record retention, central export destination, encryption, access policy, and a person who checks delivery. A configured switch is insufficient: perform a benign test operation and confirm its record arrives in the intended archive. Alert on failed ingestion and unexpected configuration changes.
Control plane exports are disabled by default and delivered on a best-effort basis. Treat missing records as an evidence limitation; corroborate with other sources rather than interpreting an empty search as proof that no action occurred.
Each source answers a different question:
Table. Columns: Evidence, Question it can answer, Important boundary.
Row 1. Evidence: Kubernetes API audit; Question it can answer: Who requested a resource operation, against what object, and with what result?; Important boundary: It does not capture every process action inside a container.
Row 2. Evidence: Pod/application logs; Question it can answer: What did the application report?; Important boundary: An attacker or crash may suppress logs; rotation and deletion can remove local history.
Row 3. Evidence: Runtime sensor; Question it can answer: Which process, file, or network behavior was observed?; Important boundary: Coverage depends on supported compute and agent health.
Row 4. Evidence: CloudTrail; Question it can answer: Which AWS principal called an AWS API?; Important boundary: Kubernetes resource operations and AWS API events are different streams.
Row 5. Evidence: Network telemetry; Question it can answer: Which connections were observed?; Important boundary: Collection scope and aggregation affect what can be inferred.
End of table.
Kubernetes auditing defines metadata, request, and response recording levels. A metadata-only event cannot tell you the complete request body. Sensitive request content requires controlled handling; absence from an audit record does not prove it was never read.
The default CloudTrail event history covers 90 days of regional management events; it does not include data events. Configure the relevant data-event collection before relying on it to investigate object-level access. Make cross-account and cross-region search boundaries explicit.
GuardDuty Runtime Monitoring observes runtime behavior using a security agent and does not support EKS workloads on Fargate. Verify actual coverage instead of assuming that an account-level enablement setting covers every pod.
Start with a bounded evidence bundle
First verify where your commands will run. An alert's cluster name and your current kubeconfig context can differ. Confirm the AWS account, cluster ARN, and API endpoint against the incident record before collecting or changing resources.
Code example. Refer to the written article for the complete code.
The following example collects an existing pod and its namespace context. If the pod has already been deleted, begin with surviving controller records, centralized logs, and audit history instead; this script requires a successful pod identity fetch. It deliberately excludes Secret objects. Pod specifications can still contain literal environment secrets, annotations, or personal data, so use an encrypted responder volume, protect the resulting directory, and copy it only to approved evidence storage. Permission failures and unavailable artifacts must be recorded, not silently discarded.
Code example. Refer to the written article for the complete code.
--previous targets an available previous container instance, not an arbitrary history of deleted pods. Events are short-lived operational hints; do not treat them as a durable audit archive. The Kubernetes pod debugging guide explains the useful distinction between object state, events, and container logs.
Record the hash manifest separately in the incident system. Hashes detect later byte changes; they do not prove that a compromised source reported truthful data. Maintain collector identity, custody transfers, and original timestamps. Do not paste the bundle into a chat room.
Playbook 1: unexpected pod deletion or availability loss
Trigger: a pod disappears alongside degraded availability, an unexpected deletion actor, or suspicious changes. A rollout or a controller replacing a failed pod is not automatically a security incident.
First decision: is this a normal state transition, a reliability failure, or an unauthorized action? Investigate the owner and replacement workload before trying to recreate the vanished pod manually.
Code example. Refer to the written article for the complete code.
The selector app=payments-api is illustrative. Use the Deployment's actual selector; an inaccurate selector can exclude affected pods or include unrelated workloads. Compare image digest, owner references, rollout revision, desired replicas, available replicas, and deletion timestamps. Inspect whether replacements are Pending, repeatedly crashing, unable to pull an image, or failing readiness.
For an EKS audit log group /aws/eks/<cluster-name>/cluster, select the incident time window in CloudWatch Logs Insights and query completed deletion requests:
Code example. Refer to the written article for the complete code.
This is a starting query, not a complete incident detector. Also inspect create against the eviction subresource, changes to the owning Deployment, scaling operations, and controller activity. Expand the time window around the first impact. The EKS audit analysis guidance recommends using authorization metadata and resource activity to explain API behavior.
Do not infer a human attacker from a username alone. Service accounts, controllers, assumed roles, automation, and impersonation can complicate attribution. A user agent is client supplied; an IP address can be a proxy. Record these as correlation clues, then verify the identity and approved deployment history.
A PodDisruptionBudget constrains supported voluntary evictions; it is not a guarantee against direct pod deletion, involuntary node failure, or every rollout. Check the exact operation rather than assuming “the PDB should have stopped it.” See Kubernetes disruptions.
Recovery gate: the intended controller is producing healthy replicas from the approved configuration, the customer journey works, and the deletion actor or reliability cause is understood. If the deleting identity is still active and unexplained, restored replicas are a temporary availability improvement rather than proof of security recovery.
Playbook 2: a compromised container or workload
Trigger: credible evidence of unwanted execution, suspicious outbound activity, tampered files, or exploited application behavior.
Scope expansion: use the pod UID, runtime image ID, workload owner, service account, and node to locate related exposure. A mutable tag is insufficient evidence of identical image content. Collect regular, init, and ephemeral container status; compare resolved image IDs where available.
Code example. Refer to the written article for the complete code.
Treat sibling pods as exposed until the exploit path is understood; do not claim every sibling is proven compromised merely because it shares a Deployment. Determine whether the vulnerability is in the image, configuration, input path, mounted volume, or stolen credential. Those imply different remediation scopes.
Network containment must be verified
This illustrative policy selects an incident label and permits no ingress or egress itself:
Code example. Refer to the written article for the complete code.
Applying the file and labelling the exact affected pod are production changes:
Code example. Refer to the written article for the complete code.
A standard Kubernetes NetworkPolicy that allows no traffic does not override other selecting standard NetworkPolicies. Kubernetes combines allowed traffic across these policies. Established-connection behavior is implementation dependent, so a new policy may not sever an existing connection. A pod on a compromised host also requires a stronger boundary. These limits are documented in Kubernetes NetworkPolicy semantics.
Before using this approach in production, enumerate every policy selecting the pod and test quarantine in a representative namespace. Existing broad allow policies can defeat the intended result. Updating their selectors can affect other traffic and requires a reviewed design. A custom incident label does not magically exclude the pod from those policies or from its Service.
The Amazon VPC CNI network policy documentation limits support to eligible EC2 Linux configurations; it does not support Fargate or Windows nodes. Verify enablement, agent health, supported configuration, and enforcement before relying on a manifest. For EKS Fargate, establish a tested alternative using available VPC/security-group and workload controls, with careful awareness of the shared resources those changes affect.
Containment gate: confirm attempted new connections along the exploit path fail, examine existing connection behavior, and monitor for continued suspicious activity. Use known benign probes from authorized test locations and available network telemetry. A successful kubectl apply proves that the API accepted an object, not that packets were blocked.
Playbook 3: stolen Kubernetes or AWS workload identity
Trigger: unexpected Kubernetes API access by a service account, or AWS calls by a workload role that cannot be explained by the application's intended behavior.
Kubernetes RBAC, EKS cluster access, and AWS IAM workload permissions are separate authorization paths. Disabling one does not inherently disable the others. EKS supports both IRSA and EKS Pod Identity for workload AWS access; understand which credential mechanism is actually configured. See EKS workload service accounts.
Image. Evidence correlation map linking Kubernetes pod UID and service account to API audit logs, workload role sessions, CloudTrail events, node evidence, and customer impact
Correlate workload identity and evidence across authorization boundaries. A Kubernetes service account and an AWS role session represent different access paths.
Enumerate other pods sharing the affected Kubernetes service account before changing its access:
Code example. Refer to the written article for the complete code.
The last operation inventories Pod Identity associations; use association details to inspect the actual role. For IRSA, inspect the service account's role annotation and role trust policy. Also investigate unintended access to a node role or credentials injected through another mechanism.
Correlate audit timestamps, service account, pod UID, role ARN, role session information, and CloudTrail activity. Make the confidence of each match explicit. A workload can be deleted and recreated with the same name; UIDs and session timing help separate those instances.
Use a credential-specific containment plan:
Table. Columns: Access path, Containment task, Verification.
Row 1. Access path: Kubernetes service account; Containment task: Remove unauthorized RBAC grants; address the exposed token and workloads that can obtain new tokens; Verification: Attempt the formerly permitted operation with a controlled test identity and confirm authorization denial.
Row 2. Access path: EKS human/automation access; Containment task: Review access entries, access-policy associations, and any relevant RBAC mappings; Verification: Confirm the suspect principal cannot use the prohibited cluster access path.
Row 3. Access path: Workload AWS role; Containment task: Restrict effective permissions; prevent further unauthorized assumption; consider supported role-session revocation; Verification: Verify both old-session behavior and new-session behavior against the affected resources.
Row 4. Access path: Node instance role; Containment task: Scope every workload using the role before modifying permissions; Verification: Confirm containment without creating an unplanned cluster-wide dependency failure.
End of table.
EKS access entries are a cluster access mechanism; they do not replace workload IAM or Kubernetes service-account investigation.
Deleting a pod, removing an IRSA annotation, changing a trust relationship, or removing a Pod Identity association must not be treated as proof that credentials already copied elsewhere are unusable. AWS documents IAM role-session revocation using a time-based deny policy. Evaluate its role-wide impact and propagation, then test denial; blocking new assumptions and restricting existing sessions solve different parts of the problem.
Rotate affected application secrets after closing the path that exposed them. Otherwise, the attacker can simply obtain the replacement. Investigate downstream databases, queues, object stores, and external APIs to which the stolen identity had access.
Playbook 4: suspected worker-node compromise
Trigger: credible host escape, unauthorized privileged execution, tampered node processes, or evidence that the runtime or operating system cannot be trusted.
Identify the node and its AWS instance reference from the affected pod. Record all workloads there before any network or instance action:
Code example. Refer to the written article for the complete code.
Cordoning prevents normal new scheduling onto the node. It does not stop running containers or block their network traffic. A scheduling taint is also distinct from isolation; a NoExecute taint can evict pods and alter evidence, so do not substitute it casually.
For EC2-backed workers, establish a tested infrastructure quarantine process that accounts for instance ENIs, security groups, shared group membership, responder access, control plane reachability, and monitoring. Network changes can break healthy workloads on the same node. Consider autoscaler, node-group replacement, and lifecycle controllers that might terminate the evidence source.
kubectl drain is a workload evacuation tool, not an evidence-preservation tool. It can evict pods and remove volatile state; do not execute it as the first forensic action. Its behavior and flags are documented in the kubectl drain reference.
Decide whether active harm requires immediate termination or whether isolated forensic capture is feasible. In a controlled forensic workflow, specialists can capture volatile state and storage artifacts before replacement. An EBS snapshot preserves block-volume contents, not RAM; a container image is not the running container's writable filesystem. Memory acquisition and filesystem collection can themselves alter the system and require appropriate tooling and custody records.
The EKS incident and forensics guidance describes identifying affected pods and nodes, isolation, credential response, evidence capture, and redeployment. Adapt these steps to your compute model: EC2 node access and preservation assumptions do not transfer directly to Fargate.
Recovery: fix the path, rebuild from trust, then prove correctness
Do not recover by deleting compromised pods while leaving their vulnerable controller template unchanged. The controller can recreate the same exposure. Separate the emergency availability action from the permanent corrective change.
Recover in this order:
Fix the exploit path: application vulnerability, unauthorized identity grant, image compromise, unsafe admission configuration, or exposed secret.
Preserve necessary evidence and record the final scope of affected workloads and data.
Deploy a reviewed configuration and verified artifact from an intact build/release path. If the node is suspect, replace it rather than treating a pod restart as host remediation.
Rotate affected secrets and credentials after closing the exposure path. Review downstream impact and consistency.
Restore traffic gradually and watch service behavior and security telemetry together.
Independently verify that the previously abused action fails and the intended customer journey succeeds.
For an approved image change, use the real container name and a real reviewed digest:
Code example. Refer to the written article for the complete code.
The placeholder must be replaced; this is a change pattern, not an immediately runnable deployment. A rollout success is one gate. Also inspect actual resolved image IDs, run representative authenticated transactions, verify error budgets or agreed health thresholds, and check for recurring suspicious behavior. If data may have been altered, pod health does not establish data integrity.
Keep a rollback decision explicit. Rolling back to a known vulnerable artifact may restore availability while reopening the attack path. Alternative recovery options include disabling the vulnerable feature, restricting exposure, or deploying a forward fix with reduced functionality.
Why this playbook separates evidence, containment, and recovery
These phases have different correctness criteria. Evidence collection asks whether the relevant observations are preserved and traceable. Containment asks whether the attacker's effective path is blocked. Recovery asks whether the intended service and data are trustworthy and functioning. Combining them into “restart and check green pods” hides gaps.
Automation should encode those boundaries. Use incident IDs, resource UIDs, expected account/cluster checks, bounded selectors, dry-run previews where supported, and a recorded rollback. Permit low-risk evidence collection automatically; put broad permission removal, node termination, shared security-group changes, and destructive cleanup behind the established incident decision owner. Re-check resource identity immediately before mutation to avoid acting on a newly recreated object with the same name.
Store playbooks with application and infrastructure changes. Assign an owner and a tested-on record with cluster version, compute model, CNI configuration, credential mechanism, and evidence destination. A playbook last tested against EC2 Linux networking is not automatically valid for another environment.
Exercise the playbook and measure the gates
Use a nonproduction environment and benign simulated signals. A useful exercise is more specific than “respond to a deleted pod”: have a sanctioned identity delete a test replica, introduce a broad allow policy that exposes a quarantine weakness, and verify the responder notices the difference between object creation and actual isolation. Test stolen-identity handling with a least-privileged sandbox identity rather than live credentials.
Record UTC milestones: first actionable signal, acknowledgment, incident declaration, first preserved evidence, verified containment, restored customer journey, and closure. Derive durations from these observations:
Code example. Refer to the written article for the complete code.
Measure missing artifacts, failed identity correlations, untested containment paths, and repeated compromise after recovery. Report median and tail durations only when you have enough consistently defined incidents to make them meaningful. This article supplies no fabricated benchmark or promised reduction in response time.
Close with a review of the failure mechanism and the controls that could have prevented or detected it. Give each corrective action an owner, due date, and verification method. The most useful next step is a rehearsal of one concrete scenario with your real logging, identities, and networking configuration; use the linked EKS incident response guidance as a reference while documenting the differences in your environment.
Frequently asked questions
Is a deleted Kubernetes pod always a security incident?
No. Rollouts, scaling, eviction, and controller reconciliation can remove pods legitimately. Investigate when customer impact, an unexpected actor, suspicious timing, or unauthorized configuration changes make the deletion abnormal.
Does a deny-all NetworkPolicy guarantee pod isolation?
No. Policies are additive, enforcement depends on the network implementation, and established connections may behave differently. Check all selecting policies, platform support, host networking, and actual traffic before declaring containment.
Does deleting a pod revoke its AWS credentials?
No. Temporary AWS credentials copied outside the pod can remain usable until expiry or effective permission denial. Contain the relevant IAM role and access path, then verify attempted use fails without disrupting unrelated workloads unnecessarily.
What evidence should responders capture first?
Record UTC timestamps, exact resource identities, pod and owner specifications, container image digests, current and previous logs, events, and relevant audit records. Protect the archive because specifications and logs may contain secrets or personal data.
How does EKS Fargate change incident response?
EKS Fargate does not provide the same customer-managed node access as EC2 worker nodes. VPC CNI NetworkPolicy and GuardDuty Runtime Monitoring do not support EKS Fargate, so validate alternative network containment and use available audit, application, identity, and network evidence.