AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
AWS DevOps Agent for EKS: Make Every Diagnostic Tool Earn Its Access
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Connect native EKS inspection and selected MCP tools without confusing network reachability, diagnostic evidence and remediation authority.
TL;DR: AWS DevOps Agent can investigate EKS incidents through native cluster inspection and custom MCP evidence sources. These paths have different access boundaries. This article provides a complete workflow for existing infrastructure: configure the documented EKS access entry, register a supported MCP endpoint, select tools and collect bounded Service, EndpointSlice, Pod and log evidence. It separates missing authorization from application failure, explains private connectivity and tool classification, and keeps remediation under explicit control. The example was statically reviewed; no AWS investigation, cloud deployment, recovery measurement or incident benchmark was executed.
Start with an evidence boundary
An EKS service returns 503s after a deployment. An investigation agent can gather Kubernetes state, logs and change history quickly, but its usefulness depends on what it can actually observe. A convincing explanation without the affected namespace, the right time window or a reachable diagnostic source is still a hypothesis.
AWS DevOps Agent is generally available. Its production investigations are available across supported Regions, while some other capabilities remain regional previews. Choose the Agent Space Region deliberately: its investigation data lives there, even when associated workloads run elsewhere. AWS availability, Regional boundaries.
Treat the Agent Space as a configured investigative identity. Define the cluster, namespaces, evidence sources and permitted operations before an incident. Keep directed actions disabled for this diagnostic workflow. The wider product supports elevated, operator-approved mutation; “the agent can never change anything” would be an incorrect security claim. Directed actions.
Image. Custom MCP node-evidence sequence: authenticated tool invocation crosses a separate collector boundary before returning sourced evidence.
The diagram depicts the custom node-evidence path; native EKS access is configured separately. Collection authority and provenance must be reviewed before using evidence to propose remediation. It is an architecture illustration, not a measured incident timeline.
Give native EKS access a deliberate scope
Use an existing Agent Space and a cluster whose authentication mode includes the EKS API. In the Agent Space, find the primary cloud source role ARN under Capabilities → Cloud → Primary Source → Edit. In the EKS console, open Access, create an IAM access entry for that exact ARN, and associate the AmazonAIOpsAssistantPolicy EKS access policy. Select only the approved namespaces when cluster-wide visibility is unnecessary.
This is an EKS access policy, not a similarly named IAM policy attached arbitrarily to a role. Verify the principal and scope before creating the entry. Start a small investigation asking for Pods in an allowed namespace. Native access supports resource inspection, logs and events, and explicitly cannot create, modify or delete cluster resources. EKS setup.
Namespace scoping has a diagnostic cost: node health or an upstream service in another namespace may remain outside visibility. Record that limitation instead of treating an incomplete graph as proof of isolation. Separately review the AWS account role's permissions and the service's access guardrails. Account access controls.
Extend missing evidence through a supported MCP path
A missing node log does not justify a general remote shell. Prefer a narrow diagnostic tool with explicit inputs, bounded output and resource restrictions.
For an existing reviewed server, open Capability Providers → MCP Server → Register. Supply its HTTPS endpoint and supported authentication configuration. For an existing OAuth Client Credentials server, select that flow and enter its client ID, client secret, token exchange URL and required scopes; supply exchange parameters required by that authorization server. Submit and verify connection validation before adding tools. Only Streamable HTTP is supported; a local stdio server is not directly interchangeable. After registration, open the Agent Space's Capabilities → MCP Servers → Add, choose the provider and select specific tools. Keep downstream credentials read-only.
Inspect classification too: programmatically registered tools without toolDetails default to read-only and can run without approval. Metadata alone cannot make a mutating implementation safe. MCP integration.
AWS's documented node-diagnostics extension uses AgentCore Gateway with Lambda tools and OAuth. It also requires worker-node SSM access. That architecture crosses a host-level boundary: review collection commands, roles and data retention before adoption. This walkthrough uses an existing server; it does not deploy the sample's cloud resources. AWS extension architecture.
AgentCore Gateway exposes tools. DevOps Agent private connections provide network reachability through VPC Lattice. These are distinct functions. For a private OAuth integration, both the MCP endpoint and token exchange endpoint must be reachable. Private connectivity.
Run a bounded 503 investigation
Prerequisites are an existing Agent Space, approved access entries, an existing EKS workload, Bash and a working kubectl installation compatible with that cluster. The operator's kubeconfig identity is separate from the agent's role. Success through one does not verify the other's access. Record the client/server versions when adopting this runbook; a managed agent service has no local binary version to pin.
For an existing api Service in namespace catalog, supply the actual kubeconfig context, selector, Pod and container. Read the Service's selector before setting POD_SELECTOR; do not guess labels from its name. Save this script as collect-eks-evidence.sh:
Code example. Refer to the written article for the complete code.
Invoke it with explicit existing identifiers:
Code example. Refer to the written article for the complete code.
The selector is illustrative and must match the actual Service. The script performs reads only, stops at the first command failure and bounds log output. Its sequence is not an atomic snapshot; record collection times and repeat affected reads if a rollout changes resources during collection. Pod descriptions and application logs can contain sensitive operational data. Keep raw evidence in approved storage and redact it before sharing.
This log command covers a recent incident. For an older incident, retrieve the actual interval from your approved log platform rather than treating recent Pod logs as historical evidence.
In Incident Response, start an investigation, set the actual incident time, and supply this scoped request:
Code example. Refer to the written article for the complete code.
The web application's Summary and Timeline expose findings and tool activity. Compare them with independently collected evidence rather than accepting a fluent summary. Investigation workflow.
Distinguish backend failure from missing visibility
Consider an illustrative failure: the Service selector uses app=api-v2, while intended Pods retain app=api. Inspecting selectors, Pod labels and EndpointSlices supports a selector-mismatch hypothesis. An empty backend set alone is insufficient: readiness, controller state and intentionally manual EndpointSlices require different explanations. Service diagnosis.
Recovery belongs in the owning deployment process. Review the intended selector and labels, correct the managed manifest through the approved change path, then repeat endpoint and readiness checks. Do not silently patch production because the agent proposed a plausible fix. Confirm customer-facing recovery through the relevant traffic and error signals; populated EndpointSlices do not prove successful requests.
A forbidden response means a read was denied. A timeout means evidence retrieval failed. Neither demonstrates that application Pods are unhealthy. Preserve the error, affected tool, principal and target before drawing conclusions. For later evaluation, count supported findings and inaccessible evidence separately; no incident was executed here and no accuracy or latency benchmark is claimed.
Repair integration failures without widening authority
If tool discovery succeeds but invocation fails, compare discovery credentials, invocation identity and backend authorization. If OAuth exchange fails, verify its endpoint reachability rather than assuming the MCP server itself is down. For private connections, review target routing, security groups and configured ports. Public service IP allowlists are not the private connection's traffic source. Network troubleshooting.
Restore only the missing approved access. Recheck the EKS role ARN and access-policy association for native failures; expand namespace scope only after establishing that the missing namespace is necessary. Cluster-admin access is not a useful substitute for diagnosing an incorrect principal.
Logs and tool responses are external input. Prompt-injection defenses help, but restrictive credentials and narrow tools remain necessary. AWS security guidance.
Cleanup requires no workload deletion: the script creates no cluster objects. For a temporary integration, remove this Agent Space's tool association and review any temporary access entry with its owner. Do not deregister an account-level provider used by other spaces. Apply normal evidence retention rules to collected files.
FAQ
Can I connect any EKS MCP server? Only a server with supported Streamable HTTP transport and authentication fits this integration; tool behavior still requires review.
Does native EKS access enable remediation? No. Native cluster access is read-only. Directed actions are a separate elevated capability with opt-in and operator approval.
Does a private connection grant Kubernetes access? No. Network reachability and authorization are independent prerequisites.
Was this workflow executed against AWS? No. This is a source-verified, statically reviewed workflow for existing infrastructure. It makes no claim of cloud execution, measured recovery time or proven root cause.