Kubernetes Deployment for AI Coding Agents

Kubernetes is the de facto container orchestration platform in production engineering today. The CNCF's 2025 Annual Cloud Native Survey confirms it. Most teams running containers are running Kubernetes. But here's the thing: the Kubernetes mental model was built around stateless microservices, not AI agents. And the gap between what Kubernetes was designed for and what AI coding agents actually need is wide enough to cause real operational pain if you don't plan for it. This article walks through that gap, layer by layer, and builds a reference architecture that actually holds up under the weird, bursty, trust-violating reality of production AI coding agents.

How AI agent workloads differ from traditional Kubernetes primitives

Let's start with the uncomfortable truth. An AI coding agent is not a microservice. Not even close.

A microservice is stateless, deterministic, and request/response. You get a request, you process it, you return a response. You can run a hundred copies of it. If one dies, another picks up. Beautiful. Boring. Kubernetes loves it.

An AI coding agent is the opposite of all of that.

• Singleton, stateful: Each agent is a single-instance workspace. You can't just spin up three copies and load-balance across them. The agent has a scratchpad, a working directory, a context window that has been built up over multiple interactions.
• Bursty activity profile: Most of the time, the agent is idle. Waiting on a slow API. Waiting for a user. Then, suddenly, it's doing a lot of work at once. CPU and memory metrics don't capture this pattern well because they lag behind the real demand signal.
• Untrusted code execution: This is the big one. Coding agents write code and then run it. That's the whole point. This is not a side effect or an edge case. It's the core operation.
• Long-lived context: Losing state is not graceful degradation. It breaks the task. An agent that loses its scratchpad mid-task has to start over, which is a bad experience at best and a catastrophic failure at worst.

Standard Kubernetes primitives don't map cleanly here. You can approximate a coding agent with a StatefulSet of size one, a headless Service, and a PVC per agent. That works at tiny scale. At hundreds of agents, it becomes an operational nightmare to manage, debug, and scale.

Multi-agent coordination makes this harder. When agents need to delegate tasks to each other, they need stable network identities — ephemeral pod IPs are like postcards sent to a house that keeps moving: the address is right until it suddenly isn't. Each layer of the architecture needs to be designed for agents specifically. You can't just lift and shift microservice patterns and hope for the best.

The sandbox layer: giving each agent an isolated execution environment

Sandboxing is non-negotiable. Full stop.

Default container runtimes share the host kernel. If an agent executes generated code that contains a kernel exploit, that exploit reaches the node. You don't want that. At all.

There are two runtime isolation options that see real production use:

• gVisor: Intercepts syscalls in userspace. Lower overhead. Good default for most coding agent workloads.
• Kata Containers: Full VM-level isolation. Stronger boundary. Higher overhead. Worth it for higher-risk execution contexts.

Agent Sandbox is a new Kubernetes primitive introduced at KubeCon NA 2025, now being developed as a CNCF project under SIG Apps. It's worth knowing about because it directly closes the primitive gap we just described.

• Declarative API designed specifically for singleton, stateful workloads
• Built on gVisor, with an optional Kata Containers backend
• Each Sandbox gets a stable hostname and network identity out of the box
• Supports scale-to-zero for idle agents, with resumption to prior state

That last point matters more than it sounds. Scale-to-zero only works if resumption is fast. Which brings us to the cold-start problem.

Provisioning a new pod adds meaningful latency. For a microservice, that's acceptable. For a user invoking an idle agent mid-conversation, it's a broken experience. The answer is SandboxWarmPool, which pre-provisions a pool of ready Sandbox pods. A SandboxClaim against a SandboxTemplate hands over a warm, isolated environment immediately. No waiting.

If your team isn't on Agent Sandbox yet, that's fine. But at minimum, enforce a non-default runtime class. Running agent containers with the default runc runtime is an unacceptable risk profile for code-executing workloads. gVisor or Kata, pick one, and make it the standard.

Structuring agent identity, persistence, and lifecycle as Kubernetes resources

An agent needs to be a first-class citizen in your cluster. Not a workaround. Not a hack. An actual resource with identity, storage, and a defined lifecycle.

Persistent identity means each agent has a stable name within the cluster. Agent Sandbox provides this. Without it, teams have to assign a headless Service per agent manually. That scales poorly and is tedious to maintain.

Storage means PersistentVolumeClaims for the agent's scratchpad and context. Storage class selection matters here. Agents doing frequent small writes behave very differently than inference workloads doing large sequential reads. Choose accordingly, or you'll pay for it in I/O latency.

Lifecycle states need to be explicitly modeled:

• Active: Agent is processing a task. Resources are allocated.
• Idle/Suspended: Task is complete. Agent should scale to zero or release GPU resources.
• Resuming: A new task arrives. Warm pool or checkpoint restoration brings the agent back.

For very long-running agents where reconstructing context from scratch is expensive, checkpoint/restore via CRIU is worth evaluating. The tradeoff is complexity versus resume latency. Most teams won't need it immediately, but it's good to know the option exists.

The best way to manage agents at scale is to represent them as Kubernetes custom resources. Two projects that do this well:

• kagent (Solo.io, CNCF Sandbox): Represents agents as Kubernetes CRDs. Versioned in Git. Reviewed in PRs. Deployed with standard tooling.
• kagenti (Red Hat): Similar philosophy, with additional security tooling baked in.
• OpenHands: Can be deployed as a containerized agent platform on Kubernetes. Model-agnostic by design, meaning the model backend is a configurable parameter rather than a hardcoded dependency.

That last point deserves emphasis. Agent configuration, including model endpoint, tool permissions, and resource limits, should live in the CRD spec or a ConfigMap. Not baked into the image. This is what makes the deployment model-agnostic. When you want to swap from an external API model to a self-hosted one, you change one value in a manifest. You don't rebuild the image.

Scheduling and resource allocation for GPU-backed agent workloads

Not every coding agent needs a GPU. Agents running against API-hosted models like those from OpenAI or Anthropic are CPU-bound. GPU scheduling becomes relevant when you're running a self-hosted inference stack.

When it is relevant, the utilization problem is severe. According to Cast AI's 2026 State of Kubernetes Optimization Report, average GPU utilization across production Kubernetes clusters sits at 5%. That means 95% of provisioned GPU capacity is idle at any given moment. At cloud GPU pricing, that idle capacity represents a significant and continuous cost. The scheduling and autoscaling layers in this architecture exist largely to solve that problem.

Here's the tool stack that addresses it:

• NVIDIA GPU Operator: Automates driver installation, container toolkit, device plugin, GPU feature discovery, and MIG partitioning. Start here for any GPU-backed cluster.
• MIG (Multi-Instance GPU): Partitions an A100 or H100 so multiple smaller agent workloads share a single GPU without contention. Relevant when agents use smaller or quantized models.
• KAI Scheduler: GPU-aware scheduler from NVIDIA. Handles bin-packing, gang scheduling, and fair-share allocation with topology awareness. Better fit than the default kube-scheduler for dedicated AI clusters.
• LimitRange at the namespace level: Required to bound per-agent resource consumption. An agent stuck in a loop can consume unbounded memory without hard limits. Set them.

For very large model inference where a model is too large for a single node, llm-d (released in late 2025) provides distributed multi-GPU, multi-node inference on Kubernetes. Relevant when agents are calling a self-hosted frontier model.

Autoscaling agent workers based on queue depth rather than CPU

Here's why the default Kubernetes Horizontal Pod Autoscaler (HPA) doesn't work for agent workloads: CPU and memory spike after work starts, not when it's queued.

By the time HPA detects the spike and provisions new pods, the burst has already landed on your existing pods. And because agents are bursty, HPA scales down too slowly on the back end too. You end up paying for capacity you're not using.

The right signal is queue depth. How many tasks are waiting, not what running pods are consuming right now.

Imagine a batch of documents arriving at once, queuing many analysis tasks simultaneously, then nothing for an extended period. The burst pattern is external. CPU tells you nothing useful until it's too late.

KEDA (Kubernetes Event-Driven Autoscaling) is the answer here. It scales on external metrics: Redis stream length, message queue depth, custom application metrics.

• minReplicaCount: 0 enables true scale-to-zero between bursts. This is the mechanism that actually closes the gap on the idle GPU problem.
• For vLLM inference backing, scaling on request queue depth is semantically more accurate than raw GPU utilization.
• KEDA's Prometheus scaler can query DCGM metrics directly from an existing Prometheus instance. No separate adapter required.

One thing that bites teams and isn't talked about enough: image pull latency. Large inference images with PyTorch, CUDA, and model weights can take several minutes to pull on a fresh node. If you fix your autoscaling signal but ignore this, scale-out events will still have a multi-minute lag. Mitigation options:

• Pull-through cache (Harbor is common)
• Pre-cached images on GPU node AMIs
• Zstandard layer compression

Cluster autoscaler, KEDA, and image pull latency all need to be tuned together. If one is out of sync with the others, the whole system underperforms.

One more thing: multi-agent workflows substantially increase token consumption compared to single-agent patterns. Factor that into your queue depth thresholds and replica limits when sizing things out.

The security threat model specific to AI coding agents on Kubernetes

Standard Kubernetes security assumes deterministic workloads. You wrote the code. You know what it does. You secure accordingly.

AI coding agents break that assumption entirely. An agent receives natural-language instructions and decides which tools to call. The execution path is non-deterministic from the cluster's perspective. That changes the threat model in ways most security teams haven't fully internalized yet.

Prompt injection is the attack vector that keeps security researchers up at night. An attacker embeds instructions in a document, a web page, or a tool output that the agent processes. The agent follows the embedded instruction, accesses credentials, and exfiltrates them. OWASP's Top 10 for Agentic Applications 2026 classifies this as ASI01: Agent Goal Hijack. The category was peer-reviewed by NIST, the Microsoft AI Red Team, and AWS. Coding agents make up a disproportionate share of the agentic projects where this attack surface has been mapped.

Supply chain risk is not abstract. A PyPI incident in early 2026 demonstrated that AI agent framework dependencies can be compromised and propagate across the ecosystem rapidly. Image scanning and pinned digests (not tags) are required, not optional.

Credential management is more consequential with agents than with humans. Agents run continuously and can autonomously provision resources. A leaked secret has a much larger blast radius than with a human operator who logs in occasionally.

• SPIFFE/SPIRE for cryptographic workload identity: Each agent pod gets a short-lived identity certificate. No static API keys.
• kagenti implements this via automatic sidecar injection. No manual certificate management.
• Secrets in environment variables are insufficient. Use a secrets manager with dynamic lease rotation.

Resource runaway is a real operational risk. A reasoning loop that fails to terminate, or a tool returning an unusually large response, can take down a node. LimitRange and ResourceQuota at the namespace level are the backstop. Set them on every namespace, no exceptions.

Network policy should default to deny. Agents should not have unrestricted cluster-internal access. Define explicit allowlists for model endpoints, tool APIs, and inter-agent communication. This is especially important given prompt injection risk. If an agent is compromised, you want the blast radius contained.

There's a confidence gap worth naming directly. A large majority of executives believe existing policies already protect against unauthorized agent actions. The actual incident rate among enterprises tells a different story. Most organizations currently sit inside that gap, which is exactly where attackers operate.

Networking patterns for inter-agent communication and tool access

Two protocol layers matter here and they do different things.

MCP (Model Context Protocol) standardizes how an agent connects to tools, APIs, and data sources. Think of it as the "agent to outside world" interface. By early 2026, MCP had crossed tens of millions of monthly SDK downloads, which tells you it's not a niche experiment anymore.

A2A (Agent-to-Agent protocol) structures how autonomous agents delegate tasks to each other. The "agent to agent" interface. Essential for multi-agent workflows where one agent is orchestrating others.

Stable network identity is what makes multi-agent systems actually function. If agents can be evicted and rescheduled with new IPs, inter-agent communication breaks. Agent Sandbox assigns a stable hostname per Sandbox. That solves the identity problem cleanly.

Service mesh: mTLS between agent pods provides encryption and identity verification in transit. Relevant when agents pass sensitive context or credentials to each other, which they will in most multi-agent workflows.

MCP server deployment pattern: kagent ships a built-in MCP server that bridges agents to the cloud-native stack, including Helm, Argo, Prometheus, Istio, and Cilium. This is the right pattern for giving agents read/write access to cluster state. Controlled interface. Not raw API access.

Egress controls for external tool calls: Agents calling external APIs, whether for code search, documentation, or package registries, should route through an egress gateway. This enables logging, rate limiting, and circuit breaking without touching agent code.

A2A and the confused deputy problem: Inter-agent delegation should require the same identity verification as external access. An agent should not be able to instruct another agent to exceed its own permissions. If you don't enforce this, a compromised agent becomes a privilege escalation vector into the rest of the fleet.

OpenHands is worth highlighting here specifically because the model endpoint is a runtime configuration parameter. Swapping between a locally-hosted vLLM instance and an external API changes one value in the deployment manifest. No new image. No architecture change. That's the model-agnostic pattern done right.

Putting the layers together into a reference deployment architecture

Everything above connects. Here's how the layers stack.

Sandbox layer. Agent Sandbox CRD with gVisor runtime class. SandboxWarmPool for cold-start elimination. If you're not on Agent Sandbox yet, enforce gVisor or Kata Containers as your runtime class today.

Identity and persistence. Agent defined as a CRD using kagent, kagenti, or an OpenHands deployment manifest. PVC for scratchpad storage with an appropriate storage class for small, frequent writes. SPIFFE identity injected via sidecar. No static secrets in environment variables.

Compute. NVIDIA GPU Operator with MIG partitioning for shared GPU access across smaller agent workloads. KAI Scheduler for GPU-aware bin-packing. LimitRange on every namespace, no exceptions. For large model inference across multiple nodes, llm-d handles distributed multi-GPU serving.

Autoscaling. KEDA scaling on queue depth with minReplicaCount: 0 for true scale-to-zero. Cluster autoscaler tuned to respond to KEDA's pending pod signals. Image pull latency addressed via pull-through cache or pre-cached AMIs. All three tuned together.

Security. Default-deny network policy with explicit egress allowlists. SPIFFE/SPIRE for workload identity. Pinned image digests. Dynamic secret rotation. LimitRange and ResourceQuota as resource runaway backstops. Prompt injection treated as a first-class threat, not an edge case.

Networking. MCP server (kagent's built-in is a good starting point) for controlled agent-to-tool access. Egress gateway for external API calls. A2A with enforced identity verification for inter-agent delegation. Service mesh with mTLS for in-cluster agent communication.

None of these layers is optional. Each one closes a specific gap that generic Kubernetes guides don't address because generic Kubernetes guides weren't written for workloads that write and execute their own code, maintain long-lived state, and operate autonomously at odd hours with access to production credentials.

The teams getting this right aren't doing anything magical. They're just taking each layer seriously, in sequence, and not assuming that what worked for their microservices will work here. It won't. But this architecture will.