KV Cache Behavior in Long Agent Contexts

There are two distinct operating regimes here, not one smooth curve. And the gap between them is where most teams get surprised.

At short contexts, the GPU is mostly waiting for data to arrive. Memory transfer time dominates compute time. The hardware is memory-bound. At long contexts, attention computation grows large enough that the gap between compute time and memory transfer time starts to close. The bottleneck changes character entirely. Strategies that help in one regime can be useless or worse in the other.

Batch size makes everything worse faster than people expect. Cache for multiple concurrent users scales directly with parallel requests. A benchmark you ran with a single user understates the real problem badly. At 128,000 token context with 8 concurrent users, the KV cache alone reaches more than twice the size of the model weights for something like Llama 3.1 70B. The GPU memory you budgeted for weights and activations gets eaten by cache storage. That forces a choice: smaller batch sizes, shorter effective context, or more hardware.

Here's the part that trips people up. Throughput degrades before you actually run out of memory. Memory bandwidth saturates before VRAM fills up. The GPU stalls moving cache tensors around, not just storing them. So the first sign of trouble is often slow generation, not an OOM crash. It looks like a performance problem. It feels like a mystery. It's the cache — the ghost in the machine that haunts your throughput before it ever crashes the system.

To make that concrete: caching a very large number of tokens in Qwen2.5-14B at FP16 requires approximately 33 GB. The model's own parameters take up roughly 28 GB at equivalent precision. The cache outweighs the model. This is not a bug. It is just arithmetic.

Why agent workloads stress the cache in ways single-turn inference does not

A standard single-turn inference request has a prompt and a response. Fairly predictable. An agent trajectory is a different animal. Every round of tool calls, retrieved documents, reasoning traces, and environment feedback gets appended to the context. The model attends over all of it for every future step.

The asymmetry that makes agents genuinely unusual: the actions an agent generates are often very short. Fewer than 10 tokens in a lot of cases. The context feeding those actions can run to tens of thousands of tokens. The cache is enormous relative to what it produces. You're storing a library to write a sentence.

Most cache optimization research assumes the full context is available before generation begins, so compression decisions can be made with knowledge of the query. In a live agent, future queries are unknown when past tokens need to be evicted or retained. That gap between research assumptions and production reality is not a minor caveat. It is the whole problem.

Multi-agent pipelines add structural redundancy on top of that. Multiple agents frequently share overlapping retrieved passages or upstream outputs, but each one redundantly recomputes KV entries for those shared tokens. When upstream outputs get reused under changing prefixes, the KV cache gets rebuilt at every affected downstream agent. It cascades. Embodied and planning agents have their own version of this. Keeping raw text memory of past environment states causes prefill latency to spike with each new observation.

Agent context is adversarial to caching. It is long, dynamic, partially shared, and arrives in a sequence that cannot be predicted in advance.

The attention sink pattern and what it reveals about which tokens the cache must keep

Something genuinely strange happens inside transformers: models pay disproportionate attention to the first few tokens of a sequence, regardless of whether those tokens are semantically meaningful. This is called the attention sink. It shows up in large models and small models alike.

The mechanical reason is simple. Softmax normalization forces all attention weights to sum to 1. When no nearby token is particularly relevant to the current query, the residual weight has to go somewhere. It gravitates toward the initial tokens, which are always visible under causal decoding. The model isn't choosing to focus there because those tokens matter. It's just where the math dumps the leftover weight.

Why does this matter for the cache? Because any eviction policy that uses attention weight as a proxy for token importance will consistently rank sink tokens at the top of the retention list. They look critical. They often aren't. It's like keeping the most-thumbed page of a book — not because it holds the answer, but because everyone's fingers keep landing there by habit.

StreamingLLM found a practical use for this. Retaining a small set of sink tokens plus a rolling recency window avoids performance collapse in long or streaming contexts, cutting memory requirements by 22.2× over baseline recomputation. Real win. The tradeoff is that everything outside the recency window gets permanently discarded.

The deeper issue: "important in the attention distribution" and "important for future reasoning" are not the same thing. A token that barely registers in attention right now might be exactly what the model needs three tool calls from now. Eviction policies have struggled with this gap since the beginning. It is still not closed.

How eviction policies fail when context keeps growing

Three main families of eviction strategy exist, and each has a real failure mode.

Attention-guided eviction uses recent attention weights as a proxy for importance. H2O tracks cumulative attention scores. SnapKV does window-based clustering at prefill. StreamingLLM uses sinks plus a recency window. All reasonable heuristics. None of them know what future steps will need.

Static positional heuristics keep the first N and last N tokens regardless of content. Simple, fast, and they lose everything in between. For a lot of agent tasks, the middle is where the actually useful retrieved information lives.

Query-aware dynamic selection, like the Quest approach, retains the full KV cache and selects relevant entries per decoding step. Avoids permanent eviction. Also does not reduce physical memory, which is often the actual constraint.

The failure mode that cuts across all of these is the saliency shift problem. The set of tokens with high attention changes across decoding steps. A token that looked unimportant at eviction time can become critical several turns later, when a new tool call returns related information and suddenly that old retrieved document matters. But it's gone. There's no recovery.

The research on timing is instructive. Delaying compaction so the agent's future queries are available before eviction decisions are made recovers much of the accuracy that immediate compaction loses. Token eviction can preserve most accuracy while reducing KV cache by around 80% and improving throughput over a no-compaction baseline. The catch is that you need to know something about future query structure before you decide what to throw away. For a live agent, that's genuinely hard.

Eviction is not solved for agents. It is a scheduling problem wearing a memory problem's clothes.

Architectural choices that reduce the cache before inference begins

Model architecture can do a lot of heavy lifting before any eviction policy runs.

Grouped Query Attention (GQA) is the most widely deployed solution right now. Instead of one KV head per query head, query heads share KV projections across groups. The cache scales with the number of groups, not the number of query heads. Llama 3 uses 8 KV heads against 64 query heads. That's an 8× reduction in KV cache size versus standard multi-head attention. GQA is now the default in Llama 2, Llama 3, Mistral, Mixtral, Gemma, PaLM, and most major open-weight releases.

Multi-Head Latent Attention (MLA), used in DeepSeek-V2, compresses keys and values into a low-dimensional latent space before caching them. The result is a 93.3% reduction in KV cache compared to DeepSeek's prior dense model at the same parameter scale, while matching or exceeding standard attention on quality benchmarks.

Cross-Layer Attention (CLA) shares KV projections between adjacent transformer layers, compressing from the layer dimension rather than the head dimension. When combined with GQA in Hunyuan-Large, the two together save nearly 95% of KV cache versus the original multi-head attention baseline, with limited performance degradation.

For teams building long-context agents, model selection is a memory budget decision as much as a quality decision. Choosing a model with GQA or MLA directly determines how much headroom you have for batch size and context length. It is the highest-leverage call you make before writing a single line of agent code.

The residual problem: architecture lowers the constant but doesn't change the law. Cache still grows linearly with context. At extreme lengths, it is still large. Architectural compression buys you room. It doesn't buy you out of the problem entirely.

Systems-level infrastructure for managing cache across multi-turn and multi-agent sessions

LLMs have no intrinsic session memory. Each new turn is a fresh request whose prompt is the full concatenation of everything that came before. Every turn shares a prefix with the previous turn, which makes prefix caching the natural optimization: store the KV tensors for shared token prefixes and reuse them instead of recomputing.

Tree-structured radix caches generalize this to branching prompt structures, useful when multiple agents share parts of the same context. The concept is clean. The implementation challenge is entirely in the eviction policy.

Standard serving systems evict the least-recently-used prefix cache entries. Simple access-time heuristic, no knowledge of which agents are about to execute. The result is KV entries getting evicted shortly before their owning agent resumes its next step, forcing full recomputation or CPU-to-GPU swapping at exactly the wrong moment. You clear the desk right before you sit down to work.

KVFlow, presented at NeurIPS 2025, addresses this directly. It models the agent execution schedule as a graph, assigns each agent a "steps-to-execution" estimate, and uses that to prioritize retention of cache entries closest to reuse. It also introduces overlapped KV prefetching, proactively moving required tensors from CPU to GPU in background threads before the agent's next step fires. The result is up to 1.83× throughput speedup over SGLang with hierarchical radix cache for single workflows with large prompts.

The shared-prefix opportunity in multi-agent systems is real but requires the serving layer to actively identify and deduplicate overlapping prefixes across concurrent agent requests. That doesn't happen automatically. It requires infrastructure that understands what agents are doing and when.

The serving infrastructure's eviction policy is as consequential as the model's attention architecture. For teams running persistent, multi-turn agents at scale, these are not separate concerns.

What engineering teams building long-context agents can actually control

Here's where it gets concrete.

Before a request is made:

• Model selection. Prefer architectures with GQA, MLA, or CLA. Reduces per-token cache footprint before anything else runs. Order the stable shared content (system instructions, retrieved corpus) as a prefix to maximize cache hit rates across turns and across parallel agents. Put the stable stuff first. Order matters more than people think.
• Memory representation. Storing structured or compressed environment state rather than raw text reduces token count at prefill and reduces how often cache invalidation happens. This one doesn't get enough attention.

The eviction timing tradeoff is real. Immediate compaction is cheap but hurts accuracy. Deferred compaction recovers accuracy but holds more cache longer. Where the right balance sits depends on how predictable your agent's future query structure is from its current trajectory. Some agent types are predictable enough to anticipate. Others aren't. Knowing which kind of agent you have is a prerequisite for tuning this, not an afterthought.

Observability is not optional. To tune any of these levers, you need visibility into per-request cache hit rates, eviction frequency, and the ratio of prefill to decode compute. Without instrumentation, cache behavior is invisible until throughput collapses or OOM errors appear. By then you're debugging a crisis instead of managing a system. I've been there. It's not fun.

The infrastructure ownership question is the one that determines everything else. Running agents on infrastructure you control, rather than a closed API, is what makes it possible to configure serving-layer eviction policies, expose cache metrics, and deploy systems like KVFlow or custom prefix management. On a black-box API, none of those knobs exist. OpenHands, which runs agent workloads on self-hosted infrastructure with model-agnostic execution, is positioned to expose and act on exactly this level of serving-layer control. That matters when you're running concurrent agents at scale and need to actually see and adjust what's happening.

Know your inflection point. At small scale, with a single developer and short sessions, KV cache behavior is genuinely negligible. The engineering investment in cache-aware design pays off when concurrent agents, long-horizon tasks, or sustained multi-turn sessions push context length into the tens of thousands of tokens. Diagnosing where that inflection lies for your specific workload is the first step. Until you hit it, most of this reads like theory. Once you hit it, it's the only thing on your mind at 2am.