Context Window Management for Long Tasks

The growth has been genuinely impressive. In early 2023, most models ran on 4K to 8K tokens. That was your budget, full stop. Today, leading models routinely support 200K tokens or more. Some sit at a million. One open-weights model is at ten million. Think of it like upgrading from a sticky note to a whiteboard — the surface area exploded, but a bigger whiteboard doesn't make you a better writer.

The current landscape (as of 2025 and into 2026) looks roughly like this:

• Gemini 2.5 Pro / Flash: 1,000,000 tokens
• GPT-5: 200,000 tokens
• Claude Sonnet 4 / 4.5: 200,000 tokens
• Meta Llama 4 Scout: 10,000,000 tokens (open-weights)
• Claude Opus 4.6 (beta): 1,000,000 tokens

Per Epoch AI research from June 2025, the longest context windows have grown roughly 30x per year since mid-2023. More encouragingly, the input length at which top models actually reach 80% accuracy on long-context benchmarks rose over 250x in just nine months. Effective use is improving faster than raw size.

But those numbers describe a ceiling. They do not describe what happens on the way up to it. Buying a 200K-token window does not mean your agent runs at full capability from token one to token 200,000. It means the window accepts that much input. What the model actually does with that input across the full range is a completely different question, and one the spec sheet won't answer for you.

The Gap Between a Model's Advertised Window and the Context It Actually Uses Well

Research analyzing 22 leading AI models found that effective context windows typically run at 60 to 70 percent of advertised specs. And the drop isn't gradual. It's a cliff.

Concrete example: Llama 3.1, trained to a 128K context, was found in practice to leverage roughly 64K effectively. Half the advertised window. That's not a flaw in the model. It's just what happens when you look past the marketing.

There's also the "Lost in the Middle" problem. LLMs show a U-shaped retrieval pattern: information at the beginning and end of a long context gets recalled better than information buried in the middle. A 2023 paper documented this directly, using needle-in-a-haystack evaluations to surface the pattern. If your agent accumulates 20 steps of reasoning traces and tool outputs, the stuff sandwiched in the middle is the stuff the model will most reliably fumble.

Then there's context rot. Research from Chroma published in July 2025 studied systematic accuracy degradation across 18 frontier models, including GPT-4.1, the Claude 4 family, Gemini 2.5, and Qwen3. A 200K-token window can show serious accuracy loss at just 50K tokens of input. Drops of 30 to 50 percent, well before you've hit anything close to the documented limit. And the degradation is non-uniform. It doesn't announce itself. There's no warning light.

What this means for agents: an agent filling its context window sequentially is not operating at full model capability by the time it hits the midpoint of a long task. The window is full. The model is degraded. And the agent has absolutely no idea — what practitioners call context drift in production.

Getting a bigger window doesn't fix that. Managing what goes inside the window does.

Why Long Contexts Also Get Expensive Faster Than Most Teams Anticipate

There's a physics problem quietly inflating your inference bill.

Transformers compute pairwise relationships between tokens via the attention mechanism. Every token attends to every other token. That scales as n-squared. Double your context size and your attention computation doesn't double. It quadruples. A 128K-token context window costs roughly 256x more to process than an 8K context window. The math is indifferent to your budget. It's like a dinner tab that doesn't just grow with the number of guests — it grows with every possible conversation those guests could have with each other.

Pricing cliffs make this worse in practice. Some major providers charge double for input tokens beyond a certain threshold. The cost curve isn't linear. It steps up, hard. So when you're planning a long-context agent at scale, the cost picture looks nothing like what a single-query benchmark suggests.

Processing a full 1M-token context runs approximately $0.50 per query, compared to roughly $0.05 for a 200K query using selective retrieval. That's directional, pulled from enterprise data in 2026, not a precision guarantee. But the order-of-magnitude difference is real and consistent with what teams actually report.

Per CloudZero's State of AI Costs report, average monthly enterprise AI spend reached about $63,000 in 2024, with projections heading toward $85,000 in 2025. Context bloat contributes meaningfully to those numbers.

The trap is that falling unit prices make it easy to over-prompt. GPT-4-equivalent performance now costs a fraction of what it did in early 2023, so teams pass 10,000 tokens into a model because they can, not because it helps. For a single query, easy to absorb. For agents running dozens of steps across parallel workers? The bill compounds in ways the original cost estimate never accounted for, and nobody notices until the invoice arrives.

Summarization and Compaction: Keeping the Agent's "Plot" Alive Without Preserving Every Word

The core idea is compression. Instead of discarding old context or keeping every word of it, you progressively compact older exchanges into summaries while keeping recent exchanges at full resolution.

What compaction preserves:

• The agent's current goals
• Decisions already made
• Errors encountered and how they were handled
• The overall arc of what's been accomplished

What it loses is fine-grained detail from earlier steps. Fidelity degrades with each compression pass. That is the tradeoff, not a failure mode. The alternative is running out of window entirely, which is worse.

For longer tasks, hierarchical or recursive summarization works well. Older messages get compressed into a summary. That summary feeds the next context window. The agent maintains mission continuity across long execution horizons without replaying every raw step.

This pattern is showing up in real tooling. Codex CLI introduced a dedicated compaction item type to formalize it. Claude Code exposes a /compact command for user-initiated compaction and runs automatic compaction when a configurable limit is exceeded. These aren't research prototypes. They're production features, which tells you something about how widespread the underlying problem has become.

One important limitation: summarization handles narrative continuity well, but it's fuzzy by nature. It will lose exact file paths, variable names, dataset identifiers, intermediate numeric results. If your agent needs to remember that it was working on /data/v3/cleaned_output.csv, a natural-language summary will not preserve that reliably. Which is exactly why structured state exists.

Structured State and Scratchpad Management for Agents That Need Precise Intermediate Results

Summarization is great at keeping the story alive. It is genuinely bad at keeping facts precise.

If your agent loaded a specific file, produced an intermediate calculation, or set a variable that downstream steps depend on, a narrative summary is not sufficient. "The agent was working on some data file" is not a substitute for the actual path.

The structured state approach replaces the running chat transcript with a managed JSON object. It tracks goals, facts, errors, and current progress explicitly. A scratchpad that's compact by design, not by accident.

A hybrid approach works well in practice:

• Recent messages stay verbatim, at full resolution
• Older exchanges get compressed by a fast inference model into a structured summary that explicitly preserves critical operational state (file paths, variable values, step progress) while discarding conversational filler

This matters for multi-step agents because workflows can then resume even when raw token history exceeds the context window. The agent knows where it is. It reads the scratchpad. It doesn't need to replay everything.

Structured state and natural-language summarization are complements. Summaries carry narrative continuity. Scratchpads carry operational precision. For complex long-horizon tasks, you need both running in parallel.

Retrieval-Augmented Generation as an Alternative to Loading Everything Upfront

RAG flips the default assumption. Instead of loading a full knowledge base or codebase into context and reasoning over all of it, you retrieve only the chunks relevant to the current step at inference time.

The cost difference can be dramatic. Per Elasticsearch Labs research, RAG systems achieved over 1,000x lower cost per query compared to pure long-context approaches for large, frequently updated datasets.

RAG clearly wins in a few scenarios:

• Large document collections where most content is irrelevant to any given step. Why load a million tokens when you need 5,000?
• Real-time or near-real-time data. Incremental indexing beats reloading a full context window every time something changes.
• Cost-constrained deployments running many agent steps in parallel.

But RAG has a real limitation that tends to get glossed over. It presupposes you can write a query. If the agent doesn't know which part of a codebase is relevant, retrieval will miss exactly what matters. "What are the concerning clauses in this contract?" is not a query you can use to retrieve the concerning clauses. You have to read the contract first to know they exist. That's a multi-hop reasoning problem requiring iterative retrieval, and RAG can't solve it alone.

Research (Li et al., 2024) also found that stronger closed-source models often outperform RAG when given full context. The story isn't clean. The right choice depends on model capability, task structure, and available compute, and it changes depending on all three.

Why the Most Effective Production Systems Route Between Retrieval and Long Context Rather Than Choosing One

The emerging default in production as of 2026 isn't RAG or long context. It's both, applied selectively depending on what the task actually needs.

The rough pattern: retrieve 50K to 200K relevant tokens, then reason over them with a long-context model. Pure RAG misses single-document reasoning. Pure long context degrades under its own weight. The hybrid sidesteps both failure modes without fully solving either one in isolation.

Research presented at EMNLP 2024 looked at a self-routing approach where the model reflects on whether it needs full context or focused retrieval before answering. This improved overall accuracy while cutting computational cost meaningfully. A real result from a published study, not a vendor claim.

The routing logic in practice:

• Simple, targeted query with a known answer location? Retrieve the relevant chunk. Keep context small.
• Complex multi-hop question requiring global understanding? Long context over the full relevant material.
• "What should I be worried about in this document?" Long context. The agent can't write the retrieval query because it doesn't yet know the answer.

Static knowledge corpora favor long context for single-document analysis, while dynamic corpora favor RAG with incremental indexing. Dynamic, frequently updated datasets favor RAG with incremental indexing. The point is that neither wins universally, and pretending otherwise is how teams end up with the wrong tool for the job.

The practical implication: agent systems need a routing layer. A lightweight classification step that decides which mode to invoke. Not a fixed strategy applied uniformly to every query regardless of what that query actually is.

Emerging Research on Agents That Learn to Manage Their Own Context

Most of the strategies above assume engineers make the decisions. They decide when to summarize. They configure the compaction threshold. They design the routing logic. Emerging research is shifting that assumption, and the early results are interesting enough to pay attention to.

A few papers worth knowing:

ReSum (Wu et al., 2025): Trains the agent to call a summarization tool when its context approaches the limit, compressing interaction history before continuing. Context management becomes a first-class agent action, not an external control imposed by the engineer.

MemAct (Zhang et al., 2025): Exposes a pruning tool directly to the agent. The agent can summarize selected historical messages, remove the originals, and append the summary. The agent decides what to keep.

AdaCoM (May 2026): Substantially improves performance by preserving task constraints and progress while pruning stale content. It also identifies a Fidelity-Reliability Trade-off worth understanding: higher-performing agents benefit from higher-fidelity preservation, while lower-performing agents need more aggressive compression to stay within a reliable reasoning regime. Your compression strategy should depend on your baseline agent capability, which is not obvious but is genuinely useful.

MEM1-7B (Zhou et al., June 2025): A 7-billion-parameter model that improves performance 3.5x while reducing memory usage 3.7x compared to a larger baseline on a multi-hop QA task. It also generalizes beyond its training horizon. An early but real demonstration that reasoning-driven memory consolidation can scale.

These results are promising. Most evaluations are on controlled benchmarks, and generalization to open-ended, multi-domain agent tasks in production is still an open question. But the direction is clear: context management is becoming something agents do for themselves, not just something engineers configure around them.

Prompt Caching as a Low-Effort Complement to Context Management

This one is simple, underused, and worth doing.

Cache the static portion of your prompt. System message, standing instructions, frequently retrieved chunks that don't change between steps. Stop reprocessing them on every agent call.

Per-call cost drops 50 to 90 percent when cache hit rates are high. Most major providers support some form of prompt caching, typically backed by a KV cache, as of 2026. The implementation lift is low relative to the savings.

It also reframes the optimization target in a useful way. Instead of "minimize context size," you start thinking about "maximize cache hit rate." That's a different design instinct. It changes how you structure prompts. You front-load stable, reusable content. You separate static from dynamic. Small architectural shift, real cost impact.

One thing to keep straight: caching reduces cost per call. It does not address the window-filling problem as tasks accumulate history. It doesn't replace summarization or retrieval. It's the last layer of the stack, not the first.

What a Deliberate Context Management Strategy Looks Like for a Real Agent Deployment

Treat context as a managed resource, not a buffer. Every token in the window should be there for a reason, which is what an explicit token budget enforces. The system should have an explicit policy for what enters, ages, and exits. "We'll deal with it when we hit the limit" is not a policy.

Research suggests a significant portion of enterprise AI failures in 2025 were attributed to context drift or memory loss during multi-step reasoning. The cost of not having a strategy is measurable, not theoretical.

A layered approach for long agent tasks:

Layer 1: Structured scratchpad for operational precision. Current goals. File paths. Intermediate results. Anything downstream steps depend on. Compact by design.

Layer 2: Hierarchical summarization for narrative continuity. What decisions were made and why. What was tried and didn't work. The plot of the task, not the transcript.

Layer 3: Hybrid RAG routing for external knowledge. Retrieve when the query is known and specific. Load full context when the question requires reading to discover the answer. Route between both based on task type.

Layer 4: Prompt caching for static elements. System prompts, tool definitions, standing instructions. Cache them. Don't reprocess them on every step.

None of these layers is optional for complex, long-horizon tasks. Each one handles a failure mode the others don't touch. Structured state handles precision. Summarization handles continuity. RAG routing handles cost and relevance. Caching handles per-call overhead.

The agents that hold up in production aren't the ones with the biggest context windows. They're the ones built by teams who noticed, early, that a larger window just gives you more space to fill with the wrong things.