AI Agent Context Windows and Working Sets: Choosing What the Delegate Can See

A narrative guide to context windows, working sets, source selection, memory discipline, privacy, and keeping AI agent work grounded without flooding it with irrelevant material.

A desk with blank note cards, file folders, abstract workflow screens, and a human operator selecting a small working set from a larger archive.
On this page

An AI agent can only work with the world it can see. That sounds obvious until a task fails because the agent saw too much of the wrong thing and too little of the right thing.

People often talk about context windows as if they are storage boxes. A larger box sounds better. More documents, more chat history, more logs, more screenshots, more tickets, more files. Surely the agent will do better if nothing is missing.

Sometimes it will. Often it will not. A context window is not only a container. It is the agent’s working room. If the room is full of old drafts, stale decisions, duplicate files, private notes, abandoned plans, and loosely related background, the agent has to infer what matters. The model may still produce a confident answer, but the confidence may be built from clutter.

Good delegation is not dumping everything into view. It is choosing the working set.

A contextual Ai Agents guidebook scene for AI Agent Context Windows and Working Sets: Choosing What the Delegate Can See

The working set is the agent’s desk

A working set is the small collection of materials the agent should actively use for a task. It may include a user request, a relevant file, a short policy note, a recent error log, a customer record, an API reference, a design decision, or a previous run summary. The important part is not the format. The important part is that the materials belong to the job.

The desk metaphor helps. A person writing a careful memo may have three documents open and a folder nearby. They do not cover the desk with every document the company has ever produced. If they need more, they retrieve it. If something becomes irrelevant, they move it aside.

Agents need the same discipline. They can search, read, summarize, and retrieve, but they still benefit from a clean active surface. A good working set tells the agent what to treat as live evidence. It reduces the chance that the model will blend old and new instructions, mistake examples for requirements, or let a noisy transcript overpower the actual task.

AI Agent Memory and Context explains what agents should remember and what they should forget. The working set is the practical version of that idea during a single task. It is memory with a job to do.

Make room for the work

A context window is the bounded amount of information a model can use in a request and its response. It is measured in tokens, which are pieces of text rather than a fixed number of words. Instructions, conversation history, retrieved passages, tool definitions and results can all consume space; images and other inputs can count too. The exact accounting and limits depend on the model and provider. For example, Anthropic’s context window documentation describes how these inputs and generated output count in its API.

A working set is a selection for the job, not necessarily everything the system sends to the model. The application may also add instructions, tool definitions, or conversation history. Stored files and memory outside the request are not automatically in the window: they must be retrieved or otherwise supplied. Conversely, removing something from the visible working set does not necessarily delete it from storage or logs.

Before a long run, check the chosen model’s current limits and count or estimate the full request with the provider’s tools. Leave room for the answer, any model reasoning that counts against the limit, and the next tool results; do not plan to fill the window with source documents alone. For an illustrative 100,000-token combined limit, a team might reserve 20,000 tokens for output and future work, leaving at most 80,000 for current input. That is a planning example, not a universal allocation: separate input or output caps and provider rules may make the usable budget smaller. If the material will not fit, retrieve narrower passages, split the task at a meaningful boundary, or save a checkpoint and resume with the sources needed for the next step.

That 80,000 is the whole current input budget, not a document allowance. If instructions and tool definitions use 10,000 tokens and the conversation uses 15,000, at most 55,000 remain for current sources in this example. As new tool results arrive, count again: each later model request may include a different mix of history, results, and generated text. A reserve is useful only if the application actually enforces it or checks the request before sending.

Know what the application does at the limit. Depending on the API and its configuration, a request may fail, older material may be removed, or the application may replace history with a summary. For example, the OpenAI Realtime API documentation describes configurable truncation of older conversation items. Do not assume every API behaves that way. Check the actual settings and preserve the goal, governing constraints, and source references in a verified checkpoint before dropping history. After a reset or compaction, inspect what survived and reload essential evidence; a summary is not a substitute for a source when a detail matters.

Bigger context changes the failure mode

A larger context window can be useful when the task genuinely depends on many pieces. A code migration may need several files. A legal or compliance review may need policy, exception history, and source documents. A customer support agent may need account context, recent messages, and product behavior. In those cases, the extra room prevents blind spots.

But bigger context does not remove the need for selection. It changes the failure mode. Instead of missing a key detail, the agent may find a misleading one. Instead of asking for a source, it may rely on a stale source that was included accidentally. Instead of following the latest instruction, it may reconcile contradictory instructions into something nobody intended.

This is why context design is an operations problem, not only a model capability problem. The question is not “How much can the model hold?” The better question is “What should be present when this decision is made?”

For repeated workflows, answer that question in the runbook. AI Agent Runbooks is the natural companion to this guide because a runbook can specify the working set: which files to read first, which records to ignore, which logs are authoritative, when to retrieve more, and when to stop because the needed context is not available.

Capacity is not the same as reliable use. In Lost in the Middle , researchers found that the position of relevant information affected performance on multi-document question answering and key-value retrieval for models they tested. Those results do not establish a fixed rule for every current model or task. They are a reason to test the actual workflow with evidence in different positions and with realistic amounts of distracting material, rather than assuming that fitting inside the limit is enough.

Old context can be more dangerous than missing context

Missing context often reveals itself. The agent says it cannot find a file, asks a clarifying question, or makes a gap visible in the handoff. Old context can be quieter. It looks like knowledge.

An agent may see last month’s product requirement beside this week’s decision. It may read an outdated policy before a newer correction. It may include a resolved bug as if it is still open. It may use a previous user’s preference in a new user’s task. None of this requires the agent to be malicious or careless. It only requires the working set to contain sediment.

The fix is not to delete history. History matters. The fix is to mark authority. A task should make it clear which source is current, which source is background, which source is only an example, and which source should not be used unless requested. Humans do this casually with phrases like “ignore the old draft” or “the customer changed their mind.” Agents need that same clarity in the materials they receive.

Dates help, but dates are not enough. A document can be recent and wrong. A policy can be old and still authoritative. A decision can be informal and final. The working set should explain status, not only chronology.

For example, a checkout-error investigation might begin with this compact briefing. The labels and identifiers are illustrative:

Goal: reproduce and explain the checkout error; propose a scoped fix
Current: incident ticket 42 and failing log from 15 May, revision 3
Authority: current checkout specification, section 4, revision 7
Background: incident 18 was resolved; use it only for comparison
Boundary: read checkout files and tests; do not expose customer details
Open question: does the failure occur with a new session?
Next: reproduce with a redacted test account; report if that is unavailable

The log is evidence of a failure, not necessarily the authority on intended behavior. If the specification and a later decision conflict, locate the decision owner or record instead of silently choosing one. In a real briefing, use links or paths that resolve to the actual versions and include the relevant log lines or a way to retrieve them. If a source is missing, say what is missing and what conclusion it prevents.

Retrieval should be deliberate

Many agent systems use retrieval to bring in relevant material as the task unfolds. This is powerful because the agent does not need everything at once. It can search the archive, read the top candidates, and expand the working set when the task asks for it.

Retrieval also needs judgment. A search result is not the same as evidence. A similar title is not the same as a relevant source. A long document is not necessarily the best document. The agent should know whether it is allowed to browse broadly, whether it should prefer official sources, whether private records require approval, and how to report what it used.

Good retrieval leaves tracks. If the agent changes a file because it found a policy, the handoff should name the policy. If it summarizes a customer issue, it should say which records were read. If it skips a likely source because access was restricted, it should say that. The point is not to burden the human with a full transcript. The point is to make trust inspectable.

Keep the source, version or date, and relevant location with each excerpt. Read enough surrounding text to preserve qualifications and exceptions; a matching paragraph can be misleading when its scope is on the preceding page. Check whether the source is current before relying on it, and search for conflicting evidence when the decision warrants it. If a summary replaces a long source, retain a link back and verify important details against the original. A summary can omit a condition or introduce an error.

In the checkout investigation, search first for the exact error and the affected component, then open the current specification and the relevant log around the failure. If the search finds only the resolved incident, expand the query or inspect the component’s tests and change history; do not treat the old incident as proof of the current cause. If the authoritative specification cannot be accessed, report that limitation and keep the proposed fix conditional. An empty search result means the search found nothing, not that no relevant source exists.

Retrieved pages, messages, and tool output can also contain instructions addressed to the agent. Treat those as source content, not permission to change the assignment, reveal data, or run commands. Separate them from trusted instructions and enforce tool permissions outside the model. Labels alone are not a complete defense. OpenAI’s guidance on agent safety describes prompt injection and safeguards for untrusted inputs; AI Agent Data Boundaries covers limiting what a workflow exposes.

AI Agent Observability is useful here because context is part of the trace. A reviewer should be able to see not only what the agent did, but what it had in view when it did it.

Privacy is a context decision

Context windows create privacy risk because adding material to the working set can expose it to tools, logs, reviewers, and downstream summaries. The safest data is not the data that was included carefully. It is the data that did not need to be included at all.

This matters for personal agents and workplace agents alike. A calendar task may not need the content of every meeting note. A support task may not need payment details. A code task may not need production secrets. A research task may not need private customer names. The working set should be shaped by minimum useful context, not maximum available context.

Permissions help, but permissions are only part of the answer. AI Agent Permissions describes what an agent may access and do. Context discipline describes what it should actually see for this run. A person with a company badge may be allowed into many rooms. That does not mean every meeting should happen in the records archive.

When sensitive material is necessary, identify it as sensitive, limit access and retention under the applicable policy, and omit unnecessary details from later summaries. Redaction reduces exposure but should be checked for identifying details in surrounding text. Removing material from a later prompt or summary does not erase earlier copies in logs, tool providers, or stored records; use the relevant retention and deletion controls. Memory should not become a junk drawer for private details that happened to pass through a task once.

The human still frames the room

The agent may be able to gather context, but the human still frames the room. A good assignment says what the agent should treat as the live problem, what sources matter, what uncertainty remains acceptable, and what should trigger a question. It also says what success looks like after the context has been used.

This framing does not have to be long. It has to be specific. Instead of saying “review this whole project,” ask the agent to inspect the files related to checkout errors, use the latest failing log as evidence of the observed failure, check the current specification for intended behavior, avoid unrelated refactors, and report missing reproduction steps. That is context design. It narrows the working set before the model starts filling gaps with guesswork.

As the task changes, refresh the working set. After a reproduction succeeds, the agent may need the relevant implementation and tests rather than the entire incident history. Before a handoff or restart, save the goal, source references and versions, decisions, artifact state, unresolved questions, and checks that must be repeated. AI Agent Checkpoints explains how to make that state resumable. Recheck time-sensitive facts and permissions; a checkpoint is a starting point, not proof that the world stayed still.

For a recurring workflow, compare results on representative tasks with a carefully selected set and with a broader set. Check whether the agent finds the right evidence, preserves exceptions, distinguishes stale sources, and reports missing information. Include cases where the needed passage is absent or appears in a different position. Record the model, source versions, and outcome so the comparison can be repeated after a change. A shorter prompt may reduce cost, but it is only an improvement if the work remains accurate enough for its purpose.

Make the comparison concrete: assemble a small set of past tasks with known source locations and expected outcomes, including at least one stale-source conflict and one case where the answer cannot be established from the available material. Have a reviewer check the conclusions and cited passages against those sources, then record errors, unsupported claims, missed exceptions, and appropriate requests for more evidence alongside token use and elapsed time. Use the same tasks and scoring criteria for both working sets. The sample will not prove performance on every future case, but it can reveal whether a smaller set discarded something the work needed. AI Agent Output Verification covers checking the result itself.

The mature use of agent context feels less like hoarding and more like mise en place. The right ingredients are on the counter. The pantry still exists. The cook can fetch more, but the active surface is clean enough to work.

When an agent fails, do not only ask whether the model was strong enough. Ask what it could see. Ask what it could not see. Ask what stale material sat beside the live request. Ask whether the runbook taught it how to retrieve more. Ask whether private context was included because it was necessary or because nobody took the time to exclude it.

The context window is where delegation becomes concrete. It is the difference between asking a capable helper to work at a clear desk and asking them to make sense of a storage room with a deadline.

Written By

JJ Ben-Joseph

Founder and CEO · TensorSpace

Founder and CEO of TensorSpace and the editor of Fondsites: an engineer working across software, AI, cybersecurity, and biosecurity who builds the site's games and tools.

Keep reading