Context Windows

usable26.9k

placed26.9k

dropped94.5k

compressed1

context windowstoken budget · hover the prompt

windowsize

packstrategy

run

how it works

A context window is the finite workspace a model can use for one request. System rules, chat history, retrieved documents, tool results, the current question, and the future answer all compete for one token budget.

A failing CI run shows the tradeoff. The prompt may contain a failure brief, early fix attempts, review notes, fresh logs, workflow configuration, source files, retrieved error lines, and the current fix request. The model can use only the pieces that fit, so each packing strategy decides which facts remain available when it proposes the next patch.

The first computation is prefill, where the transformer reads the chosen material and stores attention state in the KV cache. Larger windows carry more of the CI thread, but they also increase memory use, first-token latency, and irrelevant text.

Long context still has position bias. Models often use information near the beginning and end more reliably than information buried in the middle, a behavior studied as lost in the middle. Tokens can fit in the window and still have weak influence on the final answer.

The packing controls show three losses. Truncation keeps the newest logs and current request, but drops earlier constraints and failed commands. Retrieval spends the budget on high-scoring matches like workflow files, source files, and error lines. Compaction rewrites the prior conversation into a smaller memory, preserving decisions and evidence while losing exact wording and weak signals.

Vaswani et al. 2017 · Liu et al. 2023