Managing Context Windows: Chunking, Summarization, and Memory

Large context windows solved the wrong problem. Even with a million tokens available, dumping everything into the prompt degrades quality — models attend unevenly across long context, relevant details get lost in the middle, and every extra token costs latency and money. Context management is still a real engineering discipline, not something a bigger window makes obsolete.

Long context isn't free retrieval

It's tempting to skip building a retrieval system and just pass the whole document set into a large context window. This works until it doesn't: cost scales linearly with tokens sent on every single call, latency grows with input length, and empirically models are less reliable at pulling out a specific fact buried in the middle of a huge context than a targeted retrieval of the top three relevant chunks would be. Long context is best used for cases where the model genuinely needs to reason across the whole document — synthesis, comparison, holistic review — not as a substitute for retrieving the right five paragraphs.

Conversation memory needs a policy, not just a buffer

The naive approach — append every turn to the context and truncate from the oldest when you hit the limit — throws away information in an arbitrary order, often losing the user's original goal from early in the conversation while keeping recent small talk. A better structure separates memory into tiers:

class ConversationMemory:
    def __init__(self):
        self.pinned = []       # goal, key facts stated once, never dropped
        self.recent = []       # last N turns, verbatim
        self.summary = None    # rolling summary of everything older than `recent`

    def build_context(self):
        parts = [f"Key facts: {self.pinned}"]
        if self.summary:
            parts.append(f"Earlier in this conversation: {self.summary}")
        parts.append(f"Recent turns: {self.recent}")
        return "\n\n".join(parts)

When recent exceeds a threshold, summarize the oldest chunk of it into summary rather than dropping it outright, and keep pinned facts (stated preferences, the original request, IDs referenced) out of the compaction entirely.

Summarization is lossy — decide what you're willing to lose

Rolling summarization trades completeness for length, and the failure mode is subtle: a summary written without knowing what will matter later can drop a detail the user references three turns after it was summarized away. Two mitigations help. First, summarize with an explicit instruction to preserve specifics (numbers, names, IDs, decisions made) over narrative flow — "the user asked about pricing" is worse than "the user asked about the $49/mo Pro tier specifically." Second, keep the raw turns available in storage even after they're summarized out of the active context, so a tool call can retrieve the verbatim exchange if the summary turns out to have dropped something load-bearing.

Position matters as much as content

Models attend more reliably to information near the start and end of the context than content buried in the middle — the "lost in the middle" effect holds across most current model families to varying degrees. When you control the ordering, put the highest-priority information (the actual user question, critical constraints) at the very start or very end of the prompt, and treat the middle of a long context as the place to put supporting material the model doesn't strictly need to answer correctly.

A rough decision guide

The common thread is that context budget is a resource to allocate deliberately, the same way you'd think about memory or bandwidth in any other system — not a pool to fill until it overflows.