13 Context compression and pruning

Long context in retrieval-augmented generation can make systems both slower and less accurate, even when the right document has been retrieved. As prompts grow, attention gets diluted, useful information can end up “lost in the middle,” and the model may read past the evidence and fall back on its own priors. The result is a mix of extraction failures, higher latency, and rising token costs, so the chapter argues that the real task is not simply retrieving more text, but handing the generator a smaller, cleaner context that preserves what matters.

The chapter traces context compression as an evolution of increasingly aligned scoring methods. Early heuristics like TF-IDF or BM25 were cheap but blind to the user’s question. Proxy perplexity improved things by using a small language model to score tokens, and LongLLMLingua made this practical by introducing contrastive perplexity, which keeps tokens that become more informative when the question is included. It then layers in coarse document filtering, reordering to exploit positional bias, fine-grained token pruning, and recovery for fragmented entities. AttentionRAG goes one step further by reading the generator’s own attention weights, reducing the mismatch between a proxy scorer and the model that will actually answer.

Beyond these runtime pruning methods, the chapter shows that stronger compression can come from trained synthesis methods such as RECOMP and CORE, which rewrite or select content to better support the downstream task, and from systems integration patterns like query-aware post-processors and recursive retrieval. It emphasizes that extractive compression is best when factual spans must be preserved, while abstractive compression is better for multi-hop or summarization tasks, though it adds hallucination risk. The main practical lesson is that compression ratio must be tuned to the task, the documents, and the deployment constraints: too little compression wastes money and attention, while too much can destroy meaning and flip the answer.

The LongLLMLingua Pipeline.

The two obstacles to reading attention, and the answer hint prefix that resolves them. Long sequences dilute softmax mass; multi-token queries scatter focal points. Reformulating the query as a fill-in-the-blank prefix forces a single focal token whose attention concentrates on the context that informs the prediction.

The AttentionRAG pipeline. Stage 1 retrieves chunks for the user's query. Stage 2 builds the answer hint prefix from the same query using a lightweight LLM. Stage 3 feeds each chunk and the prefix to the small open-weight pruning model and reads the attention from the chunk's tokens to the prefix's blank token, summed across all transformer layers; the highest-scoring sentences become the compressed context. The final generator can be any model, including a closed-source API.

• Long contexts make models reason worse: Attention spreads thin, and position starts to matter more than content.
• Contrastive perplexity finds what the question actually needs: Comparing a token's surprisal with and without the question conditioned on it separates relevant text from filler.
• LongLLMLingua budgets per document and reorders them: The best passage goes first and the second-best last, because models read the ends of a prompt and lose the middle.
• AttentionRAG reads the model's own attention: Framing the query as next-token prediction extracts the signal directly, with no proxy model to train or serve.
• Abstractive compression writes new text and can invent: Extractive pruning only deletes, which is the safer default when the output is going to be cited.
• Compression helps until it does not: Moderate ratios improve accuracy by removing distractors, and past roughly 6x the meaning goes with the tokens.
• KV-cache eviction is the other half: It shrinks the decoder cache rather than the prompt, so it pays off on long generations where prompt compression does nothing.

FAQ