4 In-context learning, few-shot, and RAG

This chapter introduces in-context learning as a practical first step for adapting an LLM, showing how few-shot prompting can solve a support-ticket classification task without any training. The core idea is to use the prompt itself to steer the model by providing a clear instruction, a small set of labeled examples, and a query in a consistent schema. The examples should be diverse, representative of real production traffic, and ordered so the most relevant ones appear last, since recency can meaningfully affect results. The chapter also shows that prompt quality is not just about accuracy on a design set: you need held-out evaluation, run-to-run stability checks, and attention to brittleness before shipping anything to production.

From there, the chapter expands from few-shot to many-shot prompting for tasks whose input distributions are too broad for a tiny example bank. Long-context models make it possible to include dozens or even hundreds of demonstrations, which can raise accuracy when coverage is the main problem, especially for high-variability classification and structured extraction. But the extra examples are not free: they increase token cost, can suffer from lost-in-the-middle attention effects, and eventually hit diminishing returns. The chapter emphasizes diagnosing the failure mode first—coverage gap, reasoning bottleneck, or missing knowledge—because more examples only help in the first case. It also shows a lightweight way to assemble many-shot prompts by selecting examples through semantic similarity rather than manual picking.

When the model lacks needed facts rather than pattern coverage, the chapter recommends retrieval-augmented generation. RAG adds a retrieval step in front of the model so the prompt includes relevant source material at inference time, making it the right tool for fresh, private, or internal knowledge the base model does not already contain. The chapter breaks RAG into its main components—chunking, embeddings, vector indexing, retrieval, and prompt assembly—and explains how to evaluate retrieval quality separately from answer quality. It closes with a decision framework for choosing among prompting, RAG, and parameter-efficient fine-tuning: stay with prompting when it works, use RAG for knowledge gaps, and move to fine-tuning when prompting plateaus, the task needs a behavior change, or the economics and latency of repeated prompting no longer make sense.

The in-context learning ladder: zero-shot, one-shot, few-shot, many-shot, and RAG sit on a single rung of the adaptation continuum, with engineering cost climbing gradually within the rung before chapter 5's parameter-update techniques take over.

Few-shot prompt anatomy: a system instruction names the task, labeled examples set the input/output schema, and a trailing “Category:” marker shows the model where its answer goes.

Lost-in-the-middle U-shape (illustrative): accuracy is high when the relevant content sits near the start or end of the context window and drops in the middle, so put your highest-signal examples at the boundaries, not the trough.

Typical validation cycle: six steps from raw production data to a shippable prompt.

Minimal RAG architecture: the ingestion path (chunker, embedder, vector index) runs once at index time; the query path (retriever, prompt assembly, LLM) runs on every request, and both paths meet at the shared vector index.

Accuracy versus shot count on the ticket-classification task (illustrative): the first plateau near eight shots is where most teams stop, with a smaller second climb between fifty and one hundred shots, where many-shot prompting earns its place.

The graduation decision tree: start with prompting, add RAG when the diagnostic shows a knowledge gap, and graduate to parameter-efficient fine-tuning when prompting plateaus or per-request cost flips the economics.

• A few-shot prompt with five to eight well-chosen examples often delivers most of the quality of fine-tuning on classification, summarization, and extraction tasks at the cost of inference tokens only.
• Long-context models enable many-shot prompting with fifty to two hundred examples; this scales further than the diminishing-returns curve from earlier years suggests, but per-request cost grows linearly with shot count.
• Held-out sets scale with the decision: 20 for a smoke test, 50 to resolve large gaps, and a few hundred to report small differences. Run-to-run variability measurement is what separates a brittle prompt from a production-ready one.
• Retrieval-augmented generation closes knowledge gaps; fine-tuning closes behavior gaps; the diagnostic signal (right-document-but-wrong-format vs. wrong-fact) tells you which technique to use.
• A minimal RAG pipeline (vector embeddings, fifty lines of NumPy retrieval) ships in days; production extensions add chunking strategy, reranking, query rewriting, and hybrid retrieval. Evaluate retrieval hit rate, answer faithfulness, and end-to-end accuracy separately. The three answer different debugging questions.
• The prompting-to-fine-tuning crossover is set by per-request cost and accuracy ceiling; RAG sits as a third cost lane that wins when the gap is knowledge-bound, and parameter-efficient fine-tuning becomes the right tool when the gap is behavior-bound, and volume is high.

FAQ