5 Optimizing prompt examples

The chapter explains prompt optimization in DSPy as a data-driven process similar to machine learning training: generate many candidate prompts, test them on validation data, and keep the one that best improves the target metric. It focuses first on optimizing the demonstrations in a prompt rather than the instructions, using an intent classification task to show how labeled examples can make vague classes easier for a language model to distinguish. The key idea is that the best prompt is not found by exhaustive search, since the number of possible demo combinations is enormous, so DSPy provides practical optimizers and evaluation tools to search efficiently.

It then walks through several ways to choose demonstrations. LabeledFewShot simply samples labeled training examples at random and inserts them into the prompt, which is easy to use but not especially smart; repeated runs and different values of k can improve results. BootstrapFewShot adds a verification step using a metric function, accepting only examples whose LM predictions are good enough, which is especially useful when demonstrations need to include generated fields such as reasoning. The chapter also shows how teacher models can help during bootstrapping, and how BootstrapFewShotWithRandomSearch extends the idea by trying multiple candidate prompt sets and selecting the best-performing one on validation data.

Finally, the chapter introduces KNN optimization, which changes the process from selecting one fixed demo set to choosing the most similar training examples for each individual LM call using embeddings and vector similarity. This can reduce prompt size while keeping demonstrations relevant to each input, though it depends on the quality of the training set and embedder. Across the experiments, every optimizer improves over the unoptimized baseline, with random-search variants and well-chosen labeled demos often performing best, while KNN remains competitive and efficient. The main takeaway is that prompt examples matter a great deal, and DSPy makes it much easier to search for and evaluate strong demo sets than manual prompt engineering alone.