6 Optimizing prompt instructions
This chapter introduces how DSPy optimizes prompt instructions, complementing the earlier focus on demonstrations. It explains that several optimizers can be used for this purpose, including COPRO, MIPROv2, InferRules, SIMBA, GEPA, and Ensemble, and emphasizes that no single method is universally best. Instead, the right choice depends on the task, the model, and the amount of compute you can spend, so it is often worthwhile to try multiple optimizers and compare results.
COPRO serves as the simplest starting point, using a hill-climbing style search that repeatedly generates instruction variants, evaluates them on validation data, and keeps the best one. The chapter then shows how MIPROv2 expands this idea by optimizing instructions and demonstrations together, using grounding information about the program and dataset to produce better candidate prompts, and then using Bayesian optimization to efficiently search combinations. InferRules takes a more rule-driven approach by extracting concise natural-language rules from examples and inserting them into the prompt, while SIMBA and GEPA add more reflective, iterative strategies that use feedback, history, and in GEPA’s case a genetic-algorithm style Pareto-front search to evolve stronger prompts.
The chapter also highlights practical concerns such as cost, because these optimizers can make many LM calls, especially with larger search settings. It explains how prompt and task models can be separated so a stronger model can be used during optimization and a cheaper one in production, how optimized programs can be chained together or combined through ensembling, and how final programs can be saved and reloaded. Overall, the main message is that instruction optimization in DSPy is a flexible search problem: better prompts usually come from guided iteration, grounded candidate generation, and careful evaluation rather than from a single manually written instruction.
The COPRO algorithm. COPRO loops a specified number of iterations. Each iteration takes the current best prompt, creates several variations, and evaluates each one. It then determines the new best prompt and loops back. In the end, COPRO returns the best prompt discovered. This may be discovered in any iteration, but usually it’s one of the later iterations, as each iteration tends to produce stronger prompts.
The MIPROv2 algorithm Step 3. In Steps 1 and 2, we create the candidate sets of demos and the candidate instructions. In Step 3, we try various ways to combine these until we determine the one that performs the best. As there is often a large number of combinations, this uses a Bayesian Optimization method to determine which combinations to try. In this example, we try Set of demos #5 and Instructions #3, creating a prompt that is tested on all or part of the validation set with the task LM. The score is recorded, which allows us to track which sets of demos work well, which instructions work well, and which work well together. This helps us determine what other combinations to test in subsequent iterations.
The first 4 steps in generating and evaluating the candidate prompts in the GEPA optimizer. The nodes indicate a single prompt. Those in gray are in the Pareto Front and may be used to create subsequent candidate prompts. After Step 1, we have a single prompt, which is the prompt related to the DSPy program passed to the GEPA optimizer. In Step 2, we mutate this, creating Prompt 2, and evaluate that on the validation set. As long as this prompt is the top-performing prompt on at least one Example in the validation set, it becomes part of the Pareto Front. In Step 3 we can either mutate or combine prompts from the Pareto Front. In this case we combine Prompts 1 and 2, creating Prompt 3. This is the top-scoring on at least one Example, so it enters the Pareto Front, but Prompt 1 is now no longer part of the Pareto Front. In Step 4, we can again either mutate or combine elements of the Pareto Front. In this case, we mutate Prompt 3, creating Prompt 4, which also becomes part of the Pareto Front.
Summary
- DSPy includes multiple optimizers to optimize the instructions.
- Some optimizers (MIPROv2, InferRules, and SIMBA) can also select demonstrations to include in the prompts.
- As well as including demonstrations and instructions, some optimizers (including InferRules and SIMBA) can generate rules summarizing the patterns in the training examples.
- COPRO uses a method similar to Coordinate Ascent or Hill Climbing to iteratively improve upon a set of instructions.
- MIPROv2 optimizes both instructions and demonstrations through Bayesian optimization
- SIMBA iteratively improves a prompt, each step either adding a rule or a demonstration and potentially removing demonstrations.
- GEPA uses a Genetic Algorithm to maintain a Pareto front to identify the strongest prompt.
- Both SIMBA and GEPA use reflection, examining a history of previous prompts, their responses, and the scores, to generate improved prompts.
- Both SIMBA and GEPA also support feedback from the metric function
- Many programs can be combined using the Ensemble optimizer.
- Many optimizers can be applied, one after the other, to a single program to create a highly optimized program.
- SIMBA and GEPA are currently considered the strongest optimizers within DSPy, but the best performing for any given task depends on the task and LM.
Building LLM Applications with DSPy ebook for free
