4 Evaluating DSPy programs

Before putting an LM-based application into production, it needs to be evaluated both to guide optimization and to estimate how reliable it will be in real use. The chapter emphasizes that evaluation is especially important for prompt-based systems because model outputs can vary from call to call, making it hard to judge which prompt, model, or module is actually best without an automated testing setup. DSPy is presented as a practical way to handle this process, reducing manual effort and making it easier to compare baseline behavior, consistency, cost, and speed across many runs.

A central part of the evaluation workflow is preparing the right data and metric. The text explains how to build test examples with DSPy’s Example class, including the required inputs and gold labels, and how to assemble datasets from real, synthetic, or mixed sources. It also discusses how to split data into training, validation, development, and test sets so that optimization and final assessment remain separate. Because the metric is often the hardest piece to define, the chapter shows how to write a custom Python function that scores each prediction, starting with simple exact-match classification logic and then considering more flexible scoring options when needed.

Once the data and metric are in place, the chapter compares manual evaluation with DSPy’s Evaluate class. Manual loops can work, but Evaluate is cleaner and faster because it runs examples in parallel, manages progress reporting, and returns structured results automatically. The chapter then shows how to go beyond a single accuracy number by examining per-class performance, calculating F1, checking consistency by repeating examples, and comparing different models and prompting modules. Across these experiments, the main lesson is that evaluation should be repeated, multi-dimensional, and grounded in realistic data, since small data splits and stochastic model behavior can easily mislead conclusions.

The middle stage of prompt programming—Baseline Evaluation

Sample of output from a call to Evaluate. For each of the 5 Examples, all fields are shown from the Example and from the Prediction (here, a single field in the pred_intent_label column), and the score is given by the metric function. With this metric function, the scores are boolean, so they will all be True or False.

• Evaluation is central to prompt programming and is well-supported by DSPy.
• Evaluation is important both to optimize models and to estimate how well they would perform if put into production.
• We normally split the data we have into at least three sets: the train, validation, and test sets, though more sets may be used.
• As well as real data, it’s common, and useful, to use both manually created and LM created synthetic data.
• Defining a metric function is typically the most difficult part of working with DSPy, though it's straightforward with classification problems.
• The metric function will score a single LM response for a single example.
• Reducing the per-example scores to a meaningful overall score indicates how well the module performs on a given test set.
• To determine if we’re improving the model, it’s useful to evaluate the first (baseline) model as well as other programs we create later.
• For optimization, we use a single metric function, but for evaluation, we may want to use multiple functions to gain a fuller picture of how well the module performs.
• It’s important to test the metric function before using it to evaluate a module.
• Creating unit tests allows us to maintain a set of tests for the metric functions.
• We can manually evaluate models, but using DSPy’s support for evaluation is generally easier, cleaner, and faster, leading to greater consistency.
• We should be mindful of LM rate limits and also take advantage of our ability to execute the evaluation in parallel.
• Evaluating LM-based applications without tools such as DSPy is possible, but requires more effort and more code.
• As well as the quality of the LM responses, it’s useful to evaluate the consistency.
• Currently, out of the LMs and modules tested so far, using gpt-4o-mini with Predict appears to be the best combination.
• This result isn't conclusive at this point, as what’s most relevant is which combination, once optimized, performs best.

FAQ