8 Summarization and more effective metric functions
The chapter explains that summarization is a useful but difficult LM task to evaluate because quality depends on more than exact wording. A good summary must preserve the important ideas, avoid hallucinations, stay concise, and remain readable. DSPy helps by providing a framework for building both summarizers and the metric functions needed to optimize them, but the user still has to choose what “good” means for the task.
It first contrasts extractive and abstractive summarization, then shows how labeled and unlabeled data can be used for evaluation. With labeled examples, the chapter explores lexical metrics such as Jaccard similarity, n-gram overlap, precision, recall, and F1, while noting their limits with paraphrases and meaning changes. It then moves to semantic evaluation with embeddings, which compare meaning more directly but can be harder to interpret.
The chapter’s larger theme is that LLM-based judging can be the most flexible approach for summarization. It demonstrates judges for groundedness and completeness, and shows how multiple criteria can be combined into a single metric, including Q&A-style evaluation tailored to specific requirements like salience and named entities. By building, testing, and optimizing these judges separately from the summarizer, DSPy supports a practical workflow for improving summary quality and making evaluation more aligned with real application goals.
Summary
- There are two main types of summaries: extractive summaries based on quotes, and abstractive summaries that capture the main ideas, but may use different words or phrases.
- Like many other LM-based tasks, summaries are difficult to evaluate.
- To develop effective LM-based applications, it’s necessary to develop some meaningful way to evaluate the LM responses.
- When ground truth summaries are available, we can evaluate how similar the LM’s summaries are to the ground truth using lexical comparisons, embeddings, or LLM as a Judge-based methods.
- Lexical methods look at how similar the words or n-grams are between the two summaries.
- Lexical methods can struggle to evaluate summaries well, but they are very efficient and useful where they are sufficient.
- Using LLM as a Judge allows us to evaluate summaries when we don't have labeled data.
- Using LLM as a Judge with DSPy, we create DSPy programs to evaluate LM responses and execute them from within the DSPy metric functions.
- These judge programs can themselves be evaluated and optimized.
- To evaluate and optimize the judges, we should create a dataset that includes both good and poor responses.
- Using LLM as a Judge methods allows us to specify exactly how to evaluate the summary.
- Common measures include groundedness, completeness, readability, and conciseness.
- We may also check semantic precision and recall. Although similar to lexical precision and recall, semantic precision and recall check for matching ideas, as opposed to words or n-grams.
- We can create specific judges to evaluate each of these individual concerns or create a single judge with many output fields that checks each of the relevant concerns.
- QA-based methods allow us to enumerate exactly what we want to evaluate.
Building LLM Applications with DSPy ebook for free
