9 Evaluation and Performance for LLMs and Agents
This material explains how reliable LLM and agent systems require evaluation and performance engineering together. As applications grow from simple prompts to structured outputs, retrieval, tool use, and multi-agent workflows, failures become harder to detect: hallucinations may appear confident, agents may choose the wrong tools, and production systems may slow down or become too expensive under load. The central message is that correctness, scalability, latency, and cost must be measured continuously rather than treated as one-time development concerns.
A major focus is hallucination detection. The text presents a systematic process: define authoritative grounding data, build both typical and adversarial test sets, extract factual claims from model responses, validate them against trusted sources, and report metrics such as Grounding Defect Rate, severity scores, topic-level breakdowns, FActScore, ROUGE, and LLM-as-judge evaluations. It also emphasizes the limits of each method: ROUGE can miss meaning, LLM judges can introduce their own biases, and adversarial red teaming is needed to uncover edge cases that automated tests may not find. Production monitoring frameworks are introduced as a way to track hallucination rates over time, detect drift, and alert teams when output quality degrades.
The material then turns to performance patterns for production LLM systems: token streaming for perceived responsiveness, batching for traffic surges and offline workloads, semantic caching to reuse answers for repeated or similar queries, and multi-model routing so simple questions use cheaper models while complex ones escalate to stronger models. These techniques are combined in an e-commerce support example designed to remain fast and cost-effective during high-traffic events. Finally, the text explains that agent evaluation must go beyond final-answer accuracy: teams should test end-to-end task completion, verify tool calls, inspect execution trajectories, and compare each decision step against the expected workflow. Together, these practices form a toolkit for building LLM and agent systems that are accurate, efficient, observable, and resilient at scale.
FActScore metric that evaluates factual consistency.
The Red-Teaming process for Large Language models
Four key patterns to make LLM deployments more efficient: token streaming, batch processing, semantic caching, multi-model selection
Batching user queries so they don’t overwhelm your server
Semantic caching explained: User query, embeddings generation, vector database, response,
The Agent end-to-end testing and evaluation pipeline
Summary
- Hallucination measurement requires establishing grounding data, creating generic and adversarial test sets, validating claims, and tracking metrics like GDR and Hallucination Severity Score
- FActScore breaks responses into atomic facts for granular accuracy measurement, while ROUGE provides faster text-overlap comparisons against reference responses
- LLM-as-judge offers semantic evaluation but requires mitigation strategies like majority voting and human calibration to address judge bias
- Token streaming delivers responses incrementally for an interactive experience, while full-response generation provides complete answers when coherence is paramount
- System-level batching groups requests to handle traffic surges efficiently, whereas OpenAI's Batch API processes large offline jobs at reduced cost
- Semantic caching identifies similar queries using vector embeddings, dramatically reducing redundant API calls and lowering costs while maintaining response quality
- Multi-model fallback routes simpler queries to faster, cheaper models (like gpt-5.4-mini) and reserves advanced models (like gpt-5.4) only for complex questions
- Agent evaluation requires assessing both the end results and the execution process, using techniques like LLM-as-judge to verify output correctness
- Trajectory analysis helps identify where agents make wrong decisions during multi-step processing workflows
- Combining these optimization patterns creates resilient, cost-effective LLM applications that can handle production-level demands while maintaining performance
FAQ
What are the four steps for identifying and measuring hallucinations in LLM systems?
The chapter recommends a four-step methodology:
- Identify grounding data: Establish authoritative sources such as product catalogs, shipping policies, return rules, and support knowledge bases.
- Create measurement test sets: Build both generic test sets for common production-like queries and adversarial test sets for edge cases and ambiguous prompts.
- Extract and validate claims: Break model responses into factual claims and compare them against grounding data using automated checks and human review.
- Report metrics: Track measures such as Grounding Defect Rate, Hallucination Severity Score, and topic-wise performance.
What is Grounding Defect Rate, and why is it useful?
Grounding Defect Rate (GDR) is the ratio of ungrounded responses to total generated outputs. For example, if 8 out of 100 responses contain unsupported or incorrect claims, the GDR is 8%.
GDR is useful because it gives teams a simple way to track hallucination risk over time. It can reveal regressions after prompt, retrieval, or model changes, and it can be broken down by topic to show where the system struggles most, such as shipping policies or product specifications.
How does FActScore differ from response-level hallucination metrics?
Response-level metrics often treat an entire response as either grounded or hallucinated. FActScore is more granular: it breaks a response into atomic factual claims and checks each claim independently.
For example, if a response says a product has a 6-hour battery life, a 1-year warranty, and a sale price of $79.99, each statement is evaluated separately. If two of the three facts are correct, the FActScore is 0.67. This gives more nuance than simply labeling the whole response as wrong.
When is ROUGE useful for evaluating LLM outputs, and what are its limitations?
ROUGE measures text overlap between generated output and a reference answer. ROUGE-1 checks unigram overlap, ROUGE-2 checks bigram overlap, and ROUGE-L checks the longest common subsequence.
ROUGE is useful when you have reference responses and want a fast signal for summarization or expected-answer tasks. However, it has important limitations for hallucination detection: a high ROUGE score does not guarantee factual correctness, and a low ROUGE score may simply mean the model paraphrased the answer accurately. It measures surface similarity, not true factual grounding.
What is the LLM-as-judge approach, and what risks does it introduce?
LLM-as-judge uses another language model to evaluate generated responses against a reference or grounding source. The judge model can identify unsupported claims, assign severity, and provide explanations.
The main risk is that the judge model can hallucinate or show bias itself. It may favor longer answers, prefer responses written in its own style, or miss subtle factual errors. Production systems reduce this risk by using multiple judges, majority voting, human validation on random samples, and retrieval-based fact-checking against structured data.
Why is red teaming important for LLM and agent evaluation?
Red teaming is adversarial testing performed by people or systems trying to make the AI fail. It is important because automated metrics usually catch known or systematic problems, while red teams uncover unexpected edge cases and vulnerabilities.
For an e-commerce bot, red teams might try to make it invent refund policies, reveal internal business information, promise impossible delivery times, or hallucinate nonexistent products. Red teaming should be used as a complement to metrics, benchmarks, and monitoring, not as a replacement.
How do monitoring frameworks like Arize AI and Phoenix help detect hallucinations in production?
Production systems process too many interactions for manual review, so monitoring frameworks help evaluate outputs at scale. Arize AI provides LLM observability, while Phoenix offers open-source evaluation tools and templates.
Phoenix includes components such as HALLUCINATION_PROMPT_TEMPLATE, HALLUCINATION_PROMPT_RAILS_MAP, and llm_classify. These allow teams to compare responses against references, label outputs as factual or hallucinated, collect explanations, track hallucination rates over time, and alert when rates exceed a threshold.
What are the four main architectural patterns for improving LLM performance?
The chapter highlights four core performance patterns:
- Token streaming: Show tokens as they are generated to improve perceived responsiveness.
- Batching: Smooth traffic spikes through system-level queues or use the OpenAI Batch API for large offline jobs.
- Semantic caching: Reuse answers for repeated or semantically similar queries to reduce cost and latency.
- Multi-model fallback: Route simple queries to cheaper models and complex queries to more capable models.
These patterns can be combined to build systems that remain responsive and cost-effective under heavy traffic.
How does semantic caching differ from exact string caching?
Exact string caching only works when a user repeats the same query character-for-character. For example, it can reuse an answer for “What are your store hours?” only if the exact same text appears again.
Semantic caching uses embeddings to match meaning rather than exact wording. It can recognize that “What time do you close?” and “Tell me your operating hours” are asking for similar information. If the similarity score exceeds a threshold, such as 0.7 or 0.75, the system returns the cached response instead of calling the LLM again.
Semantic caching is especially valuable for customer support bots, e-commerce assistants, and knowledge retrieval systems where users often ask the same questions in different words.
How should agents be evaluated differently from standalone LLMs?
Standalone LLM evaluation often focuses on final text quality, hallucination rate, latency, and token efficiency. Agents require additional evaluation because they classify intent, select tools, execute actions, and follow multi-step workflows.
The chapter recommends evaluating agents at multiple levels:
- End-to-end testing: Check whether the agent completes the full user task correctly.
- Final-output judging: Use LLM-as-judge or reference comparisons to assess the final answer.
- Tool-call evaluation: Verify that the agent called the correct tools, such as refund processing or database lookup.
- Trajectory analysis: Compare the agent’s actual execution path against the expected sequence of steps.
This helps reveal hidden workflow problems even when the final response appears correct.
Building Reliable AI Systems ebook for free
