7 Building robust agents with evaluation and feedback

Building reliable agentic systems depends on treating evaluation and feedback as core design elements rather than afterthoughts. The chapter explains that robust agents are shaped through multiple kinds of checks, including red teaming, benchmarks, human review, automated evaluators, grounding, and critic-style agents. It also emphasizes that feedback can come from users, other agents, or internal self-assessment, and that these mechanisms should be considered early and applied throughout development, not just at deployment.

A major theme is test-driven agent development, which adapts test-first software practices to agents by defining benchmarks before refining prompts and tools. The process starts with expected and incorrect outputs, then runs repeated tests because LLM behavior is variable, and then makes minimal changes to improve results while preserving simplicity and performance. The chapter shows how this works for a retrieval-augmented knowledge agent, first with deterministic string matching, then with clearer tool design and stronger instructions, and later with a separate evaluation agent that can judge more natural responses using typed pass/fail output and feedback.

The chapter then expands into grounding, critic, and rubric-based evaluation patterns for more complex outputs. Grounding agents verify that answers stay tied to source context and can act as guardrails that block or reject ungrounded responses, while critic agents assess generated content against rubric criteria and drive regeneration until standards are met. Finally, the chapter introduces Phoenix as an observability and evaluation platform for tracing agent behavior, attaching metadata and sessions, running experiments, and adding annotations, reinforcing the broader message that production agents should always be monitored, evaluated, and improved through feedback loops.

highlights the external agent evaluation and feedback systems for a single user-facing agent. In the figure a generative agent may face evaluation from a critic for general output feedback or a grounding agent for context specific feedback and evalution.

highlights the human process of developing agents using TDAD. Where we first think of goals and benchmarks, then test, develop the prompt, refactor, and then repeat the process.

Architecture of a simple RAG Knowledge Agent. The goal of the agent is to answer questions from a benchmark by searching its knowledge base and returning answers. A simple evaluator LLM evaluates the answer based on the expected and wrong answers from the benchmark test. The accurary of the RAG agent can be determined by the percentage of correct answers answered from the benchmark.

Three forms of agents for evaluating agent output: grunding, critic, and evaluation. The grounding is specific to evaluating RAG agent output. The critic agent is designed to critique generated content, and the evaluation agent is used for more complex output.

RAG knowledge agent coupled with a grounding agent. Both agents are provided with the same context. The RAG agent generates an answer from the context, and the grounding agent verifies that the answer is grounded by reviewing the same context.

Arize Phoenix deployment patterns and use case methods of tracing, tracking and evaluating agents.

Arize Phoenix projects dashboard showing an agent workflow, agent and LLM activity.

Viewing the metadata column and filtering traces by an attribute.

The process to add spans to a dataset for later experimentation and evaluation.

Viewing datasets and then clicking Run Experiment to view the generated code you can copy to a Python file to generate an experiment or evaluation.

demonstrates how to create a new Annotation in Phoenix.

demonstrates how to apply an annotation to an LLM response within an agent workflow, agent, and LLM response spans.

• Robust agents require evaluation and feedback baked in from the start—tie the Learn step of Sense-Plan-Act-Learn to external evaluators, a feedback store, and (optionally) a monitoring/alerting feedback agent.
• Use a mix of methods: red-team testing for safety breaks, benchmark testing for goal attainment, human feedback (thumbs/comments) with verification to reduce bias, and automated evaluators (grounding/critic/evaluation agents) where appropriate.
• Test-Driven Agent Development (TDAD) flips the process: define benchmarks (incl. wrong examples), expect initial failures, run each test multiple times for consistency, then make the minimum prompt/tool/model changes needed to pass—retest and refactor iteratively.
• When benchmarks conflict, decide explicitly: accept partial pass targets, split into specialized agents (plus triage), or refactor the agent’s instructions from first principles.
• Prefer tool guidance in the tool’s name/docstring over prompt clutter—rename tools and write clear docstrings; keep prompts lean and role-focused.
• Add evaluation agents to score complex, natural-language outputs with typed results (e.g., pass/fail + feedback), avoiding brittle string matching and enabling programmatic gating.
• Grounding agents are essential for RAG—verify answers are supported by provided context/citations; on failure, regenerate, block with a static response, or pass grounded feedback forward.
• Guardrails operationalize grounding—wrap agents with output guardrails that trip on ungrounded answers and surface feedback/flags for policy (retry, block, or annotate).
• Rubrics turn subjective quality into criteria and scales—use them with critic agents for high-variance generation (images, reports, graphs); loop generation ↔ critique until a passing score or attempt limit.
• Because LLMs are stochastic, benchmark for stability—not just accuracy—by rerunning tests and watching variance, latency, and token use.
• Observability is non-negotiable: OpenAI Traces show every tool call and LLM turn; Arize Phoenix (OpenTelemetry) adds projects, sessions, metrics, costs, experiments, and evaluators.
• Instrument Phoenix cleanly—set the collector endpoint, replace default processors, register a tracer, and wrap runs with named traces; attach sessions and metadata for filtering and cohort analysis.
• Build datasets from spans, run experiments/evaluators, and use annotations to capture structured human feedback—feed these signals back into prompts, tools, and policies.
• Production rule: never promote agents without active evaluation/feedback loops—benchmarks, grounding/guardrails, critics with rubrics, tracing, and monitored KPIs (latency, cost, quality) working together.

FAQ