8 The Experimentation Service: improving what your AI does

The chapter presents the Experimentation Service as the mechanism that turns AI observability into improvement. While observability answers what is happening, experimentation answers what should change by managing versioned targets, curated datasets, evaluations, online scoring, A/B tests, and human annotation. Together, these components create a continuous loop where production traces reveal weaknesses, scores highlight low-quality behavior, and those findings feed the next round of changes.

A central idea is that every change to an AI system should follow a disciplined lifecycle. Prompts, model configurations, and retrieval settings are treated as experiment targets that move from draft to testing to active, with clear metadata, change descriptions, and evaluation summaries attached. The service does not duplicate content stored elsewhere; instead, it references existing artifacts and adds the structure needed to compare versions, preserve history, and promote only improvements that are backed by evidence.

The chapter also explains how evaluation works at multiple levels. Offline evaluation uses test datasets built from both curated cases and real production traces, giving teams a way to catch regressions before deployment. Online scoring monitors live traffic asynchronously with automated and LLM-based judges, while annotation queues route selected traces to humans for ground truth, calibration, and dataset growth. Finally, A/B testing assigns users consistently to variants, records outcomes, and applies statistical analysis so teams can make informed tradeoffs based on reliable results rather than intuition.

The improvement loop. Production traffic generates traces. Scores (automated, LLM-based, and human) attach quality signals. Low-scoring traces flow into evaluation datasets. Offline evals compare prompt versions. Winning versions enter A/B tests. Results feed back into production. This continuous cycle turns static AI deployments into systems that improve over time.

Experiment target lifecycle. Targets flow from DRAFT through TESTING to ACTIVE. Failed evaluations return a target to DRAFT for iteration. Promoting a new target to ACTIVE automatically deprecates the previous active one. This lifecycle prevents untested changes from reaching production and maintains an audit trail of every experiment.

Offline evaluation pipeline. Targets and a test dataset feed into the pipeline. For each test case and target combination, the pipeline generates a response through the Model Service, scores it with each configured metric, and aggregates results. The output includes per-target scores, per-test-case details for debugging, and a comparison with improvement percentages.

A/B testing architecture. Incoming requests are assigned to variants through consistent hashing on the patient ID, ensuring the same patient always sees the same variant (preventing jarring style changes mid-conversation). The 50/50 traffic split sends half to the current prompt and half to the new candidate. Outcome metrics (quality scores, turns to resolution, cost) are collected for both variants and compared with statistical significance testing. The experiment concludes when enough samples exist to confidently recommend a winner.

Asynchronous scoring and experiment feedback loop for a patient intake request. After the response is returned, scoring rules evaluate the completed trace and record results. If an A/B test is active, the experimentation service associates those outcomes with the assigned variant to support comparison and improvement.

• The Experimentation Service manages the improvement loop: register experiment targets, curate datasets, run offline evaluations, score production traffic online, A/B test with statistical significance, and route traces to human reviewers through annotation queues.
• Experiment targets are generic: the same lifecycle (draft → testing → active → deprecated) applies to prompt versions, model configurations, and retrieval parameters. The Experimentation Service doesn't duplicate storage; it references artifacts in their home services and adds lifecycle states, evaluation summaries, and change descriptions.
• Evaluation datasets combine hand-curated test cases with production traces. The production-to-dataset pipeline flows low-scoring and edge-case traces into test suites, keeping evaluation aligned with real user behavior.
• Offline evaluation compares variants against datasets; online evaluation scores live traffic; human annotation queues provide ground truth. The three modes reinforce each other in a continuous improvement cycle.
• A/B testing uses consistent hashing to ensure the same user always sees the same variant, preventing jarring experience changes mid-conversation. Statistical significance tracking tells teams when enough data exists to make a confident decision.
• The developer's workflow code stays clean. Online scoring evaluates traces asynchronously, and the only experimentation-specific code is variant assignment and outcome recording, both of which are no-ops when no experiment is running.

FAQ