4 Understanding what your users really want

Successful conversational AI hinges on reliably understanding what users want. The chapter explains how weak understanding shows up in practice—irrelevant or wrong answers, rising escalations and abandonment, and declining satisfaction—and traces it to accuracy and coverage gaps, stale or non-representative training data, and unvetted changes that compound over time. It stresses that chatbots are not “set and forget” assets: they require ongoing ownership, guardrails for updates, and a maintenance mindset akin to developing and coaching a digital employee.

It contrasts how traditional classification systems and generative models convey “understanding.” Classifiers learn intents and entities and must balance breadth versus depth, focusing on high- and medium-volume topics while handling the long tail with fallbacks. Generative AI simulates understanding through model selection, prompt design, few-shot examples, parameter tuning, and retrieval-augmented generation; most production systems blend both approaches. Measuring understanding depends on the stack and phase: for classifiers, use accuracy, precision, and recall via k-fold, AI-generated blind sets, or representative blind sets from production; for generative AI, define clear quality criteria (format, tone, safety, factuality, completeness) and build “golden” examples through human judgment. Direct user feedback (e.g., ratings) can complement these methods when used intentionally.

The chapter provides a practical, data-first workflow to assess current performance and drive iterative improvement. Start by auditing scope, training data, and dialog design (for classifiers) or prompts, models, parameters, data sources, and RAG configuration (for generative). Gather production logs over time, pare them to strong candidates (often first-turn utterances), filter out unintelligible, out-of-domain, ambiguous, and multi-intent cases unless designed for them, scrub PII, and preserve realistic noise like typos or common speech mis-transcriptions. Annotate utterances to create reusable “golden” test sets—intent labels for classifiers or ideal responses for generative—and compute a baseline; compare real demand to training distribution to spot gaps and misalignment. Use these insights to prioritize fixes, gatekeep changes, retest to confirm impact, and refresh tests regularly as user needs and the external world evolve.

Accuracy or coverage problems frustrate the user because it takes more time — and sometimes multiple contacts — to achieve their goal. It also causes the user to lose confidence in the virtual agent.

A retroactive assessment of classifier performance shows a hard-won lesson on the impact of untested changes over time. Had each version been tested as part of a pre-deployment process, the team would have postponed any version updates until the classifier issues were resolved. It took several weeks to get the classifier back into good working order, but it took several more weeks to plan and implement the corresponding changes now that the conversational logic was tied to 20 additional intents.

As request volume tapers off to the right, the chart has the appearance of a long tail. Each use case must define the optimum tradeoff of depth and breadth as it relates to topic coverage. The cutoff point for business value is usually somewhere in the “moderate volume” range. This is not to say that all low volume requests should be excluded, but there may be diminishing returns associated with extending your classifier’s coverage for these topics.

Traditional classification models use supervised learning to predict one of several pre-defined classifications. They look for the intent, or meaning, of a user input. Generative models use decoding transformers to create a text completion. They predict the next sequence of tokens (loosely, words or characters) that are most likely to occur after the user input.

K-fold cross validation and generated test data are suitable for situations where representative data is not available. Once a solution is deployed to production, representative blind test data will produce the most reliable measurement of your classifier’s ability to understand.

This classifier has thirteen intents. The training example counts range from seven to thirty.

Once you obtain some data, each utterance should first be sorted into buckets to identify potential candidates; this will separate the good, usable test data from the bad or irrelevant user inputs. The data may also need to be scrubbed to fix issues like PII. After that, the data will need to be annotated. (For classifiers, labeled with the correct intent. For generative AI, associated with an ideal output response.) Finally, the data needs to be converted into one or more sets that can be consumed by your testing tool.

Raw chat logs show that a user’s primary goal is often captured in the first turn of a conversation, but sometimes it occurs as an additional request later in the conversation, or after exchanging a pleasantry. It might even follow an opt-out request. Selecting the first row will usually yield enough usable data while reducing the amount of time your annotators spend sorting through the utterances that aren’t useful to the classifier, such as button clicks, common responses, and PII or other user-specific information. (The structure of your logs may vary by tool.)

The dark bars represent the number of training examples in a system. The light bars represent the number of annotated utterances for each intent. If your chart follows a similar pattern, your training priorities are probably in good alignment with the demands of your solution.

The training example counts (dark bars) show a large disparity across many intents, as compared to the annotated log data (light bars). Without accuracy numbers for each intent, we cannot immediately tell if this disparity is having a negative impact. However, we can make some observations, such as: 1) the first five intents are not nearly as important to our users as we thought they might be, and 2) the intents with the highest volume in our logs (the light grey bars) may be a lot more important to our users than we predicted.

• Chatbots demonstrate good understanding when they identify what a user wants and they provide a satisfactory answer or progress the user towards their goal.
• For traditional AI, understanding relies on at least two mechanisms: correct classification of an intent and an ability to deliver an output based on that classification. (Additional mechanisms such as entity detection or context may modify or personalize outputs.)
• For generative AI, understanding relies on the utterance and any accompanying prompt to create a response meant to address a user’s question or goal.
• Weak understanding is detrimental to business value and is often exhibited by a chatbot returning wrong answers, or no answers at all.
• You can’t assess the performance of your chatbot without first collecting some data.
• Chatbot understanding is usually measured in terms of accuracy, or the rate at which the solution delivers a correct answer or takes the correct action.
• There are multiple tools and methods for measuring understanding. Some are dependent on the type of AI and/or the current phase, whether pre-deployment or post.
• A representative “golden” test set, curated from real user utterances (production logs), can be used to measure the bot’s baseline performance, AND can be converted into a reusable asset to measure the impact of future changes.
• You should plan to monitor and re-train your solution throughout the life of the bot.
• Updates to training may require corresponding updates to the blind test set.

FAQ