Effective Conversational AI you own this product

Chatbots that work

Andrew Freed, Cari Jacobs, Eniko Rozsa
Foreword by Jesús Mantas

April 2025
ISBN 9781633436404
328 pages

Included with a Manning Online subscription

printed in black & white

available in Russian

catalog / Data Science / Machine Learning / Natural Language Processing

resources: Source code Book forum Source code on GitHub

table of content

Part 1 Framework for improving conversational AI

1 What makes conversational AI work?

1.1 Introduction to conversational AI

1.1.1 Why use conversational AI?

1.1.2 How does conversational AI work?

1.1.3 How you build conversational AI

1.2 Introduction to generative AI in conversational AI

1.2.1 What is generative AI

1.2.2 Generative AI guardrails

1.2.3 Effectively using generative AI in conversational AI

1.3 Introducing continuous improvement in conversational AI

1.3.1 Why continuously improve

1.3.2 The continuous improvement cycle

1.3.3 Communicating continuous improvement to stakeholders

1.4 Follow along

Summary

2 Building a conversational AI

2.1 Building an FAQ bot

2.1.1 FAQ bot foundations

2.1.2 Static question and answering

2.1.3 Dynamic question and answering

2.2 Routing agents and process-oriented bots

2.2.1 Routing agents

2.2.2 Transitioning from a routing agent to a process-oriented bot

2.3 Responding to the user with generative AI

2.3.1 Integrating with an LLM

2.3.2 Routing requests to an LLM

Summary

3 Planning for improvement

3.1 Knowing when you need to improve

3.2 Your cross-functional team

3.3 Driving to the same goal

3.3.1 Revisit business goals

3.3.2 Effectiveness

3.3.3 Coverage

3.4 Identifying and resolving problems

3.4.1 Finding problems

3.4.2 Group review

3.4.3 Determining acceptance criteria

3.5 Developing and delivering fixes

3.5.1 Sprint planning

3.5.2 Measure again

Summary

Part 2 Pattern: AI doesn’t understand

4 Understanding what your users really want

4.1 Fundamentals of understanding

4.1.1 The impact of weak understanding

4.1.2 What causes weak understanding?

4.1.3 How do we achieve understanding with traditional conversational AI?

4.1.4 How do we achieve understanding with generative AI?

4.2 How is understanding measured?

4.2.1 Measuring understanding for traditional (classification-based) AI

4.2.2 Measuring understanding for generative AI

4.2.3 Measuring understanding with direct user feedback

4.3 Assessing where you are today

4.3.1 Assessing your traditional (classification-based) AI solution

4.3.2 Assessing your generative AI solution

4.4 Obtaining and preparing test data from logs

4.4.1 Obtaining production logs

4.4.2 Guidelines for identifying candidate test utterances

4.4.3 Preparing and scrubbing data for use in iterative improvements

4.4.4 The annotation process

4.5 What does the data tell us?

4.5.1 Interpreting annotated logs for traditional (classification-based) AI

4.5.2 Interpreting annotated logs for generative AI

4.5.3 The case for iterative improvement

Summary

5 Improving weak understanding for traditional AI

5.1 Building your improvement plan

5.1.1 Identify problematic patterns in misunderstood utterances

5.1.2 Incremental improvements

5.1.3 Where to start: Identifying the biggest problems

5.2 Solving “wrong intent matched”

5.2.1 Improve recall for one intent

5.2.2 Improve precision for one intent

5.2.3 Improve the F1 score for one intent

5.2.4 Improve precision and recall for multiple intents

5.3 Solving “no intent matched”

5.3.1 Clustering utterances for new intents

5.3.2 When to stop adding intents

5.4 Supplementing traditional AI with generative content

5.4.1 Combining traditional and generative AI for an intent

5.4.2 Prompting to convey understanding

Summary

6 Enhancing responses with retrieval-augmented generation

6.1 Beyond intents: The role of search in conversational AI

6.1.1 Using search in conversational AI

6.1.2 Benefits of traditional search

6.1.3 Drawbacks of traditional search

6.2 Beyond search: Generating answers with RAG

6.2.1 Using RAG in conversational AI

6.2.2 Benefits of RAG

6.2.3 Combining RAG with other generative AI use cases

6.2.4 Comparing intents, search, and RAG approaches

6.3 How is RAG implemented?

6.3.1 High-level implementation

6.3.2 Preparing your document repository for RAG

6.4 Additional considerations of RAG implementations

6.4.1 Can’t we just use an LLM directly?

6.4.2 Keeping answers current and relevant with RAG

6.4.3 How easy is it to set up the ingestion pipeline?

6.4.4 Handling latency

6.4.5 When to use a fallback mechanism and when to search

6.5 Evaluating and analyzing RAG performance

6.5.1 Indexing metrics

6.5.2 Retrieval metrics

6.5.3 Generation metrics

6.5.4 Comparing efficiency of indexing and embedding solutions for RAG

Summary

7 Augmenting intent data with generative AI

7.1 Getting started

7.1.1 Why do it: Pros and cons

7.1.2 What you need

7.1.3 How to use the augmented data

7.2 Hardening your existing intents

7.2.1 Get creative with synonyms

7.2.2 Generate new grammatical variations

7.2.3 Build strong intents from LLM output

7.2.4 Creating even more examples with templates

7.3 Getting more creative

7.3.1 Brainstorm additional intents

7.3.2 Check for confusion

Summary

Part 3 Pattern: AI is too complex

8 Streamlining complex flows

8.1 The pain of complexity

8.1.1 Complexity’s effect on the end user

8.1.2 Complexity’s effect on business metrics

8.1.3 The incremental cost and benefit of reducing complexity for the user

8.2 Simplifying and streamlining the user journey

8.2.1 Spotting complex dialogue flows

8.2.2 Using what is known about the user

8.2.3 Aligning with the user’s mental model

8.2.4 Allowing flexibility in the expected user responses

8.2.5 Supporting self-service task flows with API/backend processes

Summary

9 Harnessing context for an adaptive virtual assistant experience

9.1 Importance of context in virtual assistant performance

9.1.1 How context influences user interactions

9.1.2 What is contextual information?

9.2 Understanding modality

9.2.1 Comparing modalities

9.2.2 Importance of modality in designing virtual assistant flows

9.2.3 Examples of how modality affects user experience

9.2.4 Voice bot design considerations

9.3 Enhancing context awareness and improving the overall user experience with RAG

9.3.1 Designing adaptive flows with RAG

9.3.2 Strategies for retrieving and generating contextually relevant responses

9.3.3 Maintaining and updating adaptive flows

Summary

10 Reducing complexity with generative AI

10.1 AI-assisted process flows at build time

10.1.1 Generating dialogue flows with generative AI

10.1.2 Improving dialogue flow with generative AI

10.2 AI-assisted process flows at run time

10.2.1 Executing dialogue flows with generative AI

10.2.2 Using LLM for a search process

10.3 AI-assisted flows at test time

10.3.1 Setting up generative AI to be the user

10.3.2 Setting up the conversational test

Summary

Part 4 Pattern: Reduce friction

11 Reducing opt-outs

11.1 What drives opt-out behavior?

11.1.1 Immediate opt-out drivers

11.1.2 Motivations for later opt-outs

11.1.3 Gathering data on opt-out behavior

11.2 Reducing immediate opt-outs

11.2.1 Start with a great experience: Greetings and introductions

11.2.2 Convey capabilities and set expectations

11.2.3 Incentivize self-service

11.2.4 Allow the user to opt in

11.3 Reducing other opt-outs

11.3.1 Try hard to understand

11.3.2 Try hard to be understood

11.3.3 Be flexible and accommodating

11.3.4 Convey progress

11.3.5 Anticipate additional user needs

11.3.6 Don’t be rude

11.4 Opt-out retention

11.4.1 Start right away by collecting opt-out data

11.4.2 Implementing an opt-out retention flow

11.5 Improving dialogue with generative AI

11.5.1 Improving error messages with generative AI

11.5.2 Improving greeting messages with generative AI

11.6 Sometimes it’s okay to escalate

Summary

12 Conversational summarization for smooth handoff

12.1 Intro to summarization

12.1.1 Why summarization is needed

12.1.2 Elements of effective summaries

12.2 Preparing your chatbot for summarization

12.2.1 Using out-of-the-box elements

12.2.2 Instrumenting your chatbot for transcripts

12.2.3 Instrumenting your chatbot (for data points)

12.3 Improving summaries with generative AI

12.3.1 Generating a text summary of a transcript with summarizing prompts

12.3.2 Generating a structured summary of a transcript with extractive prompts

Summary

Overview

12 Conversational summarization for smooth handoff

This chapter explains why concise, targeted conversation summaries are essential for smooth handoffs from conversational AI to human agents. Blind transfers or dumping full transcripts slow agents down and frustrate users who have to repeat themselves. A good summary gives the agent just enough context to continue seamlessly: a compact free-text overview paired with a few high-value data points, enabling faster resolution, less repetition, and a better user experience.

The chapter defines two complementary components of an effective summary: structured metadata and a brief prose recap. Metadata should capture only the essentials for the use case (for example, user identifiers, verification status, and the latest objects of interest like member, provider, or claim IDs), ideally presented as a screen pop in agent tools to avoid re-asking questions. The prose summary should be 1–2 sentences, non-redundant with the metadata, and focused on what has happened and what remains. To enable this, teams should instrument their bots to record transcripts and key fields via platform features, session variables, and webhooks; decide transcript formats; add optional context like timestamps and transfer reasons; and handle sensitive data minimally and compliantly.

The chapter then shows how to improve summaries with generative AI, even without modifying the bot. With a transcript and a clear target format, LLMs can generate compact text summaries and extract structured details (for example, JSON fields) using well-crafted prompts. Instruction tuning, one-shot/few-shot examples, and schema guidance help steer outputs, reduce speculation, and enforce exact key names and data types; deterministic decoding (e.g., greedy) improves repeatability. Teams should validate extracted values against transcripts to catch hallucinations and weigh control versus cost when adding examples. The result is a reliable, scalable method to deliver combined text-and-structure summaries that let agents pick up instantly where the bot left off.

An effective summary helps an agent get up to speed quickly, even if the user previously had a lengthy conversation with a bot.

An effective summary pulls out key details from the conversation. Here the summary includes a summary of the conversation and the last claim searched. The AI portion of the call may have taken two minutes but the human agent can read the summary in seconds.

Not every closed-form question needs to be stored in the summary. In this medical insurance claim review, the most important information is the provider ID, member ID, and claim ID.

When a summary is integrated with contact center software the human agent can have a wealth of information at their fingertips. When an insurance agent clicks on the member ID in their software, they could get additional details on that member.

Accessing the conversation transcript in through the "Session History" variable.

You can use your chatbot’s orchestration layer to create a conversational transcript in whatever format you need.

Storing contextually important information into a context variable so that it can be retrieved later by a summary.

Using a low-code expression editor to combine multiple data elements into a summary.

Summary

Transfers to human agents are an inevitable part of many conversational AI solutions. Agents benefit from receiving brief summarizations that extract key highlights from the conversation, both in prose and in structured format.
A summary requires a conversational transcript. Most conversational AI platforms generate a transcript for you, but you can configure your conversational AI to generate one in your desired format.
Structured summaries can be generated by enhancing your conversational AI to store key data points as they are collected or can be extracted using LLMs when the conversation completes.
You need a clear picture of a good summary before you ask an LLM to generate one.
LLMs can generate prose summaries and extract key details from transcripts. Use clear instructions and examples to generate the summary you desire.

FAQ

What problem do conversation summaries solve in AI-to-human handoffs?

They let agents grasp the user’s goal and progress in seconds instead of reading a long transcript. This reduces user wait time, avoids re-asking questions, and helps the agent pick up exactly where the bot left off.

What are the essential elements of an effective conversation summary?

An effective summary combines: - Structured metadata: key IDs and context needed for lookup and continuity. - A brief free-text synopsis (1–2 sentences) that captures what happened and why the user is escalating. It should be minimal yet complete for the agent to proceed without rereading the transcript.

Which structured metadata should I include (and what can I omit)?

Include only the high-value, actionable fields for your use case. Examples: - User/caller identifier (login ID or phone number) - Collected or found IDs (e.g., member, provider, claim) - Session count or sentiment (optional) Omit intermediate validation details (e.g., DOB or name confirmations) if a single verified ID suffices. In the claims example, provider ID, member ID, and claim ID were enough.

How should I write the free-text portion of the summary?

Keep it simple, 1–2 sentences. Avoid repeating structured fields or step-by-step questions. Capture: - The core task completed or attempted - The immediate context before escalation - Any likely reason for transfer (without speculation) Example: “User searched a claim and saw it was paid; requested a representative for more details.”

How can I capture the conversation transcript on my platform?

Use your platform’s built-in transcript/session variable (e.g., “Session History”) if available. If not, construct your own transcript via orchestration (webhooks) by appending user and bot messages to a context variable. Optionally include timestamps, button clicks, and normalized inputs.

How do webhooks help build custom transcripts and summaries?

Pre/post webhooks can: - Append each turn (User/Bot) to a transcript in your preferred format (string, array, JSON) - Initialize and enrich context (session start time, channel, transfer reason) - Assemble a structured summary at transfer time by reading stored context variables

How do I instrument my bot to capture domain-specific identifiers safely?

Store meaningful, fixed-format responses (e.g., claim, member, provider IDs) in context/session variables. Be minimalist with sensitive data (PII/PHI): collect only what’s needed, store it securely for the minimum time, and review logging/summarization with legal compliance.

How can I use LLMs to generate concise prose summaries?

Provide the transcript (JSON or human-readable) and clear instructions to produce 1–2 sentence summaries emphasizing the last user task before escalation. Use instructible models and greedy decoding for repeatability; add one-shot/few-shot examples to shape tone and reduce speculation.

How can I extract a structured JSON summary with an LLM without changing my bot?

Prompt the LLM with: - A schema (exact keys and types) and strict instructions (e.g., numeric values, allowed keys) - One-shot/few-shot examples illustrating minimal, valid JSON Verify outputs by checking that each extracted value appears in the transcript; adjust prompts to fix key names, types, and duplication.

How should I present the summary to agents for maximum impact?

Use a screen pop that highlights structured fields (e.g., MemberID, ClaimID) and links them to back-end systems (click-through to member profile or claim). Pair with the brief text summary. This avoids re-asking questions, speeds resolution, and improves user experience.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$64.99 $38.99

you save $26.00 (40%)

include audio $24.99 $17.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$64.99 $38.99

you save $26.00 (40%)

include audio $24.99 $17.49

eBook

pdf, ePub, online

$64.99 $38.99

you save $26.00 (40%)

include audio $24.99 $17.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more