6 Communication strategies: Disclaimers, feedback, and prompt tuning

This chapter emphasizes that successful Enterprise RAG systems are as much about communication as they are about code. It starts by establishing the role of a clear, always-visible disclaimer and a concise user guide to align expectations: define the domain, enumerate accessible data sources and known limitations, and show users how to phrase effective queries (for example, using full sentences). By setting scope early and consistently, teams reduce confusion, preempt unanswerable requests, and create a shared baseline that turns each interaction into actionable feedback rather than frustration. Courtesy, transparency, and predictable messaging become the foundation for trust as the system evolves.

The chapter then operationalizes feedback as a product pipeline. A lightweight UI captures simple thumbs-up/down ratings with optional comments and an email for follow-up, while a secure, serverless function relays this data to a database for analysis—keeping credentials server-side and decoupling quick, bursty writes from long-lived chat streaming. It walks through local setup, environment configuration, deployment, and CORS considerations, all in service of a robust, low-friction path from user input to structured signal. The goal is steady telemetry that teams can review regularly, spot patterns, and prioritize fixes without exposing secrets or entangling components.

Finally, the chapter shows how to close the loop and turn feedback into better answers. Each negative rating is traced to either retrieval (wrong or missing documents) or generation (right documents, wrong answer). Retrieval issues are addressed by verifying embeddings and index settings, tightening search parameters, and giving agents explicit field lists and filter syntax (plus a simple preprocessor to pass filters cleanly). Generation issues lead to prompt tuning for the writer agent—adding concrete rules, few-shot examples, citation requirements, brevity and language preferences, and business-specific instructions (such as prioritizing certain products), sometimes alongside small code changes for ordering. After testing and redeploying, the team contacts the reporting user to confirm the fix. Over time, this courteous, evidence-driven cycle—anchored by clear disclaimers, structured feedback capture, and targeted prompt updates—steadily reduces errors and builds durable user trust.

This mental model shows where we are in the overall process of building our RAG system, see the section highlighted by the dashed circle above.

The whole cycle for incorporating feedback into improving your RAG. The user searches for a product, gives feedback and then we search the logs and diagnose the problem. We update the code to fix the problem, redeploy, check to make sure the answer is improved, and then email the user to thank them and inform them the problem is solved.

After the user receives the answer from the RAG chatbot, the user gives feedback, which is sent to a function app, and then to a CosmosDB database for safe keeping and later analysis.

The relationship between your web browser and CosmosDB. We have a function app in the middle for database security.

A promotional poster for the band Thumb Direction, on tour now.

We will complain about a query, send feedback, check our logs, diagnose the problem, fix it, redeploy, observe the improved answer, and email the user to tell them that the problem is fixed.

This is the workflow to follow when your RAG chatbot is returning the wrong data. Check the chat history logs. If you see bad search results, modify your search parameters. If you see no search results, check to make sure your API keys are set for your search service functions.

Check the prompt and see if it is detailed enough. You can modify the prompt and add the user’s complaint to the prompt as an example. If the instructions you need are already there, the model may be hallucinating because the prompt is too long and complex.

• Clear, early communication is non-negotiable. If users don’t know what your chatbot can and can’t do, even perfect retrieval logic will feel broken.
• A short, up-front disclaimer and a companion user guide set realistic expectations and head off “mind-reading” questions before they happen.
• Simple thumbs-up / thumbs-down buttons (plus a comment box) transform every chat into actionable data; drop those payloads into Cosmos DB so nothing slips through the cracks.
• Route feedback through a small serverless Function (or Lambda Function) to keep database keys off the browser and scale writes without throttling your chat service.
• Treat each complaint like a bug report: diagnose whether the miss came from retrieval or prompt wording, fix it in source control, and let CI tests prove the improvement.
• Close the loop: redeploy, verify the answer, and thank the user. Nothing builds trust faster than seeing their gripe turn into a product upgrade.

FAQ