Overview
5 Hosting, scaling, and load testing
This chapter takes a RAG chatbot from “works on my machine” to a resilient, production-ready service. It promotes statelessness as the core deployment principle—containers as the immutable unit, configuration via environment variables, and durable state kept outside the runtime—so any replica can handle any request. The story ties packaging, observability, and automation together into a disciplined loop: build a container, deploy it reliably, instrument it with logs and metrics, validate it under pressure, and scale or optimize based on data.
Concretely, the prototype’s local SQLite is replaced with Cosmos DB for durable, cheap, globally accessible chat logs, and the code is refactored to upsert structured records and to return a final answer in test mode so integration checks pass. Dependencies are pinned, a lean Docker image (Python 3.11 slim plus required system libs) is built, secrets are kept out of the image, and a quick WebSocket smoke test proves parity with local runs. The deployment path then standardizes on Azure: push images to a private registry, host on Azure Web App (with WebSockets, a sensible plan tier, and Application Insights), inject configuration as environment variables, and wire GitHub Actions to build, test, push, and roll out on every commit—turning “git push” into a safe, repeatable release.
To ensure the service survives real traffic, the chapter adds structured logging and introduces Locust to simulate concurrent WebSocket users, capturing latency, error rates, and throughput while streaming logs and Application Insights highlight bottlenecks. Common pressure points and remedies are mapped: alleviate model 429s by increasing model throughput (or moving to Azure OpenAI and trimming token usage), resolve slow responses by scaling the App Service up or out, and remove search throttling by stepping up Azure AI Search capacity. The result is a pragmatic feedback cycle—deploy, observe, load-test, adjust, repeat—that converts scaling from guesswork into a measured, cost-aware practice.
This flowchart shows the key stages of this chapter. We will start by packaging our app as a container, then deploy it to Azure’s app service, then test it remotely. Finally, we will perform load testing using logging and simulated users with Locust.
The RAG container keeps no secrets and writes nothing to its own disk; it simply streams prompts from the browser, fans out to AI Search and OpenAI for retrieval and generation, and drops a JSON record into Cosmos DB for posterity. All configuration arrives at launch via environment variables or a key vault. The dashed line to “Local Disk” drives home the point: local storage is strictly off-limits in a truly stateless service.
An overview of the new logging path: each answer that streams to the user is also packaged into a JSON document and sent over HTTPS to Cosmos DB, where it can be queried later to debug slow responses, audit model behaviour, or train future ranking tweaks..
In this section, we will get our web app’s URL, add it to our test website, open it, and send a query, then wait for the streaming answer to come back.
We will add logging to our file to catch any errors, then run Locust to simulate users and cause our app to fail. We will use the Locust dashboard and logs to find bottlenecks so we can fix them.
A Locust swarm batters the public Web App endpoint while Application Insights logs every request and latency spike. Those metrics show up in real time, giving you the data needed to nudge the App Service scale slider from one replica to many.
If we experience any failures during out Locust tests, we first look at our logs in our web app. Next, once we have identified the error, we fix it by scaling out that particular part of our system. For example, if we are getting rate limits from OpenAI, we increase the amount of requests we can send.
Summary
- Rolling out a RAG app is easier when you pair an Azure Web App with GitHub Actions, because every code push magically turns into a live app.
- By moving chat logs into Cosmos DB and pinning every package in requirements.txt, we turned our scripts into cloud-worthy code that can reboot safely and retain chat logs.
- Setting up a CI/CD pipeline means tests run automatically on each commit, so broken code gets caught in the pipeline instead of embarrassing you in front of real users.
- Adding friendly logging.info() calls throughout the code gives us play-by-play insight, so when something misbehaves under load we can trace the exact moment and function where it stumbles.
- Locust lets us unleash a pretend stampede of users, revealing performance potholes in a safe sandbox, so we can prepare for the real users.
- When bottlenecks appear, we learned to crank up AI Search tiers, boost our OpenAI token quotas, or spin up extra Web App instances—scaling only what’s necessary to keep response times snappy without lighting our budget on fire.
Enterprise RAG ebook for free
