10 Model inference & serving
This chapter moves from training and validation to making models available as reliable services. It explains why ML deployment adds complexity beyond traditional software—covering inference patterns (batch vs. real-time), scaling, performance monitoring, data dependencies, hardware needs, versioning, explainability, and security—and argues for self-service deployment to accelerate iteration and collaboration without replacing production MLE roles. BentoML is presented as the main enabler: a unified, framework-agnostic way to package models and serving logic with runners and high-level APIs, supporting real-time and batch inference, adaptive micro-batching, reproducible builds, model versioning, observability, and seamless scaling on containers and Kubernetes.
The chapter then walks through a hands-on BentoML service: building a YOLOv8 runner and exposing two endpoints—/inference for JSON results and /render for annotated images—illustrating async execution, input/output validation, and why awaiting runner calls matters. It shows how multiple API servers and runners improve throughput and resilience, and how multi-runner graphs enable pre-processing chains or A/B testing across model versions. Crucially, BentoML bakes in production-minded observability: liveness/readiness health checks and Prometheus metrics by default, making local development feel close to production while easing integration with Kubernetes probes and monitoring stacks.
Finally, the chapter packages services as Bentos using a bentofile.yaml, builds immutable, versioned artifacts (Bento Tags), containerizes them for consistent deployment, and validates via Docker. It extends the pattern by pulling “prod” models from MLflow’s registry into a BentoML service (for a recommender), combining MLflow’s tracking/versioning with BentoML’s serving ergonomics. It also outlines alternatives: serving directly from MLflow for simpler, quick experiments with built-in /health and /invocations, and KServe for Kubernetes-native, highly scalable orchestration at the cost of more cluster expertise. The guidance: pick BentoML for developer-friendly, portable serving; MLflow-only for minimal tooling; and KServe when deep Kubernetes integration and control are paramount.
The mental map where we are now focusing on model deployment(6) and making the model available as an API(E)
BentoML Service architecture showing API Server distributing requests to multiple Runners.
BentoML Service architecture with multiple API Servers and Runners.
Sample output of YoloV8 object detection on a driver's license image.
Interface showing successful image inference via the /render endpoint.
Sample output of YoloV8 object detection on a driver's license image.
BentoML's built-in observability endpoints for Kubernetes environments.
Sample Prometheus metrics output from a BentoML service.
BentoML build success message and suggested next steps
BentoML web interface showing the deployed YOLOv8 service endpoints and documentation.
Logical flow of using MLFlow and BentoML together
Logical flow of using only MLFlow for the entire model lifecycle
Summary
- Model deployment introduces unique challenges, and understanding inference patterns, scalability, monitoring, and resource management is crucial.
- BentoML simplifies model deployment by providing a unified framework for packaging and serving models, allowing you to focus on the core functionality of your service.
- Building a BentoML service involves defining a Runner, creating a Service with API endpoints, and leveraging BentoML's intuitive APIs and decorators for input/output handling and async execution.
- BentoML provides built-in support for observability, including health checks, liveness/readiness probes, and Prometheus metrics, ensuring proper monitoring and management within a Kubernetes environment.
- Packaging your BentoML service into a Bento enables easy distribution and deployment across different environments, while Bento Tags provide a robust versioning and management system.
- Using MLFlow and BentoML provides a lot of advantages. Mlflow provides the lineage and tracking for models and bentoML seamlessly enables serving models in a registry.
- We can also use MLFlow by itself for the entire lifecycle. This might be preferable for smaller organizations where simplicity is preferred over feature sets.
- When selecting a model serving tool, consider alternatives like KServe and others with unique strengths. Choose based on ease of use, Kubernetes compatibility, and desired control level. BentoML shines for its simplicity and developer-friendly APIs.
FAQ
Why is deploying ML models harder than deploying traditional software?
Beyond standard CI/CD and testing, ML serving adds unique challenges:
- Inference patterns: Must support batch, real-time, or hybrid traffic.
- Scalability: Highly variable workloads and concurrency needs.
- Monitoring: Track latency, accuracy, drift, and detailed logs.
- Continuous learning: Retraining and safe rollout/rollback of new models.
- Data dependencies: Pre/post-processing parity with training pipelines.
- Hardware: Optional accelerators (e.g., GPUs) and scheduling.
- Versioning: Multiple coexisting models and rollback strategy.
- Explainability: Needed for trust and regulated use cases.
- Resource management: Balance CPU/GPU/memory across components.
- Security/privacy: Prevent leakage and resist adversarial inputs.
What are BentoML Services, API Servers, and Runners, and how do they work together?
A BentoML Service wraps one or more HTTP API Servers and one or more Runner types:
- API Server: Validates/parses inputs, exposes endpoints, and forwards calls to Runners. You can run multiple instances for horizontal scaling, load balancing, and higher throughput.
- Runner: Encapsulates model execution. Each Runner runs in its own Python worker process; multiple Runner instances can run in parallel for concurrency and performance.
- Flow: Request → API Server (I/O parsing) → Runner (inference) → Response.
How do I define endpoints and I/O types in a BentoML service?
Use the service decorator and I/O descriptors:
- Define a Service and Runners in code, then add an endpoint with
@svc.api(...). - Specify input and output schemas using descriptors from
bentoml.io, e.g.,Image(),JSON(). - Example:
@svc.api(input=Image(), output=JSON())ensures the server accepts an image and returns JSON.
Why do Runner calls often use async/await, and what happens if I don’t await?
Runner methods support asynchronous execution (e.g., async_run):
- Why async: Avoid blocking the event loop during inference; improve concurrency and responsiveness.
- await is required:
async_runreturns a future. Withoutawait, your code proceeds with an unresolved promise, leading to errors or empty/invalid results. - Benefit: While awaiting an inference, the server can serve other requests.
What’s the difference between the /inference and /render endpoints in the YOLOv8 example?
- /inference: Accepts an image and returns predictions as JSON (classes, confidences, and bounding boxes). Ideal for programmatic use and downstream systems.
- /render: Accepts an image and returns the image with drawn bounding boxes, labels, and probabilities. Great for demos, debugging, and visual verification.
When should I use multiple Runners in one service, and how do I orchestrate them?
Use multiple Runners when your workflow has distinct steps or models:
- Pre/post-processing: One Runner for transforms (e.g., grayscale), another for inference.
- A/B testing or ensembles: Drive the same preprocessed input into multiple model Runners in parallel.
- Orchestration: Chain Runner calls and use
asyncio.gather()to run multiple inferences concurrently, then aggregate results into one response.
What observability features are built into BentoML services?
- Health endpoints: Liveness and readiness probes are available out-of-the-box (e.g.,
/healthz,/readyz) for Kubernetes to restart or route traffic appropriately. - Metrics: Built-in Prometheus metrics endpoint exposes standardized time-series metrics for scraping and analysis.
- Logging: Structured logging to support debugging and performance monitoring.
How do I build and containerize a Bento for deployment?
Typical steps:
- Create
bentofile.yamlwith service entrypoint, included files (e.g., model weights), and optional Docker base image. - Build the Bento:
bentoml build. - Containerize:
bentoml containerize <service:tag>to produce a Docker image. - Run locally:
docker run --rm -p 3000:3000 <image-name>and test endpoints in the web UI or via HTTP calls.
What are Bento Tags and how do they help with versioning and reproducibility?
Bento Tags uniquely identify a built service image as name:version (e.g., yolo_v8:3jghhcfxvwsrnbsb):
- Versioning: Track iterations of code, models, and dependencies.
- Reproducibility: Pin exact builds for debugging and audits.
- Deployment: Be explicit about which version is promoted to each environment, similar to Docker tags.
How can I combine BentoML with MLflow’s Model Registry for serving?
Pattern overview:
- Use MLflow to track, register, and alias models (e.g.,
prod). - At service startup, query MLflow for the current
prodversion, obtain themodel_uri, and import it into BentoML. - Load the model and expose a prediction API via BentoML. Then build and containerize as usual.
- Benefit: Data scientists keep MLflow’s experiment and registry workflow; engineers get BentoML’s serving, scaling, and observability.
Can I serve models using only MLflow, and how does that compare to KServe and BentoML?
Only MLflow:
- Serve directly with
mlflow models serveor build Docker images withmlflow models build-docker. - Health and inference endpoints (e.g.,
/health,/invocations) are provided. - Pros: Very simple; no Kubernetes expertise required. Cons: Less fine-grained serving controls than BentoML or KServe.
BentoML:
- Developer-friendly APIs, Runners, built-in health/readiness and Prometheus metrics, easy containerization.
- Great for fast local-to-prod paths and complex multi-model workflows.
KServe:
- Kubernetes-native serving in Kubeflow; highly scalable and feature-rich for cluster environments.
- Pros: Strong orchestration at scale. Cons: Requires Kubernetes expertise and more configuration.
Choose MLflow for simplicity, BentoML for streamlined serving with strong observability and workflows, and KServe for Kubernetes-first, large-scale environments.
Machine Learning Platform Engineering ebook for free
