Machine Learning Platform Engineering you own this product

Build an internal developer platform for ML and AI systems

Benjamin Tan Wei Hao, Shanoop Padmanabhan, and Varun Mallya

February 2026
ISBN 9781633437333
504 pages

Included with a Manning Online subscription

printed in black & white

available in Korean, Russian

catalog / Data Science / Machine Learning

resources: Source code Datasets Book forum Source code on GitHub Register your pBook for a free eBook

table of content

Part 1 Laying the MLOps foundation

1 Getting started with MLOps and ML engineering

1.1 The ML life cycle

1.1.1 Experimentation phase

1.1.2 Development/staging/production phase

1.2 Skills needed for MLOps

1.2.1 Required skills for ML engineers

1.2.2 Prerequisites

1.3 Building an ML platform

1.3.1 Build vs. buy

1.3.2 Looking ahead: From MLOps to LLMOps

1.3.3 Tools used in this book

1.4 Building ML systems

1.4.1 Introducing the ML projects

1.4.2 ML projects

2 What is MLOps?

2.1 The iterative MLOps life cycle

2.1.1 Data collection

2.1.2 Exploratory Data Analysis

2.1.3 Modeling and training

2.1.4 Model evaluation

2.1.5 Deployment

2.1.6 Monitoring

2.1.7 Maintenance, updates, and review

2.2 Why is robust MLOps important ?

2.3 Role of MLOps in a mature organization

2.4 DevOps vs. MLOps

2.5 Levels of MLOps maturity

2.5.1 Level 0: Basic

2.5.2 Level 1: Intermediate

2.5.3 Level 2: Advanced

3 Building applications on Kubernetes

3.1 Containers and tooling

3.2 Docker

3.2.1 Write the application code

3.2.2 Write a Dockerfile

3.2.3 Building and pushing a Docker image

3.3 Kubernetes

3.3.1 Kubernetes architecture overview

3.3.2 Kubectl

3.3.3 Kubernetes objects

3.3.4 Networking and services

3.3.5 Other objects

3.3.6 Helm charts

3.3.7 Conclusion

3.4 Continuous integration and deployment

3.4.1 GitLab CI

3.4.2 Argo CD

3.5 Prometheus and Grafana

Part 2 Building core ML platform capabilities

4 Designing reliable ML systems

4.1 MLflow for experiment tracking

4.1.1 Data exploration

4.1.2 MLflow tracking

4.1.3 MLflow model registry

4.2 Feast as a feature store

4.2.1 Registering features

4.2.2 Retrieving features

4.2.3 Feature server

4.2.4 Using the Feast UI

5 Orchestrating ML pipelines

5.1 Kubeflow Pipelines: Task orchestrator

5.1.1 Kubeflow components

5.1.2 Income classifier pipeline

6 Productionizing ML models

6.1 BentoML as a deployment platform

6.1.1 Building a Bento

6.1.2 Building and pushing the Bento

6.1.3 Deploying a Bento

6.2 Evidently for data drift monitoring

6.2.1 Data drift detection report and dashboard

6.2.2 Data drift detection Kubeflow pipeline component

6.2.3 Data drift detection for a model deployed as an API

Part 3 Applying MLOps in practice

7 Data analysis and preparation

7.1 Data analysis

7.1.1 Launching a notebook server in Kubeflow

7.1.2 Workspace and data volumes

7.1.3 Configurations and affinity/tolerations

7.1.4 Customizing the menu

7.1.5 Creating a custom Kubeflow notebook image

7.2 Data passing

7.2.1 Scenario 1: Passing simple values to downstream components

7.2.2 Scenario 2: Passing paths for larger data

7.2.3 Overview of KFP v2 artifact types

7.3 Data preparation in action

7.3.1 Data preparation: Object detection

7.3.2 Data preparation: Movie recommender

8 Model training and validation: Part 1

8.1 Training an object detection model

8.1.1 Training YOLO on a custom dataset

8.1.2 Training the model

8.1.3 Container components for system dependencies

8.1.4 Creating the validation component

8.1.5 Creating the pipeline

8.1.6 Executing the pipeline

8.1.7 Validating model artifacts

9 Model training and validation: Part 2

9.1 Storing data with PersistentVolumeClaim

9.1.1 Refactoring the pipeline with a PVC

9.1.2 Efficient dataset management

9.1.3 Creating a VolumeOp

9.1.4 Download Op using PVC

9.1.5 Splitting the dataset directly

9.1.6 Simplifying model training

9.1.7 Simplifying model validation

9.2 Tracking training with TensorBoard

9.2.1 Launching a new TensorBoard

9.2.2 Exploring YOLOv8’s default graphs

9.3 Movie recommender project

9.3.1 Reading data from MinIO and quality assurance

9.3.2 Model training component

9.3.3 Metrics for evaluation

9.3.4 Experiment tracking with MLflow

9.3.5 Model registry with MLflow

9.3.6 Creating a pipeline from components

9.3.7 Local inference in a notebook

10 Model inference and serving

10.1 Model deployment is hard

10.2 BentoML: Simplifying model deployment

10.3 A whirlwind tour of BentoML

10.3.1 BentoML Service and Runners

10.4 Executing a BentoML Service locally

10.4.1 Loading a model with BentoML Runner

10.5 Building Bentos: Packaging your service for deployment

10.5.1 Bento tags: Versioning and managing your Bentos

10.6 BentoML and MLflow inference

10.7 Using only MLflow to create an inference service

10.8 KServe: An alternative to BentoML

11 Monitoring and explainability

11.1 Monitoring

11.1.1 Basic monitoring

11.1.2 Custom metrics

11.1.3 Logging

11.1.4 Alerting

11.2 Data drift detection

11.2.1 Object detection

11.2.2 Movie recommender

11.3 Explainability

11.3.1 Object detection

11.3.2 Movie recommendation

Part 4 Extending MLOps for large language models

12 Designing LLM-powered systems

12.1 LLMOps: New challenges, familiar principles

12.1.1 What makes LLM applications different

12.1.2 Extending our ML platform for LLMs

12.1.3 Essential tools for LLM applications

12.2 Building DataKrypt’s DakkaBot: A simple RAG architecture

12.2.1 What you’ll build

12.2.2 Beyond single API calls: Designing for composability

12.2.3 Google’s Gemini LLM and embeddings

12.2.4 The retrieval component

12.2.5 The augmentation component

12.2.6 The generation component

12.3 Giving DakkaBot a UI

12.4 Observability for LLM applications

12.4.1 Set up Langfuse via Docker

12.4.2 Integrating Langfuse with DakkaBot

12.4.3 Enhanced observability in DakkaBotCore

12.4.4 Beyond traditional metrics

13 Production LLM system design

13.1 Prompt engineering: Code for the generative AI era

13.1.1 Treating prompts as critical infrastructure

13.1.2 Langfuse prompt management for DakkaBot

13.1.3 Langfuse prompt management for production

13.2 Testing LLM applications

13.2.1 Evaluation framework for LLM responses

13.2.2 Safety and adversarial testing

13.3 Governance and safety in production

13.3.1 Implementing safety guardrails

13.4 Cost optimization strategies

13.4.1 Understanding LLM economics

13.4.2 Model selection strategy

13.4.3 Caching strategies

13.4.4 Prompt optimization for efficiency

13.4.5 Production cost monitoring

13.4.6 From traditional ML to LLMOps

Appendices

Appendix A: Installation and setup

A.1 Local installation of command-line tools (Mac and Linux)

A.1.1 The yq YAML processor

A.1.2 Kustomize

A.1.3 Kubectl

A.1.4 K8s distribution

A.1.5 K3s installation

A.1.6 MicroK8s installation

A.1.7 Argo CD

A.1.8 Kubeflow

A.1.9 Cloud provider K8s setup

A.1.10 MLflow setup

A.2 Deploy MLflow

A.2.1 Redis online store setup

A.2.2 BentoML and Yatai setup

A.2.3 Evidently UI setup

Appendix B: Basics of YAML

B.1 Basic YAML files

B.1.1 Comments

B.1.2 Scalar values

B.1.3 Lists

B.1.4 Nested structures (maps)

B.1.5 Quoted strings

B.1.6 Multiline strings

B.1.7 Data types in YAML

B.2 Aliases and anchors

B.2.1 References (merging and reusing data)

B.2.2 Complex data types

B.2.3 Custom data types

B.2.4 Block style vs. flow style

B.2.5 Key sorting and case sensitivity

B.2.6 Best practices

Overview

Appendix A. Installation and setup

This appendix provides a practical, command-focused path to standing up a full MLOps stack on Kubernetes, beginning locally and scaling to the cloud. It starts by installing essential CLI tools (yq for YAML processing, kustomize for configuration management, and kubectl for cluster operations), then guides you to choose a lightweight Kubernetes distribution: k3s for Linux or microK8s for macOS (via Multipass). Resource baselines are called out (at least 4 CPU, 12 GB RAM, and 50–100 GB disk), along with setup steps to configure kubeconfig, fix common permissions issues, enable DNS (for microK8s), and verify the cluster. On top of this base, Argo CD is introduced to enable GitOps-driven deployments; after installing and logging in with the initial admin secret, Kubeflow is deployed and accessed through the Istio ingress gateway, with notes on first-time readiness delays and default credentials.

When local resources become a bottleneck, the chapter recommends moving to a managed Kubernetes service (such as those from major cloud providers) and stresses cost monitoring with billing alerts. It then shows how to add MLflow as a tracking server and model registry using a Postgres backend (installed via Helm in a dedicated namespace) and the MinIO artifact store already present from Kubeflow. You build and push a minimal MLflow image, apply Kubernetes manifests for a Deployment and Service in an mlflow namespace, and use port forwarding to reach the UI—tying together artifact storage, metadata persistence, and cluster-native operations.

To round out the platform, the appendix covers installing Redis via Helm for use as Feast’s online store and setting up BentoML’s Yatai with Helm in its own namespace, including generating an initialization link to create the admin account for managing deployments. Finally, it packages an Evidently UI image and deploys it with straightforward Kubernetes manifests (Namespace, Deployment, Service) to visualize data drift reports within the cluster. By the end, you have a cohesive, reproducible installation recipe that layers GitOps, Kubeflow, MLflow, Redis, BentoML/Yatai, and Evidently on Kubernetes—ready for experimentation on a laptop and promotion to managed clusters as workloads grow.

FAQ

What command-line tools should I install before setting up Kubernetes and Kubeflow (Mac/Linux)?

Install these first: - yq (YAML processor) - Linux:

export VERSION=v4.2.0
export BINARY=yq_linux_amd64
wget "https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}" -O /usr/bin/yq && chmod +x /usr/bin/yq

- Mac:

brew install yq

- kustomize (K8s config manager) - Linux:

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
mv kustomize /usr/local/bin

- Mac:

brew install kustomize

- kubectl (K8s CLI) - Linux:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mv kubectl /usr/local/bin

- Helm (K8s package manager) — install from the official docs: https://helm.sh/docs/intro/install/

What are the recommended local machine resources for running a single-node Kubernetes cluster?

Minimum recommended: - CPU: 4 cores - Memory: 12 GB RAM - Disk: 50 GB free (100 GB ideal) These specs help ensure a smoother experience when running tools like Kubeflow, MLflow, and supporting services locally.

Which local Kubernetes distribution should I use: k3s or microK8s?

Both are lightweight and work well: - Use k3s for Linux-based installations (simple, minimal moving parts). - Use microK8s for macOS (via a Multipass-backed Ubuntu VM) or Linux. If you’re new to K8s, start locally (k3s or microK8s), then move to cloud once workloads grow.

How do I install k3s on Linux and configure kubectl? (Plus a fix for “Config not found”)

Install and configure: - Install k3s:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.26.3+k3s1" INSTALL_K3S_EXEC="server --no-deploy=traefik" K3S_KUBECONFIG_MODE="644" sh -s

- Set kubeconfig:

mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
export KUBECONFIG="$HOME/.kube/config"

- Ensure permissions (e.g., once via shell init or manually):

sudo chmod 644 /etc/rancher/k3s/k3s.yaml

- Reload shell if needed:

source ~/.bashrc

- Verify:

kubectl get nodes

Troubleshooting: - If you see “Config not found: ~/.kube/config”, ensure ~/.kube/config has permission 644 and try using the absolute path.

How do I install microK8s on macOS and connect it to my kubectl?

Steps on macOS: - Install microK8s (uses Multipass VM):

brew install ubuntu/microk8s/microk8s

- Create the VM with recommended resources:

microk8s install --cpu 4 --mem 12 --disk 50 --channel 1.26

- Wait for readiness:

microk8s status --wait-ready

- Enable DNS:

microk8s enable dns

- Set kubeconfig for kubectl:

cd "$HOME"
mkdir -p .kube
cd .kube
microk8s config > config
echo 'export KUBECONFIG="$HOME/.kube/config"' >> ~/.bashrc   # or ~/.zshrc
# reload your shell and verify:
kubectl get pods

Troubleshooting: - If you see “Config not found: ~/.kube/config”, ensure ~/.kube/config has permission 644.

How do I install and access Argo CD locally?

Install and log in: - Clone repo and deploy:

git clone https://github.com/vmallya-123/kubeflow-argo
cd kubeflow-argo
kubectl apply -k argocd/

- Watch for pods to be Running:

kubectl get po -n argocd -w

- Get the admin password:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

- Port-forward the UI:

kubectl port-forward svc/argocd-server -n argocd 8080:443

- Open http://localhost:8080 and log in with: - Username: admin - Password: value from the secret above

How do I deploy Kubeflow with Argo CD and reach the UI?

From the cloned repo: - Apply the Kubeflow app:

kubectl apply -f kubeflow.yaml

- Wait (can take ~30 minutes) for all Argo CD applications to turn healthy/green. - Get Istio ingress service:

kubectl get svc istio-ingressgateway -n istio-system

- Access via EXTERNAL-IP (preferred) or CLUSTER-IP using HTTPS (e.g., https://<EXTERNAL-IP>). - Default Kubeflow credentials: - Username: user@kubeflow.org - Password: 12341234

Can I use a managed (cloud) Kubernetes service instead of local, and how do I control costs?

Yes—recommended when local resources are insufficient. Options and guides: - GKE (Google Kubernetes Engine): https://cloud.google.com/kubernetes-engine/docs/how-to/creating-an-autopilot-cluster - New GCP users may get ~$300 in credits: https://cloud.google.com/free/docs/free-cloud-features#free-trial - AWS EKS: https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html - DigitalOcean Kubernetes: https://docs.digitalocean.com/products/kubernetes/getting-started/ Cost control: - Set up billing alerts/budgets in your cloud account. - Tear down unused clusters and workloads promptly.

How do I deploy MLflow on Kubernetes with a Postgres backend and MinIO artifact store?

High-level steps: - Install Postgres via Helm in a postgres namespace:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
kubectl create namespace postgres
helm install postgres-release bitnami/postgresql --namespace postgres
export POSTGRES_PASSWORD=$(kubectl get secret --namespace postgres postgres-release-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

- Build/push the MLflow image (example tag):

docker build . -t varunmallya/mlflow:v1
docker push varunmallya/mlflow:v1

- Create mlflow namespace and ensure a MinIO bucket named mlflow-artifacts exists (from the Kubeflow install). - Use a Deployment that: - Sets env: AWS_ACCESS_KEY_ID=minio, AWS_SECRET_ACCESS_KEY=minio123, AWS_ENDPOINT_URL=http://minio-service.kubeflow.svc.cluster.local:9000 - Runs MLflow server with: --default-artifact-root s3://mlflow-artifacts and

--backend-store-uri postgresql+psycopg2://postgres:<POSTGRES_PASSWORD>@postgres-release-postgresql.postgres.svc.cluster.local:5432/postgres

- Expose via a Service on port 5000 and port-forward locally:

kubectl port-forward svc/mlflow-service -n mlflow 5000:5000

- Open http://localhost:5000

How do I build and deploy the Evidently UI on Kubernetes?

Steps: - Build and push an image that runs evidently ui (example):

# Dockerfile uses python:3.9-slim-buster and ends with:
ENTRYPOINT ["evidently","ui"]

docker build . -t varunmallya/evidently-ui:latest
docker push varunmallya/evidently-ui:latest

- Create K8s resources: - Namespace evidently - Deployment using varunmallya/evidently-ui:latest exposing container port 8000 - ClusterIP Service on port 8000 - (Optional) Access locally via port-forward:

kubectl port-forward svc/evidently-ui -n evidently 8000:8000

- Open http://localhost:8000 to view the UI and drift reports.

How do I install BentoML’s Yatai on Kubernetes and create the admin account?

Install with Helm in yatai-system: - Add repo and install:

helm repo add bentoml https://bentoml.github.io/helm-charts
helm repo update bentoml
kubectl create ns yatai-system
helm install yatai-test bentoml/yatai --set ingress.enabled=false --set service.type=LoadBalancer -n yatai-system --create-namespace

- Get the initialization link to create the admin account:

export YATAI_INITIALIZATION_TOKEN=$(kubectl get secret yatai-env --namespace yatai-system -o jsonpath="{.data.YATAI_INITIALIZATION_TOKEN}" | base64 --decode)
export SERVICE_IP=$(kubectl get svc --namespace yatai-system yatai --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
echo "Create admin account at: http://$SERVICE_IP:80/setup?token=$YATAI_INITIALIZATION_TOKEN"

Open the printed URL and set the username, password, and email to complete setup.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more