9 Model training and validation: Part 2
This chapter moves beyond basic pipelines to show how to productionize model training and validation with stronger data handling, observability, and lineage. It demonstrates how to make pipelines efficient and reproducible by centralizing dataset access with Kubernetes Persistent Volumes, adding fine-grained visibility into training with TensorBoard, and instituting rigorous experiment tracking and model versioning with MLflow. The concepts are applied to two contrasting projects—an object detection workflow and a movie recommender—so readers can see how the same platform patterns generalize across modalities and teams.
The first part refactors the training workflow to use a single, shared PersistentVolumeClaim (PVC) created via the Kubeflow Pipelines Kubernetes SDK. By mounting this PVC across components, the pipeline downloads data once, performs splits in place, and reuses the same mounted paths for training and validation, eliminating redundant transfers and simplifying code. Ordering is made explicit where artifact dependencies are no longer inferred. With data management stabilized, the chapter shows how to surface run-time insights using TensorBoard: directing YOLOv8’s logs to the shared volume and launching a Kubeflow-managed TensorBoard instance to monitor loss and accuracy curves, compare runs, and visualize model architecture for faster debugging and iteration.
The second part builds a recommendation system that reads structured data directly from MinIO (discussing the trade-offs versus local volumes), performs lightweight data QA, and trains a simple PyTorch matrix factorization model. It defines practical evaluation metrics (precision@K, recall@K, RMSE) and then integrates MLflow to log hyperparameters, metrics, artifacts, model signatures, and even enable warm restarts from previous runs. Validation can write results back to the same run for cohesive lineage, and the MLflow Model Registry is used to manage versions, stage promotions via aliases, and automate basic promotion checks. The chapter closes by composing the components into a Kubeflow pipeline and illustrating local inference by loading the production model from the registry—establishing a repeatable, production-ready pattern with clear lineage, faster feedback loops, and reliable handoffs to deployment.
The mental map where we continue focusing on the second and third step/component of our pipeline - model training(4) and evaluation(5)
All the components depend on the PVC called "createpvc".
Full Training Validation pipeline with PVC.
Make sure the VolumeOp has completed before launching Tensorboard.
The name of the pipeline shows up when you click on the component.
Creating a new Tensorboard in Kubeflow.
Creating a new Tensorboard and pointing to an existing pipeline PVC.
A successful Tensorboard instantiation.
A selection of graphs made available by the YOLOv8 library.
Viewing the model architecture from Tensorboard
Example of a model dashboard and metrics on MLFlow
Recommendations from a trained model
Example of movies viewed by user 50
Summary
- A VolumeOp is usually a better way to handle large datasets within a training pipeline and overcomes the drawback of having to download the full dataset everytime a training is run.
- Like models, the extended training artifacts are important to track. These include metrics, hyperparameter sets and validation examples. Well kept lineage helps debug issues in production faster.
- MLFLow and Tensorboard help track experiments and training runs. While they both appear to do the same thing, MLFLow is best used for multiple experiment tracking and artifact storage while Tensorboard offers more introspective capabilities for a single run.
- Having a central tracking, model and experiment tracking system like MLFlow provides organizational benefits in addition to the easier workflow for developers. A single window helps keep the stakeholders, project management and developers in sync with respect to model capabilities and performance.
- Evaluation metrics are key in model development since they judge the model’s performance. These can also evolve over the course of a model's lifetime so evaluation frameworks must support iterative improvement.
- Manual evaluation of a model is important. Although good metrics can predict performance, it is faster to debug model issues manually in the beginning and build upon the metrics to better represent performance. This means that the model registry must be easily accessible and model download simple. Using registries like MLFLow that provide an API to download the model helps here.
FAQ
What problems do Kubernetes PersistentVolumeClaims (PVCs) solve in training pipelines?
PVCs provide a persistent, shared filesystem across components so you can download data once and reuse it, avoid redundant dataset downloads, enable seamless data sharing between steps, and manage storage more efficiently than ad‑hoc per-step downloads.How do I create and mount a PVC in a Kubeflow Pipelines (KFP) workflow?
- Install kfp with Kubernetes extras: pip install "kfp[kubernetes]"- Create the PVC at pipeline start (for example, kubernetes.CreatePVC with access_modes=['ReadWriteOnce'], size='1Gi', and an appropriate storage_class_name).
- Mount it on each component that needs shared data using kubernetes.mount_pvc(op, pvc_name=..., mount_path='/data'). All components then read/write under /data.
When using a shared PVC, how do I ensure components run in the right order?
Without explicit artifact dependencies, KFP cannot infer ordering. Use .after(upstream_op) on downstream tasks (for example, split_op.after(download_op), train_op.after(split_op)) so they execute in the correct sequence while sharing the same PVC.What are the trade-offs between copying data to a PVC vs. reading directly from MinIO?
- Copy to PVC: slower startup and higher storage usage but fast local reads during training and simpler POSIX-style file handling.- Read directly from MinIO: fast startup and lower storage footprint but network-intensive during training; relies on high-bandwidth networking and optimized I/O. The best choice depends on dataset size, infra bandwidth, and workflow patterns.
How does using a PVC simplify data handling for download, split, train, and validate steps?
- Download: write extracted files to /data once; no per-step MinIO code.- Split: move/copy files into /data/train|val|test/images|labels directly.
- Train: remove many path inputs; point data.yaml to /data/train|val|test paths.
- Validate: load weights and use data.yaml from the mounted /data. Overall, fewer arguments, less glue code, and more deterministic paths.
How do I launch TensorBoard in Kubeflow to monitor YOLOv8 training logs on a PVC?
1) Start a pipeline run and wait for the PVC step to succeed; note the pvc_name.2) In Kubeflow UI, go to Tensorboards → New Tensorboard.
3) Choose “PVC” as the source and select the run’s PVC; leave Mount Path empty.
4) Create and then Connect. YOLOv8 writes logs under the configured “project” folder on the PVC, which TensorBoard reads.
What insights does TensorBoard provide for YOLOv8 in this setup?
- Real-time curves: mAP@50-95, mAP@50, mAP@75, losses over epochs.- Visual debugging: detect over/underfitting trends and tune early stopping/hyperparameters.
- Model graph: understand architecture and data flow for complex models like YOLOv8.
- Multi-run comparisons to analyze effects of hyperparameter changes.
How is MLflow integrated into the movie recommender pipeline?
The training code logs hyperparameters, tags, and metrics (e.g., average train/test loss), records artifacts (like model summaries), infers and stores model signatures, and logs models to MLflow. It can also warm-start by loading a prior run’s model (runs:/<run_id>/model). The validation step loads the model from MLflow and logs evaluation metrics back to the same run, enabling clear lineage and reproducibility.How does the MLflow Model Registry support staging and production promotion?
Models logged during experiments can be registered under a named model. Using aliases like “staging” and “prod,” you can: compare new metrics to the current staging model, register a better run’s model, and set the alias to point to the new version. This creates auditable lineage, controlled promotions, and quick rollbacks by re-pointing aliases.Which evaluation metrics are used for the recommender system and why?
- Precision@K: measures how relevant the top-K recommendations are; higher is better.- Recall@K: measures how many relevant items appear in the top-K; higher is better.
- RMS (RMSE): measures rating prediction error; lower is better.
These metrics map to business goals (quality and coverage of recommendations) and should be carefully designed and validated to reflect real-world success.
Machine Learning Platform Engineering ebook for free
