5 Orchestrating ML pipelines

This chapter moves from manual, tool-by-tool ML workflows to automated, reproducible orchestration with Kubeflow Pipelines. Building on experiment tracking with MLflow and feature management with Feast, it demonstrates how to turn a batch income-classification workflow into a scalable pipeline. The focus is on expressing pipelines as code with the kfp SDK, packaging repeatable steps, and enabling parameterized, scheduled runs that improve consistency and collaboration across teams.

The core building block is the Kubeflow component: a single-responsibility task implemented in Python, containerized with Docker, and described by a YAML spec that declares typed inputs and outputs (for example, Dataset artifacts). Four reusable components illustrate the pattern end to end: reading entity IDs from MinIO, retrieving features from Feast using the cluster-facing object-store endpoint, loading a staged model from MLflow to run inference, and writing predictions back to MinIO. Artifacts flow between steps via inputPath/outputPath, ensuring clean handoffs. Design guidance emphasizes modularity, clear I/O contracts, parameterization for portability across environments, and documentation—yielding benefits in reusability, parallel team development, and isolated testing.

These components are composed into a directed acyclic graph that wires task dependencies through task.outputs. The pipeline function defines runtime parameters (for storage, feature lists, and model registry settings), is compiled to a YAML package, and is executed and tracked through the Kubeflow UI using experiments and runs. With intermediate datasets persisted in object storage, executions are reproducible and easy to audit. Overall, the chapter shows how Kubeflow Pipelines provide reliable orchestration, scaling, and tracking for batch inference workflows, setting the stage for real-time serving patterns addressed later with complementary tools.

The mental map where we are now focusing on Kubeflow pipeline orchestration(A).

Kubeflow pipeline interactions with MinIO, Feast and MLFlow. All of the intermediate datasets are stored in MinIO to be retrieved by subsequent steps. The model and features are retrieved from MinIO and Feast respectively.

Kubeflow pipeline UI that lists out all the pipelines in our account.

Kubeflow pipeline UI to create a pipeline. Need to provide a pipeline name and upload the pipeline yaml file.

Pipeline visualized by the Kubeflow pipeline UI after we upload it.

Create an experiment in Kubeflow, by providing a name and experiment description.

Create a pipeline run by providing all the pipeline arguments.

A successful Kubeflow pipeline run.

• Orchestration is the process of running multiple complex pipelines while making sure that the pipeline steps have the data and the compute needed. The kubeflow platform offers kubeflow pipelines as a solution.
• A component is a reusable and composable unit of work within a machine learning (ML) workflow. These components encapsulate individual tasks or steps that perform specific actions, such as data preprocessing, model training, evaluation, deployment, and more.
• While designing components, care should be taken to keep them as independent and generic as possible, allowing for reusability, easier development, testing, and maintenance of individual pipeline stages.
• Larger workflows can be broken down into smaller components. This approach helps create modular and easily maintainable ML pipelines from shared components.
• Kubeflow Pipelines and task orchestrators can be used to automate, sequence, and manage machine learning workflows, ensuring the efficient execution of data preprocessing, model training, and deployment tasks. It offers SDKs to build up individual components, connecting components to a pipeline as well as pipeline deployment and execution.

FAQ