Principle:Kubeflow Pipelines Pipeline Composition
| Field | Value |
|---|---|
| Sources | Kubeflow Pipelines, Pipeline Authoring |
| Domains | ML_Pipelines, Orchestration |
| Last Updated | 2026-02-13 00:00 GMT |
Overview
A design pattern that connects individual ML components into a directed acyclic graph (DAG) defining data flow and execution order.
Description
Pipeline composition involves wiring component tasks together by passing outputs as inputs. The @dsl.pipeline decorator marks a function as a pipeline definition. Inside, component tasks are instantiated and connected via data dependencies (task.output, task.outputs["key"]) or explicit ordering (task.after()). The system infers the execution DAG from these connections. Pipeline parameters become runtime arguments with optional defaults.
Usage
Use when orchestrating multiple components into a reproducible ML workflow. The pipeline function becomes the top-level graph executed on Kubeflow.
Theoretical Basis
DAG-based workflow composition. Nodes are component tasks, edges are data dependencies. Explicit ordering via .after() adds edges without data flow. Pipeline parameters allow runtime configuration.