Workflow:Kubeflow Pipelines Pipeline Control Flow
| Knowledge Sources | |
|---|---|
| Domains | ML_Ops, Pipeline_Development, SDK |
| Last Updated | 2026-02-13 14:00 GMT |
Overview
Demonstrates advanced pipeline control flow patterns including conditional branching, parallel loops, exit handlers, and recursive sub-pipelines using the KFP DSL.
Description
This workflow covers the KFP DSL control structures that enable non-linear pipeline execution. It demonstrates conditional execution (dsl.Condition) where branches run based on runtime output values, parallel fan-out using dsl.ParallelFor to iterate over static or dynamic lists, exit handlers (dsl.ExitHandler) that guarantee cleanup tasks run regardless of pipeline success or failure, and recursive execution (dsl.graph_component) for iterative processing loops. These patterns allow users to build sophisticated, production-grade ML pipelines that go beyond simple sequential task chains.
Key characteristics:
- dsl.Condition enables runtime branching based on task outputs
- dsl.ParallelFor enables fan-out over lists with configurable parallelism
- dsl.ExitHandler guarantees cleanup task execution on success or failure
- dsl.graph_component enables recursive sub-pipeline execution
- Nested conditions and loops can be combined for complex flow control
Usage
Execute this workflow when your pipeline requires non-linear execution patterns: branching logic based on runtime results, processing variable-length data in parallel, guaranteed cleanup regardless of failures, or iterative processing until a condition is met. These patterns are essential for production ML pipelines that need to handle variable data, make runtime decisions, and maintain resource hygiene.
Execution Steps
Step 1: Define Conditional Branching
Use dsl.Condition to create conditional branches that execute based on the runtime output of a previous task. The condition compares a task output against a value, and the enclosed tasks only execute when the condition is true. Conditions can be nested for multi-level decision trees.
Key considerations:
- Conditions evaluate task outputs at runtime, not at compile time
- Nested conditions create decision trees with multiple levels
- Only simple comparisons (==, !=, >, <, >=, <=) are supported in conditions
- Tasks inside a condition block are skipped (not failed) when the condition is false
Step 2: Implement Parallel Loops
Use dsl.ParallelFor to fan out execution across a list of items. The items can be statically defined in the pipeline or dynamically generated as the output of a preceding task. Each iteration runs as an independent task, and parallelism can be limited using the parallelism parameter.
Key considerations:
- Static lists are defined inline; dynamic lists come from task outputs
- Items can be simple values or structured objects (dicts) with named fields
- Use the parallelism parameter to control max concurrent iterations
- Loop outputs are collected as lists for downstream consumption
Step 3: Add Exit Handlers
Use dsl.ExitHandler to wrap a section of the pipeline with a guaranteed cleanup or notification task. The exit handler task executes after all enclosed tasks complete, regardless of whether they succeeded or failed. This pattern is essential for resource cleanup, notification, and audit logging.
Key considerations:
- The exit task runs even if enclosed tasks fail
- Only one exit task can be specified per ExitHandler block
- Exit handlers cannot access outputs from the enclosed tasks
- Use for cleanup operations like deleting temporary resources or sending notifications
Step 4: Create Recursive Sub Pipelines
Use @dsl.graph_component to define sub-pipelines that can call themselves recursively. Combined with dsl.Condition, this enables train-evaluate-retry loops where the pipeline continues processing until a quality threshold is met.
Key considerations:
- Recursive components use @dsl.graph_component decorator
- A termination condition (dsl.Condition) must guard the recursive call
- Each recursive invocation creates new task instances in the execution graph
- Be mindful of maximum recursion depth to avoid runaway pipelines
Step 5: Compose Combined Control Flow
Combine multiple control flow patterns in a single pipeline for sophisticated orchestration. For example, an exit handler can wrap a section that includes parallel loops with conditional branches inside each iteration, creating a robust and flexible execution graph.
Key considerations:
- Control structures can be freely nested and combined
- Data dependencies still drive execution order within control blocks
- The compiled IR YAML captures the full control flow graph
- Monitor complex pipelines carefully using the KFP UI DAG visualization