Implementation:Kubeflow Pipelines Pipeline Spec Protobuf
| Knowledge Sources | |
|---|---|
| Domains | Pipeline_Specification, API_Schema |
| Last Updated | 2026-02-13 14:00 GMT |
Overview
Auto-generated Go protobuf bindings for the complete KFP v2 pipeline specification schema, defining the canonical data model for pipelines, components, tasks, artifacts, parameters, and deployment configurations.
Description
The pipeline_spec.pb.go file is the single most critical API definition in KFP. Generated by `protoc-gen-go` (v1.36.6) from `pipeline_spec.proto`, it defines the Intermediate Representation (IR) that the Python SDK compiles pipelines into and that the backend interprets to orchestrate workflows. At 6,480 lines, it encompasses the entire pipeline specification language with 5 enums and 110 message types.
Key type hierarchy:
- PipelineJob — Top-level job submission container
- PipelineSpec — Component graph definition with runtime parameters
- ComponentSpec — Individual component (DAG or executor)
- DagSpec — Task dependency graph within a component
- PipelineTaskSpec — Individual task with caching, triggers, retries, iterators
- PipelineDeploymentConfig — Container, importer, resolver, and custom job specs
- ExecutorInput/ExecutorOutput — Runtime I/O for task execution
- PlatformSpec — Platform-specific deployment configuration
Enums: PrimitiveType (DOUBLE, INTEGER, STRING), ParameterType (NUMBER_DOUBLE, NUMBER_INTEGER, STRING, BOOLEAN, LIST, STRUCT, TASK_FINAL_STATUS), PipelineStateEnum, TaskConfigPassthroughType.
Usage
This protobuf package is imported by the KFP backend to deserialize pipeline specifications submitted by the SDK, and by the KFP SDK compiler to serialize pipeline definitions into the IR YAML format. Every pipeline submitted to KFP is serialized using these types.
Code Reference
Source Location
- Repository: Kubeflow_Pipelines
- File: api/v2alpha1/go/pipelinespec/pipeline_spec.pb.go
- Lines: 1-6480
Signature
package pipelinespec
// PipelineJob is the top-level pipeline execution container.
type PipelineJob struct {
Name string
PipelineSpec *PipelineSpec
RuntimeConfig *PipelineJob_RuntimeConfig
// ...
}
// PipelineSpec defines the component graph and deployment config.
type PipelineSpec struct {
Components map[string]*ComponentSpec
DeploymentSpec *PipelineDeploymentConfig
Root *ComponentSpec
// ...
}
// ComponentSpec defines a single component as either DAG or executor.
type ComponentSpec struct {
InputDefinitions *ComponentInputsSpec
OutputDefinitions *ComponentOutputsSpec
// OneOf: Dag (*DagSpec) or ExecutorLabel (string)
}
Import
import "github.com/kubeflow/pipelines/api/v2alpha1/go/pipelinespec"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| Proto descriptor | []byte | Yes | Raw protobuf descriptor bytes for pipeline_spec.proto |
Outputs
| Name | Type | Description |
|---|---|---|
| Go types | structs | 110 message types and 5 enums for pipeline specification |
| PipelineJob | struct | Top-level pipeline job submission type |
| PipelineSpec | struct | Component graph and deployment config type |
| ExecutorInput | struct | Runtime input passed to task executors |
Usage Examples
Deserializing a Pipeline Spec
package main
import (
"github.com/kubeflow/pipelines/api/v2alpha1/go/pipelinespec"
"google.golang.org/protobuf/encoding/protojson"
)
func parsePipelineSpec(jsonData []byte) (*pipelinespec.PipelineSpec, error) {
spec := &pipelinespec.PipelineSpec{}
err := protojson.Unmarshal(jsonData, spec)
if err != nil {
return nil, err
}
return spec, nil
}