Environment:Kubeflow Pipelines KFP Backend Deployment

Knowledge Sources	Kubeflow Pipelines Backend README Developer Guide
Domains	Infrastructure, Backend, ML_Pipelines
Last Updated	2026-02-13 13:35 GMT

Overview

Running KFP backend services (API server, persistence agent, scheduled workflow controller) with MySQL 8.4 or PostgreSQL 14.7, SeaweedFS 4.00, and Argo Workflows v3.7.8.

Description

This environment defines the running state of a deployed KFP backend. The API server (Go-based, built with Go 1.24.10) serves the REST and gRPC APIs. It connects to a relational database (MySQL or PostgreSQL) for pipeline/run metadata and to SeaweedFS/MinIO for artifact storage. The persistence agent watches Argo Workflow status and syncs it back to the API. The scheduled workflow controller manages recurring pipeline runs. All services communicate over in-cluster networking.

Usage

Use this environment when you need a running KFP API server to submit pipeline runs. It is required by the Client_Create_Run implementation and any workflow that involves pipeline execution (as opposed to local compilation only). It is also the target for the Standalone_Deployment workflow.

System Requirements

Category	Requirement	Notes
OS	Linux (Kubernetes node OS)	Container-based deployment
Kubernetes	v1.29+	See Kubernetes_Cluster environment
Hardware	100m CPU, 800Mi memory minimum per API server	More for production; MySQL requires dedicated resources
Disk	20GB+ PVC storage	For MySQL/PostgreSQL data and SeaweedFS artifacts
Network	Ports 8888 (HTTP), 8887 (gRPC), 3306/5432 (DB), 9000 (object store)	Internal cluster networking

Dependencies

Backend Runtime

Go 1.24.10 (for building backend from source)
Argo Workflows v3.7.8 (workflow execution engine)
ML Metadata (MLMD) gRPC server (execution metadata)

Database (choose one)

MySQL 8.4 (default; image: mysql:8.4)
- Requires mysql_native_password authentication plugin
- Binlog disabled for KFP usage
PostgreSQL 14.7 (alternative; image: postgres:14.7-alpine3.17)

Object Storage

SeaweedFS 4.00 (default; image: chrislusf/seaweedfs:4.00)
MinIO (alternative for GCP Marketplace deployment)

Container Images

ghcr.io/kubeflow/kfp-driver:2.15.0 (V2 pipeline driver)
ghcr.io/kubeflow/kfp-launcher:2.15.0 (V2 pipeline launcher)

Credentials

The following Kubernetes Secrets are required:

mysql-secret: username and password for database access.
mlpipeline-minio-artifact: accesskey and secretkey for object storage.

The following environment variables are configured on the API server deployment:

POD_NAMESPACE: Kubernetes namespace (from downward API).
DBCONFIG_HOST: Database hostname (default: mysql).
DBCONFIG_PORT: Database port (default: 3306).
DBCONFIG_USER: Database username (from Secret).
DBCONFIG_PASSWORD: Database password (from Secret).
OBJECTSTORECONFIG_HOST: Object store hostname (default: seaweedfs.kubeflow).
OBJECTSTORECONFIG_PORT: Object store port (default: 9000).
V2_DRIVER_IMAGE: Container image for pipeline driver.
V2_LAUNCHER_IMAGE: Container image for pipeline launcher.

Quick Install

# Build backend from source (development)
go build -o /tmp/apiserver backend/src/apiserver/*.go

# Build Docker image
docker build -f backend/Dockerfile . --tag api-server:dev

# Run backend tests
go test -v -cover ./backend/...

# Integration tests (requires running KFP)
LOCAL_API_SERVER=true go test -v ./backend/test/v2/integration/... -namespace kubeflow

Code Evidence

Go version from go.mod:3:

go 1.24.10

Argo Workflows dependency from go.mod:8:

github.com/argoproj/argo-workflows/v3 v3.7.8

MySQL connection configuration from manifests/kustomize/base/pipeline/ml-pipeline-apiserver-deployment.yaml:

- name: DBCONFIG_HOST
  valueFrom:
    configMapKeyRef:
      name: pipeline-install-config
      key: mysqlHost
- name: DBCONFIG_PORT
  valueFrom:
    configMapKeyRef:
      name: pipeline-install-config
      key: mysqlPort

MySQL authentication requirement from manifests/kustomize/third-party/mysql/base/mysql-deployment.yaml:

image: mysql:8.4
args:
  - --mysql-native-password=ON
  - --disable-log-bin

Common Errors

Error Message	Cause	Solution
`dial tcp mysql:3306: connection refused`	MySQL pod not ready	Wait for MySQL pod: `kubectl wait pods -l app=mysql -n kubeflow --for condition=Ready`
`Access denied for user`	Incorrect credentials in mysql-secret	Verify secret contents: `kubectl get secret mysql-secret -n kubeflow -o yaml`
`ConMaxLifeTime exceeded`	Database connection pooling issue with external DB	Increase `ConMaxLifeTime` in pipeline-install-config ConfigMap (default: 120s)
`object store bucket not found`	SeaweedFS not initialized	Verify SeaweedFS pod is running and bucket `mlpipeline` exists

Compatibility Notes

MySQL 8.4: The mysql_native_password plugin is deprecated in MySQL 8.0.34 and disabled by default in 8.4. KFP explicitly enables it via --mysql-native-password=ON.
PostgreSQL alternative: Use env/platform-agnostic-postgresql overlay for PostgreSQL backend. Database type is selected via dbType in the install ConfigMap.
GCP Cloud SQL: Use env/gcp overlay with Cloud SQL Proxy sidecar for managed database.
Artifact retention: ARTIFACT_RETENTION_DAYS=-1 means indefinite retention. Set a positive value for automatic cleanup.
Cache staleness: MAXIMUM_CACHE_STALENESS and DEFAULT_CACHE_STALENESS use ISO 8601 duration format (e.g., P30D for 30 days).

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment