Environment:Kubeflow Pipelines Kubernetes Cluster

Knowledge Sources	Kubeflow Pipelines Kustomize README AGENTS.md
Domains	Infrastructure, Kubernetes
Last Updated	2026-02-13 13:35 GMT

Overview

Kubernetes cluster environment with kubectl and kustomize required for deploying and operating Kubeflow Pipelines Standalone.

Description

This environment defines the Kubernetes cluster prerequisites for running Kubeflow Pipelines. KFP deploys as a set of microservices (API server, UI, persistence agent, scheduled workflow controller, cache server, metadata writer) onto a Kubernetes cluster. The deployment uses kustomize overlays to compose manifests, with separate cluster-scoped and namespace-scoped resource applications. The cluster must support Custom Resource Definitions (CRDs), RBAC, and PersistentVolumeClaims.

Usage

Use this environment for any KFP Standalone Deployment workflow. It is the mandatory prerequisite for running the Kubectl_Cluster_Setup, Kustomize_Build_Apply, Kubectl_Wait_Pods, and Kubectl_Port_Forward implementations. This environment is also required for running any pipeline that has been submitted to a KFP backend.

System Requirements

Category	Requirement	Notes
OS	Linux (any distribution)	Kubernetes nodes; tested on Ubuntu
Kubernetes	v1.29+	Tested against v1.29.2 and v1.34.0 in CI
Hardware	Minimum 4 CPU cores, 8GB RAM	Per node; more for production workloads
Disk	20GB+ SSD per node	PersistentVolumeClaim storage for MySQL/PostgreSQL and object store
Network	Ports 8080, 8888, 8887, 3306/5432, 9000	UI, API HTTP, API gRPC, DB, object store

Dependencies

System Packages

kubectl compatible with cluster version
kustomize (standalone binary; used by all deployment manifests)
kind (optional, for local development clusters)

Runtime Services (Deployed by KFP)

MySQL 8.4 or PostgreSQL 14.7 (database backend)
SeaweedFS 4.00 or MinIO (object storage for artifacts)
Argo Workflows v3.7.8 (workflow execution engine)
ML Metadata (MLMD) gRPC server (execution metadata store)
Envoy Proxy (gRPC-Web gateway for metadata service)

Credentials

The following Kubernetes Secrets must exist in the deployment namespace:

mysql-secret: Contains username and password fields for MySQL access (or postgres-secret with root_password for PostgreSQL).
mlpipeline-minio-artifact: Contains accesskey and secretkey for object storage access.

Quick Install

# Deploy KFP Standalone on any Kubernetes cluster
KFP_ENV=platform-agnostic
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=master"
kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/${KFP_ENV}/?ref=master"
kubectl wait pods -l application-crd-id=kubeflow-pipelines -n kubeflow --for condition=Ready --timeout=1800s
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80

Code Evidence

Deployment commands from manifests/kustomize/README.md:19-26:

KFP_ENV=platform-agnostic
kustomize build cluster-scoped-resources/ | kubectl apply -f -
kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
kustomize build "env/${KFP_ENV}/" | kubectl apply -f -
kubectl wait pods -l application-crd-id=kubeflow-pipelines -n kubeflow --for condition=Ready --timeout=1800s
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80

CRD separation constraint from manifests/kustomize/README.md:86-87:

CRDs must be applied separately, because if we apply CRs in the same
kubectl apply command, the CRD may not have been accepted by k8s api server.

Go module Kubernetes API version from go.mod:

// k8s.io/api v0.33.1
// k8s.io/apimachinery v0.33.1
// k8s.io/client-go v0.33.1

Common Errors

Error Message	Cause	Solution
`error: unable to recognize: no matches for kind "Application"`	CRDs not yet established	Apply cluster-scoped-resources first and wait for CRD establishment before applying namespace resources
`pods not ready after 1800s`	Insufficient cluster resources or image pull failures	Check node resources with `kubectl describe nodes` and verify image registry access
`connection refused on port 8080`	Port-forward not active or UI pod not ready	Verify pod status with `kubectl get pods -n kubeflow` and restart port-forward

Compatibility Notes

GKE (Google): Use env/gcp overlay for Cloud SQL and GCS integration.
OpenShift: Use env/openshift overlay; cache server and viewer CRD are disabled (scaled to 0 replicas). Security contexts are removed for OpenShift SCC compliance.
KinD (local): Use env/dev-kind overlay with host network forwarding for local development.
Multi-user: Requires Istio service mesh and Metacontroller for per-namespace resource provisioning.
cert-manager: Optional but recommended for production TLS. Eliminates the need for the cache-deployer component.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment