Workflow:Kubeflow Pipelines Standalone Deployment
| Knowledge Sources | |
|---|---|
| Domains | ML_Ops, Kubernetes, Deployment |
| Last Updated | 2026-02-13 14:00 GMT |
Overview
End-to-end process for deploying Kubeflow Pipelines as a standalone service on a Kubernetes cluster using kustomize manifests.
Description
This workflow covers the deployment of Kubeflow Pipelines Standalone on any Kubernetes cluster. It uses kustomize to compose and apply Kubernetes manifests that deploy all KFP components: the API server, UI frontend, persistence agent, scheduled workflow controller, cache server, ML Metadata service, object storage (SeaweedFS), and database (MySQL or PostgreSQL). The deployment supports multiple environment targets including platform-agnostic, GCP, OpenShift, and local development (KinD). It also supports single-user and multi-user (Istio-based) configurations.
Key characteristics:
- Uses kustomize base/overlay pattern for composable deployments
- Deploys all required services: API server, UI, Argo Workflows, database, object storage
- Supports MySQL (default) and PostgreSQL database backends
- Optional TLS via cert-manager for pod-to-pod encryption
- Optional multi-user mode with Istio service mesh for namespace isolation
Usage
Execute this workflow when you need to set up a Kubeflow Pipelines environment on a Kubernetes cluster for ML pipeline orchestration. This is appropriate for teams that want a standalone KFP installation without the full Kubeflow platform, or for development and testing environments. Choose between single-user mode (simpler) or multi-user mode (production with access control).
Execution Steps
Step 1: Prepare the Kubernetes Cluster
Ensure a Kubernetes cluster is available and kubectl is configured to communicate with it. For multi-user deployments, Istio service mesh must be installed. For cert-manager TLS deployments, cert-manager must be installed. Verify cluster has sufficient resources for all KFP components.
Key considerations:
- Requires Kubernetes 1.20+ (Docker runtime deprecated)
- Argo Workflows compatibility: v3.5, v3.6, or v3.7
- For local development, a KinD (Kubernetes in Docker) cluster can be used
- Multi-user mode requires Istio and Metacontroller pre-installed
Step 2: Apply Cluster Scoped Resources
Deploy the cluster-scoped resources first, as they define Custom Resource Definitions (CRDs) that must exist before namespace-scoped resources can reference them. These include the Application CRD, ScheduledWorkflow CRD, Viewer CRD, Pipeline CRD, and PipelineVersion CRD.
Key considerations:
- Cluster-scoped resources typically require cluster-admin permissions
- CRDs must be established before applying namespace-scoped resources
- Wait for CRD establishment before proceeding to avoid race conditions
Step 3: Select and Apply Environment Overlay
Choose the appropriate kustomize environment overlay for your target platform and apply it. The overlay composes the base KFP components with environment-specific configurations. Available overlays include platform-agnostic (default), GCP with Cloud SQL, OpenShift, and dev-kind for local development.
Key considerations:
- platform-agnostic: Default overlay for any Kubernetes cluster with MySQL and SeaweedFS
- platform-agnostic-postgresql: Uses PostgreSQL instead of MySQL
- platform-agnostic-multi-user: Adds Istio-based multi-tenancy
- cert-manager overlays: Add TLS encryption between pods
- dev-kind: Optimized for local development on KinD clusters
Step 4: Verify Deployment Health
Wait for all pods to reach Ready state and verify that core services are operational. Check that the API server, UI, database, object storage, and workflow controller are all running.
Key considerations:
- Use kubectl wait to ensure all pods with the KFP label are ready
- Timeout should be generous (e.g., 1800s) for initial image pulls
- Verify that SeaweedFS (object storage) and MySQL/PostgreSQL (database) pods are healthy
- Check that the Argo workflow controller is running
Step 5: Access the KFP UI
Set up access to the Kubeflow Pipelines web interface. For standalone deployments, this is typically done via kubectl port-forward to the ml-pipeline-ui service. For production deployments, an Ingress or LoadBalancer service can be configured.
Key considerations:
- Default port-forward: kubectl port-forward svc/ml-pipeline-ui 8080:80
- The UI is then accessible at http://localhost:8080
- For production, configure an Ingress resource or use the GCP inverse proxy
- Multi-user deployments route through the Istio gateway
Step 6: Validate with a Sample Pipeline
Submit a sample pipeline to verify the deployment is fully functional end-to-end. This confirms that the SDK can connect to the API server, pipelines can be compiled and submitted, the workflow engine can execute steps, and artifacts are stored correctly.
Key considerations:
- Install the KFP SDK: pip install kfp
- Submit a simple pipeline (e.g., the sequential sample)
- Verify the run completes successfully in the UI
- Check that artifacts are persisted in SeaweedFS