Principle:Dagster io Dagster Kubernetes Helm Test Infrastructure
| Knowledge Sources | |
|---|---|
| Domains | Kubernetes, Testing, Helm, CI_CD |
| Last Updated | 2026-02-10 12:30 GMT |
Overview
Design principle for provisioning ephemeral Kubernetes environments via Helm chart installation to integration-test orchestration infrastructure.
Description
Kubernetes Helm Test Infrastructure is the practice of using pytest session-scoped fixtures to create isolated, disposable Kubernetes environments for integration testing. Instead of mocking Kubernetes interactions, this approach deploys the actual Dagster Helm chart into real (or Kind) clusters, creating namespaces, ConfigMaps, Secrets, and running actual pods. This validates the full deployment lifecycle: Helm chart rendering, pod scheduling, service discovery, and inter-component communication. The principle addresses the gap between unit tests (which mock infrastructure) and production deployments (which are expensive to test) by providing reproducible, automated integration environments.
Usage
Apply this principle when testing Kubernetes-specific functionality such as run launchers (K8sRunLauncher, CeleryK8sRunLauncher), pod-level configuration (volumes, env vars, secrets), run monitoring, and multi-namespace deployments. This is essential for validating that Helm chart values produce correct Kubernetes resources and that Dagster components communicate correctly within a cluster.
Theoretical Basis
The core mechanism is fixture-driven environment lifecycle management:
- Namespace isolation: Each test session gets a unique namespace (e.g., dagster-test-3fcd70) to prevent interference
- Resource provisioning: ConfigMaps, Secrets, and RBAC bindings are created as composable pytest fixtures
- Helm chart installation: The actual Helm chart is installed with test-specific values (run launcher type, Celery backend, monitoring settings)
- Readiness polling: The system waits for webserver, daemon, Celery worker, and user-code-deployment pods to reach Ready state
- Port forwarding: The webserver is made accessible via kubectl port-forward for API-level assertions
- Cleanup: Resources are torn down after the session unless --no-cleanup is specified or running in ephemeral CI
Pseudo-code Logic:
# Abstract test environment lifecycle
def integration_test_session():
namespace = create_unique_namespace()
create_configmaps(namespace)
create_secrets(namespace)
helm_install(chart="dagster", namespace=namespace, values=test_config)
wait_for_pods_ready(namespace, ["webserver", "daemon", "user-code"])
url = port_forward(namespace, pod="webserver", port=80)
yield url # Tests run here
helm_uninstall(namespace)
delete_namespace(namespace)