Environment:DataTalksClub Data engineering zoomcamp Kestra Orchestration Environment

Knowledge Sources	Data Engineering Zoomcamp Kestra Documentation
Domains	Infrastructure, Workflow_Orchestration
Last Updated	2026-02-09 07:00 GMT

02-workflow-orchestration/docker-compose.yml

Overview

Docker Compose environment with Kestra v1.1, PostgreSQL 18, and pgAdmin 4 for workflow orchestration of ETL pipelines.

Description

This environment provides the Kestra workflow orchestration platform backed by PostgreSQL for both metadata storage and as the target database for ETL pipelines. The stack includes Kestra v1.1 running in standalone server mode with a dedicated PostgreSQL instance for Kestra internals, a separate PostgreSQL 18 instance for taxi data, and pgAdmin 4 for database management. Kestra requires Docker socket access to execute containerized tasks.

Usage

Use this environment for any ETL orchestration workflow built with Kestra. It is the mandatory prerequisite for running the Kestra_Docker_Compose_Setup, Kestra_Shell_Commands, Kestra_PostgreSQL_Queries_DDL, Kestra_PostgreSQL_CopyIn, Kestra_MD5_Deduplication, Kestra_SQL_Merge, and Kestra_PurgeCurrentExecutionFiles implementations.

System Requirements

Category	Requirement	Notes
OS	Linux, macOS, or Windows with Docker	Docker Desktop or Docker Engine required
Software	Docker Engine + Docker Compose V2	Docker socket access needed
Disk	~5GB free	For Docker images and persistent volumes
Network	Internet access	Kestra pulls plugin images dynamically

Dependencies

Container Images

`kestra/kestra:v1.1` (orchestration engine)
`postgres:18` x2 (one for Kestra metadata, one for taxi data)
`dpage/pgadmin4` (database UI)

Kestra Plugins (used in flows)

`io.kestra.plugin.scripts.shell.Commands` (shell task execution)
`io.kestra.plugin.jdbc.postgresql.Queries` (SQL DDL execution)
`io.kestra.plugin.jdbc.postgresql.CopyIn` (bulk data loading)
`io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles` (cleanup)

Credentials

The following credentials are configured in the Docker Compose environment:

Taxi Data PostgreSQL:

`POSTGRES_USER`: Username (default: `root`)
`POSTGRES_PASSWORD`: Password (default: `root`)
`POSTGRES_DB`: Database name (default: `ny_taxi`)

Kestra Internal PostgreSQL:

`POSTGRES_USER`: Username (default: `kestra`)
`POSTGRES_PASSWORD`: Password (default: `k3str4`)
`POSTGRES_DB`: Database name (default: `kestra`)

Kestra Web UI:

`username`: Admin login (default: `admin@kestra.io`)
`password`: Admin password (default: `Admin1234!`)

Warning: These are development-only defaults. Never use these credentials in production.

Quick Install

# Start the full Kestra stack
cd 02-workflow-orchestration
docker compose up -d

# Access Kestra UI at http://localhost:8080
# Access pgAdmin at http://localhost:8085

Code Evidence

Docker Compose service definitions from `02-workflow-orchestration/docker-compose.yml:51-57`:

  kestra:
    image: kestra/kestra:v1.1
    pull_policy: always
    # Note that this setup with a root user is intended for development purpose.
    # Our base image runs without root, but the Docker Compose implementation needs root to access the Docker socket
    # To run Kestra in a rootless mode in production, see: https://kestra.io/docs/installation/podman-compose
    user: "root"

Docker socket volume mount from `docker-compose.yml:60-62`:

    volumes:
      - kestra_data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - kestra_tmp:/tmp/kestra-wd

Common Errors

Error Message	Cause	Solution
`permission denied while trying to connect to the Docker daemon socket`	Kestra container not running as root	Ensure `user: "root"` is set in docker-compose.yml
`kestra_postgres health check failing`	PostgreSQL not ready	Wait for health check (interval: 30s, retries: 10)
Port 8080 already in use	Another service on port 8080	Change port mapping or stop conflicting service

Compatibility Notes

Docker Socket: Kestra requires access to `/var/run/docker.sock` for executing containerized tasks. This means the Kestra container must run as root in development.
Production Deployment: For rootless production deployments, use Podman Compose as documented at https://kestra.io/docs/installation/podman-compose.
ARM64 (Apple Silicon): Kestra images support ARM64 architecture.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment