Implementation:DataTalksClub Data engineering zoomcamp Kestra Docker Compose Setup

Metadata
Knowledge Sources	repo: DataTalksClub/data-engineering-zoomcamp, docs: Kestra Documentation, Docker Compose Documentation
Domains	Infrastructure, Orchestration, Docker, Container Management
Last Updated	2026-02-09 14:00 GMT

Overview

Concrete tool for provisioning the Kestra orchestration platform alongside its metadata database, a target PostgreSQL data store, and a pgAdmin web interface using a Docker Compose multi-service configuration.

Description

This implementation defines a four-service Docker Compose stack for local development of Kestra-based ETL pipelines. The stack provisions:

• kestra (image: kestra/kestra:v1.1, port 8080) -- the workflow orchestrator running in standalone server mode. It is configured with PostgreSQL-backed repository and queue storage, local file storage at /app/storage, and basic authentication with credentials admin@kestra.io / Admin1234!.
• kestra_postgres (image: postgres:18) -- the metadata database that stores Kestra's internal state including flow definitions, execution history, and task queues. Configured with database kestra, user kestra, password k3str4.
• pgdatabase (image: postgres:18, port 5432) -- the target data store for pipeline output. Configured with database ny_taxi, user root, password root.
• pgadmin (image: dpage/pgadmin4, port 8085) -- web-based database administration tool for inspecting the target data store.

Dependency ordering ensures services start in the correct sequence: kestra_postgres first (with health check), then kestra, then pgdatabase, then pgadmin. Four named volumes provide persistent storage across container restarts.

Usage

Deploy the entire stack:

docker-compose up -d

Access the Kestra UI at http://localhost:8080 and pgAdmin at http://localhost:8085. The target PostgreSQL database is accessible on localhost:5432.

Code Reference

Source Location: 02-workflow-orchestration/docker-compose.yml, Lines 1-94

Signature:

volumes:
  ny_taxi_postgres_data:
    driver: local
  kestra_postgres_data:
    driver: local
  kestra_data:
    driver: local
  kestra_tmp:
    driver: local

services:
  pgdatabase:
    image: postgres:18
    environment:
      POSTGRES_USER: root
      POSTGRES_PASSWORD: root
      POSTGRES_DB: ny_taxi
    ports:
      - "5432:5432"
    volumes:
      - ny_taxi_postgres_data:/var/lib/postgresql
    depends_on:
      kestra:
        condition: service_started

  pgadmin:
    image: dpage/pgadmin4
    environment:
      - PGADMIN_DEFAULT_EMAIL=admin@admin.com
      - PGADMIN_DEFAULT_PASSWORD=root
    ports:
      - "8085:80"
    depends_on:
      pgdatabase:
        condition: service_started

  kestra_postgres:
    image: postgres:18
    volumes:
      - kestra_postgres_data:/var/lib/postgresql
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 30s
      timeout: 10s
      retries: 10

  kestra:
    image: kestra/kestra:v1.1
    pull_policy: always
    user: "root"
    command: server standalone
    volumes:
      - kestra_data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - kestra_tmp:/tmp/kestra-wd
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://kestra_postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          server:
            basicAuth:
              username: "admin@kestra.io"
              password: Admin1234!
          repository:
            type: postgres
          storage:
            type: local
            local:
              basePath: "/app/storage"
          queue:
            type: postgres
          tasks:
            tmpDir:
              path: /tmp/kestra-wd/tmp
          url: http://localhost:8080/
    ports:
      - "8080:8080"
      - "8081:8081"
    depends_on:
      kestra_postgres:
        condition: service_started

Import: N/A -- this is a Docker Compose file executed with docker-compose up.

I/O Contract

Inputs:

Name	Type	Description
Docker Compose file	YAML configuration	docker-compose.yml defining all services, volumes, and network configuration
Docker socket	Unix socket	/var/run/docker.sock mounted for Kestra container management

Outputs:

Name	Type	Description
Kestra UI	HTTP endpoint	http://localhost:8080 -- orchestration web interface (auth: admin@kestra.io / Admin1234!)
Kestra API	HTTP endpoint	http://localhost:8081 -- Kestra management API
Target PostgreSQL	TCP endpoint	localhost:5432 -- ny_taxi database (user: root, pass: root)
pgAdmin	HTTP endpoint	http://localhost:8085 -- database admin interface (admin@admin.com / root)
Named volumes	Docker volumes	4 persistent volumes for data, metadata, Kestra storage, and temp files

Usage Examples

Starting the full stack:

cd 02-workflow-orchestration
docker-compose up -d

Verifying all services are running:

docker-compose ps

Connecting to the target database with psql:

psql -h localhost -p 5432 -U root -d ny_taxi

Tearing down the stack while preserving data volumes:

docker-compose down

Tearing down the stack and removing all data:

docker-compose down -v

Related Pages

• Principle:DataTalksClub_Data_engineering_zoomcamp_Kestra_Infrastructure_Setup
• Environment:DataTalksClub_Data_engineering_zoomcamp_Kestra_Orchestration_Environment