Environment:Apache Druid Integration Test Docker

Knowledge Sources	Apache Druid
Domains	Integration Testing, Docker, Docker Compose, CI/CD
Last Updated	2026-02-10 10:00 GMT

Overview

The Integration Test Docker environment provides a Docker Compose-based cluster that provisions a complete multi-node Apache Druid installation with ZooKeeper, Kafka, and MySQL for running automated integration tests.

Description

The integration test environment creates a self-contained Druid cluster inside Docker containers using a custom image built on eclipse-temurin:21-jre-jammy (JDK 21 runtime). The cluster runs on a custom bridge network (172.172.172.0/24) with static IP assignments for each service: ZooKeeper/Kafka at 172.172.172.2, metadata storage (MySQL) at 172.172.172.3, Overlords at 172.172.172.10-19, Coordinators at 172.172.172.20-29, Historicals at 172.172.172.30-39, Middle Managers at 172.172.172.40-49, Indexers at 172.172.172.50-59, Brokers at 172.172.172.60-69, and Routers at 172.172.172.70-79.

The Docker Compose configuration uses version 2.2 and supports multiple overlay files for different test scenarios: base configuration, CLI indexer, Hadoop, LDAP security, TLS, and schema registry variants. The it.sh script at the repository root provides a convenient interface for building, starting, running, and tearing down test clusters.

For the web console specifically, end-to-end tests use Playwright Chromium (^1.24.1) with Jest (^29.7.0), controllable via environment variables for headless mode and port configuration.

Usage

Use this environment when:

• Running the Druid integration test suite locally or in CI
• Validating end-to-end behavior of Druid ingestion, querying, and cluster management
• Testing the web console E2E tests against a real Druid cluster
• Reproducing CI failures that occur in the Docker-based test infrastructure
• Testing Druid with different metadata stores, security configurations, or indexer types

System Requirements

Category	Requirement	Notes
Docker Engine	Docker with >= 4 GB memory allocated	Required for running the multi-container cluster
Docker Compose	Version 2.2	Specified in docker-compose.base.yml:16
Docker Base Image	eclipse-temurin:21-jre-jammy	JDK 21 runtime; Dockerfile:17
Java (compile)	17	For building Druid from source
Java (Docker runtime)	21	JRE inside the Docker image
OS	Linux, macOS (Docker Desktop)	Docker must support custom bridge networks
Network	172.172.172.0/24 available	Custom bridge network; conflicts will prevent startup
Disk Space	~10 GB	For Docker images, build artifacts, and test data

Dependencies

System Packages

• Docker (with Docker Compose v2 plugin)
• Java 17 (for compiling Druid locally before building the Docker image)
• Maven (for the Druid build)
• Bash (for it.sh helper script)

Language Packages

Bundled in Docker image:

• Apache Kafka 3.9.1 (from pom.xml:78: apache.kafka.version)
• Apache ZooKeeper 3.8.4 (from pom.xml:129: zookeeper.version)
• MySQL 8.2.0 (from pom.xml:109: mysql.version)
• MariaDB Java Client 2.7.3 (from pom.xml:110: mariadb.version; used as alternative MySQL driver)
• Apache Hadoop 3.3.6 (from pom.xml:119: hadoop.compile.version; optional, for Hadoop integration tests)

For web console E2E tests:

• playwright-chromium ^1.24.1
• jest ^29.7.0

Credentials

The following environment variables configure the integration test framework:

Variable	Purpose
DRUID_INTEGRATION_TEST_INDEXER	Selects indexer type: middleManager (default) or indexer
DRUID_INTEGRATION_TEST_GROUP	Selects which test group/category to run
DRUID_INTEGRATION_TEST_OVERRIDE_CONFIG_PATH	Path to override environment config for Docker containers
DRUID_INTEGRATION_TEST_SKIP_BUILD_DOCKER	Set to skip rebuilding the Docker image
DRUID_INTEGRATION_TEST_SKIP_RUN_DOCKER	Set to skip starting Docker containers (use existing cluster)
DRUID_INTEGRATION_TEST_START_HADOOP_DOCKER	Set to include the Hadoop container
DOCKER_IP	IP address of the Docker host (used for Kafka external listener)
APACHE_ARCHIVE_MIRROR_HOST	Mirror host for downloading Kafka/ZK archives (default: https://archive.apache.org/dist)
OVERRIDE_ENV	Path to additional environment file passed to Docker Compose services
DRUID_E2E_TEST_HEADLESS	Controls headless mode for web console Playwright E2E tests
DRUID_E2E_TEST_UNIFIED_CONSOLE_PORT	Port override for web console E2E test target

Database credentials (hard-coded in Dockerfile):

• MySQL user: druid, password: diurd, database: druid

Quick Install

# Full pipeline: build Druid, build Docker image, run tests
./it.sh test <category>

# Step-by-step approach:

# 1. Build Druid and the distribution
./it.sh build

# 2. Build the integration test tools
./it.sh tools

# 3. Build the Docker image
./it.sh image

# 4. Start the cluster for a specific test category
./it.sh up <category>

# 5. Run tests against the running cluster
./it.sh run <category>

# 6. Inspect container logs if tests fail
./it.sh tail <category>

# 7. Tear down the cluster
./it.sh down <category>

# Run web console E2E tests (requires a running Druid cluster)
cd web-console
npm run test-e2e

# Clean up all Docker containers if stuck
./it.sh prune-containers
./it.sh prune-volumes

Code Evidence

Docker base image (integration-tests/docker/Dockerfile:16-17):

ARG JDK_VERSION=21-jre-jammy
FROM eclipse-temurin:$JDK_VERSION AS druidbase

Docker Compose version and network (integration-tests/docker/docker-compose.base.yml:16-34):

version: "2.2"

# IP address ranges:
# 172.172.172.2: zookeeper + kafka (dockerfile depends on this)
# 172.172.172.3: metadata store
# 172.172.172.10-19: overlord
# 172.172.172.20-29: coordinator
# 172.172.172.30-39: historicals
# 172.172.172.40-49: middle managers
# 172.172.172.50-59: indexers
# 172.172.172.60-69: brokers
# 172.172.172.70-79: routers
# 172.172.172.101+:  hadoop, other supporting infra
networks:
  druid-it-net:
    name: druid-it-net
    ipam:
      config:
        - subnet: 172.172.172.0/24

Port exposures (integration-tests/docker/Dockerfile:101-121):

# Expose ports:
# - 8081, 8281: HTTP, HTTPS (coordinator)
# - 8082, 8282: HTTP, HTTPS (broker)
# - 8083, 8283: HTTP, HTTPS (historical)
# - 8090, 8290: HTTP, HTTPS (overlord)
# - 8091, 8291: HTTP, HTTPS (middlemanager)
# - 8888-8891, 9088-9091: HTTP, HTTPS (routers)
# - 3306: MySQL
# - 2181 2888 3888: ZooKeeper
# - 8100 8101 8102 8103 8104 8105 : peon ports
# - 8300 8301 8302 8303 8304 8305 : peon HTTPS ports
EXPOSE 8081 8281
EXPOSE 8082 8282
EXPOSE 8083 8283
EXPOSE 8090 8290
EXPOSE 8091 8291
EXPOSE 3306
EXPOSE 2181 2888 3888
EXPOSE 8100 8101 8102 8103 8104 8105
EXPOSE 8300 8301 8302 8303 8304 8305
EXPOSE 9092 9093

Component versions (pom.xml):

<apache.kafka.version>3.9.1</apache.kafka.version>
<mysql.version>8.2.0</mysql.version>
<mariadb.version>2.7.3</mariadb.version>
<zookeeper.version>3.8.4</zookeeper.version>
<hadoop.compile.version>3.3.6</hadoop.compile.version>

Kafka listener configuration on the custom network (integration-tests/docker/Dockerfile:91-93):

RUN perl -pi -e "s/#listeners=.*/listeners=INTERNAL:\/\/172.172.172.2:9092,EXTERNAL:\/\/172.172.172.2:9093/" /usr/local/kafka/config/server.properties
RUN perl -pi -e "s/#advertised.listeners=.*/advertised.listeners=INTERNAL:\/\/172.172.172.2:9092,EXTERNAL:\/\/$(cat docker_ip):9093/" /usr/local/kafka/config/server.properties

Common Errors

Error Message	Cause	Solution
Cannot create network druid-it-net: subnet 172.172.172.0/24 overlaps	An existing Docker network uses the same subnet	Run docker network prune or ./it.sh prune-containers to clean up
docker: Error response from daemon: Ports are not available	Host ports already in use (e.g., 8081, 8888, 3306)	Stop conflicting services or adjust port mappings
Out of memory errors in containers	Docker engine has less than 4 GB memory	Increase Docker Desktop memory allocation to at least 4 GB (6 GB recommended)
MySQL did not start during image build	MariaDB service fails to start inside the Docker build	Retry the build; check for disk space issues or conflicting Docker builds
E2E test timeout	Druid cluster not fully initialized when tests start	Increase test timeout or verify all services are healthy before running tests
Connection refused to Kafka on port 9093	DOCKER_IP not set correctly for external Kafka listener	Set DOCKER_IP to the Docker host IP address

Compatibility Notes

• The Docker image uses JDK 21 (eclipse-temurin:21-jre-jammy) at runtime, while Druid is compiled with Java 17. This is intentional for forward compatibility testing.
• Docker Compose version 2.2 is required; version 3.x is not used because it removes certain features needed by the test framework (e.g., mem_limit).
• The 172.172.172.0/24 network is hard-coded in both the Docker Compose files and the Dockerfile Kafka configuration. Changing it requires coordinated updates.
• Kafka uses dual listeners: INTERNAL (172.172.172.2:9092) for inter-container communication and EXTERNAL (docker_ip:9093) for test code running on the host.
• The MySQL metadata store credentials (druid/diurd) are hard-coded and intended only for test environments.
• The it.sh script supports both middleManager and indexer task runner modes via the DRUID_INTEGRATION_TEST_INDEXER variable.
• Hadoop integration tests are optional and require setting DRUID_INTEGRATION_TEST_START_HADOOP_DOCKER=true to include the Hadoop container.

Related Pages

• Implementation:Apache_Druid_Task_Submission_Api