Implementation:Risingwavelabs Risingwave CI Docker Compose
| Knowledge Sources | |
|---|---|
| Domains | CI, Infrastructure, Testing |
| Last Updated | 2026-02-09 07:00 GMT |
Overview
Docker Compose configuration that defines the complete set of external services and composite test environments used in RisingWave CI pipelines for connector integration testing.
Description
The ci/docker-compose.yml file configures over 25 Docker services representing the external systems that RisingWave integrates with. These include databases (PostgreSQL, MySQL, SQL Server, ClickHouse, MongoDB, Cassandra, Doris, StarRocks), message brokers (Redpanda/Kafka, RabbitMQ, Pulsar, NATS, MQTT), storage and search (Elasticsearch, Redis), authentication (LDAP, Vault), and schema management (Confluent Schema Registry). It also defines composite test environments (e.g., source-test-env, sink-test-env, iceberg-test-env) that bundle the rw-build-env image with appropriate service dependencies. Release build environments based on manylinux_2_28 are included for cross-platform binary compilation.
Usage
Used exclusively in CI pipelines. Individual test jobs select the appropriate composite environment or specific services via docker compose --profile or depends_on relationships.
Code Reference
Source Location
- Repository: risingwave
- File: ci/docker-compose.yml
- Lines: L1-562
Signature
# Start CI source test environment
docker compose -f ci/docker-compose.yml up source-test-env
# Start CI sink test environment
docker compose -f ci/docker-compose.yml up sink-test-env
# Start specific services
docker compose -f ci/docker-compose.yml up db message_queue schemaregistry
Import
# Requires BUILD_ENV_VERSION environment variable
export BUILD_ENV_VERSION=v20240101
docker compose -f ci/docker-compose.yml up <service>
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| BUILD_ENV_VERSION | Environment variable | Yes | Version tag for the rw-build-env Docker image |
Outputs
| Name | Type | Description |
|---|---|---|
| db (PostgreSQL) | Port 5432 | PostGIS-enabled PostgreSQL 15 with WAL replication |
| mysql | Port 3306 | MySQL 8.0 for connector testing |
| mysql-meta | Port 3306 | Separate MySQL 8.0 for meta store testing |
| message_queue (Redpanda) | Ports 9092, 29092 | Kafka-compatible message broker |
| schemaregistry | Port 8082 | Confluent Schema Registry |
| pulsar-server | Ports 6650, 8080 | Apache Pulsar standalone |
| mongodb | Port 27017 | MongoDB 8 replica set |
| elasticsearch | Port 9200 | Elasticsearch 7.11 with security enabled |
| clickhouse-server | Ports 8123, 9000 | ClickHouse column store |
| redis-server | Port 6379 | Redis cache |
| cassandra-server | Port 9042 | Cassandra 4.0 |
| nats-server | Ports 4222, 8222 | NATS with JetStream |
| vault-server | Port 8200 | HashiCorp Vault in dev mode |
| ldap-server | Ports 389, 636 | OpenLDAP with TLS |
Composite Test Environments
| Environment | Dependencies | Purpose |
|---|---|---|
| source-test-env | mysql, db, message_queue, schemaregistry, pulsar, mongodb, nats, vault, rabbitmq, SASL kafka | Source connector integration tests |
| sink-test-env | mysql, db, message_queue, schemaregistry, elasticsearch, clickhouse, redis, pulsar, mqtt, cassandra, doris, starrocks, mongodb, sqlserver | Sink connector integration tests |
| iceberg-test-env | mysql, db, message_queue, schemaregistry | Apache Iceberg integration tests |
| ldap-test-env | ldap-server | LDAP authentication tests |
| ci-standard-env | mysql-meta, db | Standard CI with metadata stores |
| regress-test-env | db | PostgreSQL regression tests |
| release-env-x86 / release-env-arm | (none) | Cross-platform binary builds (manylinux_2_28) |
Usage Examples
Running Source Connector Tests
# Start the source test environment with all dependencies
export BUILD_ENV_VERSION=v20240101
docker compose -f ci/docker-compose.yml run source-test-env bash
# Inside the container, build and run tests
cd /risingwave
./risedev ci-start ci-kafka
./risedev slt './e2e_test/source/**/*.slt'
Running Sink Tests
docker compose -f ci/docker-compose.yml run sink-test-env bash
# Inside container:
cd /risingwave
./risedev ci-start ci-sink-test
./risedev slt './e2e_test/sink/**/*.slt'
Related Pages
Implements Principle
Related Implementations
- Implementation:Risingwavelabs_Risingwave_Risedev_Configuration -- CI profiles reference services defined here
- Implementation:Risingwavelabs_Risingwave_Makefile_Task_Runner -- CI tasks (ci-start, ci-kill) orchestrate test runs
- Implementation:Risingwavelabs_Risingwave_Docker_Compose_Lakekeeper -- Lakekeeper integration for Iceberg testing