Implementation:Apache Hudi Stop Demo Script

Knowledge Sources	Apache Hudi
Domains	DevOps, Development_Environment
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for shutting down the Hudi demo cluster and cleaning up host resources provided by Apache Hudi Docker demo.

Description

The stop_demo.sh script is the counterpart to setup_demo.sh and provides a clean shutdown of the entire Hudi Docker demo environment. It performs three operations:

1. Detects the host CPU architecture and selects the corresponding Docker Compose file
2. Executes docker compose down against the compose file to stop and remove all 13 demo containers plus their networks
3. Removes the host-mounted directories /tmp/hadoop_data and /tmp/hadoop_name to eliminate stale HDFS data and metadata

The script uses the HUDI_WS environment variable (derived from the script's parent directory) to locate the compose file, mirroring the same pattern used in setup_demo.sh.

The companion script generate_test_suite.sh provides an optional pre-cleanup capability for generating and executing integration test suites within the demo environment. It supports sanity, medium, long, and clustering test configurations with parameterized iteration counts, delays, and table types.

Usage

Use this script to:

• Shut down the demo cluster after a demo session
• Free system resources (CPU, memory, ports) used by demo containers
• Clean up HDFS data directories before a fresh demo restart
• Prepare the environment for a different Hudi version or configuration

Code Reference

Source Location

• Repository: Apache Hudi
• File: docker/stop_demo.sh
• Lines: 19-33
• Additional File: docker/generate_test_suite.sh
• Lines: 19-301

Script

stop_demo.sh:

#!/bin/bash
SCRIPT_PATH=$(cd `dirname $0`; pwd)
HUDI_DEMO_ENV=$1
# set up root directory
WS_ROOT=`dirname $SCRIPT_PATH`
COMPOSE_FILE_NAME="docker-compose_hadoop334_hive313_spark353_amd64.yml"
if [ "$(uname -m)" = "arm64" ]; then
  COMPOSE_FILE_NAME="docker-compose_hadoop334_hive313_spark353_arm64.yml"
fi
# shut down cluster
HUDI_WS=${WS_ROOT} docker compose -f ${SCRIPT_PATH}/compose/${COMPOSE_FILE_NAME} down

# remove host mount directory
rm -rf /tmp/hadoop_data
rm -rf /tmp/hadoop_name

generate_test_suite.sh (key invocation pattern):

# Generate and run all test suites
./generate_test_suite.sh --all true

# Generate test suites without executing
./generate_test_suite.sh --all true --execute_test_suite false

# Run only medium test suite
./generate_test_suite.sh --execute_test_suite true --include_medium_test_suite_yaml true

# Key parameters:
#   --medium_num_iterations  (DEFAULT: 20)
#   --long_num_iterations    (DEFAULT: 50)
#   --intermittent_delay_mins (DEFAULT: 1)
#   --table_type             (DEFAULT: COPY_ON_WRITE)
#   --cluster_num_itr        (DEFAULT: 30)

I/O Contract

Inputs

Name	Type	Required	Description
Running demo containers	Docker containers	Yes	The 13 containers started by setup_demo.sh must be running (or at least exist in stopped state) for docker compose down to clean them up.
Architecture-specific compose file	YAML file	Yes	docker-compose_hadoop334_hive313_spark353_amd64.yml or docker-compose_hadoop334_hive313_spark353_arm64.yml in docker/compose/. Auto-selected based on host architecture.
/tmp/hadoop_data	Host directory	No	Host-mounted directory containing HDFS DataNode block data. May not exist if the demo was never started.
/tmp/hadoop_name	Host directory	No	Host-mounted directory containing HDFS NameNode metadata (fsimage, edit logs). May not exist if the demo was never started.
generate_test_suite.sh arguments	CLI arguments	No	Optional. Used only when running integration test suites before cleanup. Supports --all, --execute_test_suite, --medium_num_iterations, --long_num_iterations, --table_type, --include_cluster_yaml, and others.

Outputs

Name	Type	Description
All containers stopped/removed	Docker state change	All 13 demo containers are stopped (SIGTERM then SIGKILL) and removed from Docker. Container names are freed for reuse.
Docker network removed	Docker state change	The hudi bridge network created by docker compose up is removed.
/tmp/hadoop_data deleted	Host filesystem	The HDFS DataNode data directory is recursively removed from the host.
/tmp/hadoop_name deleted	Host filesystem	The HDFS NameNode metadata directory is recursively removed from the host.
Named volumes preserved	Docker volumes	Named volumes (namenode, historyserver, hive-metastore-postgresql, minio-data) are preserved unless -v is manually added.
Test suite files (optional)	Container filesystem + host	If generate_test_suite.sh is run before cleanup, test YAML files and Spark command scripts are generated in demo/config/test-suite/staging/.

Usage Examples

# Stop the demo and clean up
cd docker/
./stop_demo.sh

# Verify all containers are stopped
docker ps -a --filter "label=com.docker.compose.project"

# Verify host directories are cleaned
ls /tmp/hadoop_data 2>/dev/null || echo "hadoop_data cleaned"
ls /tmp/hadoop_name 2>/dev/null || echo "hadoop_name cleaned"

# For a complete cleanup including named volumes, run manually:
HUDI_WS=$(dirname $(pwd)) docker compose \
  -f compose/docker-compose_hadoop334_hive313_spark353_amd64.yml down -v

# Optional: Run integration tests before stopping
./generate_test_suite.sh --all true
# Then stop
./stop_demo.sh

# Optional: Generate test suites without executing them
./generate_test_suite.sh --all true --execute_test_suite false

# Optional: Run specific test suite with custom parameters
./generate_test_suite.sh \
  --include_medium_test_suite_yaml true \
  --medium_num_iterations 10 \
  --table_type MERGE_ON_READ \
  --execute_test_suite true

Related Pages

Implements Principle

• Principle:Apache_Hudi_Demo_Environment_Cleanup
• Environment:Apache_Hudi_Docker_Demo_Environment