Environment:Apache Beam Java Build Environment

Knowledge Sources	Apache Beam Contributing Guide Local Env Setup
Domains	Infrastructure, Build_System
Last Updated	2026-02-09 04:00 GMT

Overview

Multi-language build environment requiring Java 8+ (11 preferred), Python 3.10-3.13, Go 1.x, and Gradle with at least 2GB RAM for building Apache Beam from source.

Description

This environment provides the complete development and build context for Apache Beam. The project uses Gradle as its build system with a custom BeamModulePlugin that manages cross-module dependencies and version constraints. The default Java compilation target is Java 8 (for SDK compatibility), but the build toolchain and certain runners (e.g., Flink 2.x) require Java 11 or higher. Python SDKs require Python 3.10-3.13 with virtual environments. Go SDKs require the latest Go 1.x release. The Gradle daemon is explicitly disabled to reduce memory pressure during CI builds.

Usage

Use this environment for any local development, building from source, or running integration tests against the Apache Beam repository. It is the mandatory prerequisite for building all runners (Direct, Dataflow, Flink, Spark, Twister2) and SDKs (Java, Python, Go).

System Requirements

Category	Requirement	Notes
OS	Linux (Debian-based) or macOS (Darwin)	Windows not supported by setup script
Memory	Minimum 2GB RAM	Docker build env checks for 2046755 KiB minimum
JVM Stack	10MB thread stack size	Gradle JVM args: `-Xss10240k`
Disk	Sufficient for Gradle caches	Parallel build generates significant intermediate artifacts

Dependencies

System Packages

`openjdk` 8 or 11 (11 preferred for development; 8 is compilation target)
`python3` >= 3.10 (versions 3.10, 3.11, 3.12, 3.13 all supported)
`python3-venv` for each Python version
`go` >= 1.x (latest recommended)
`docker` >= 1.4.1
`git-lfs`

Build Tool Packages

`gradle` (wrapper included in repository)
`tox` (for Python testing)
`grpcio-tools` (Python protobuf compilation)
`mypy-protobuf` (Python type stubs for protobuf)
`goavro` (Go Avro support)
`setuptools` (Python packaging)

Gradle Configuration

`org.gradle.parallel=true`
`org.gradle.daemon=false`
`org.gradle.configureondemand=true`
`org.gradle.jvmargs=-Xss10240k`
`org.gradle.vfs.watch=true`

Credentials

No credentials required for local builds. For integration tests:

`CLOUDSDK_COMPUTE_REGION`: GCP default region (optional, for Dataflow runner tests)
Google Cloud SDK authentication (for Dataflow integration tests)

Quick Install

# Run the automated setup script (Linux/macOS)
./local-env-setup.sh

# Or manually install core dependencies:
# Linux (Debian):
apt-get update && apt-get install -y openjdk-11-jdk python3 python3-venv python3-pip
pip3 install grpcio-tools mypy-protobuf

# macOS:
brew install openjdk@8 python@3.12 go tox docker
pip3 install setuptools grpcio-tools mypy-protobuf

Code Evidence

Java version configuration from `gradle.properties:36`:

javaVersion=1.8

Supported Python versions from `gradle.properties:44`:

python_versions=3.10,3.11,3.12,3.13

Gradle JVM arguments from `gradle.properties:22`:

org.gradle.jvmargs=-Xss10240k

Memory check from `dev-support/docker/beam_env_checks.sh:86-88`:

MINIMAL_MEMORY=2046755
INSTALLED_MEMORY=$(grep -F MemTotal /proc/meminfo | awk '{print $2}')
if [[ $((INSTALLED_MEMORY)) -lt $((MINIMAL_MEMORY)) ]]; then

Root user guard from `dev-support/docker/beam_env_checks.sh:59-81`:

function failIfUserIsRoot {
    if [ "$(id -u)" -eq "0" ]; then
        # Known possible causes:
        # 1) Running this script as the root user
        # 2) Running an old docker version (upgrade to 1.4.1 or higher)

Common Errors

Error Message	Cause	Solution
`Apparently you are inside this docker container as the user root`	Running Docker build as root or Docker < 1.4.1	Run as non-root user or upgrade Docker to >= 1.4.1
`Your system is running on very little memory`	System has < 2GB RAM	Increase available memory to at least 2GB
`Python3 and pip3 are required but failed to install`	Python 3 not found on PATH	Install Python 3.10+ manually and rerun setup
`Go is required`	Go not found on PATH	Install Go from https://golang.org/doc/install
`StackOverflowError` during Gradle build	Thread stack too small	Ensure `-Xss10240k` is set in `gradle.properties`

Compatibility Notes

Linux: Assumes Debian-based distribution (uses `apt-get`). Other distributions require manual dependency installation.
macOS: Requires Homebrew for package management. Intel and Apple Silicon both supported.
Windows: Not supported by the automated setup script. Contributors should use WSL2.
Docker: Version 1.4.1 or higher required for non-root execution inside containers.
Flink 2.x Runners: Require Java 11 or higher (overrides the default Java 8 target).

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment