Environment:Dagster io Dagster Container Resource Monitoring

Knowledge Sources	dagster container utils
Domains	Infrastructure, Monitoring
Last Updated	2026-02-10 12:00 GMT

Overview

Linux cgroup-based container resource monitoring environment for CPU and memory metrics in Docker, Kubernetes, and ECS.

Description

Dagster includes built-in container resource monitoring that reads CPU and memory utilization directly from Linux cgroup files rather than using psutil. This is critical because psutil reports host machine resources in containerized environments, not the container's allocated resources. The monitoring supports both cgroup v1 (older Docker/K8s) and cgroup v2 (modern systems) formats, with configurable paths via environment variables.

Usage

Use this environment when running Dagster in containerized deployments (Docker, Kubernetes, ECS, etc.) where accurate resource monitoring is needed. The run metrics thread uses these cgroup paths to report actual container-level CPU and memory usage for monitoring and alerting.

System Requirements

Category	Requirement	Notes
OS	Linux with cgroup support	Docker, Kubernetes, ECS, or bare-metal with cgroups enabled
Filesystem	Access to /sys/fs/cgroup/ and /proc/cpuinfo	Must be mounted (default in most container runtimes)

Dependencies

No additional Python packages are required beyond the Dagster core. The monitoring reads directly from the Linux filesystem.

Credentials

The following environment variables override default cgroup paths:

CPU Monitoring (cgroup v1):

• DAGSTER_CPU_USAGE_PATH: CPU time in nanoseconds (default: /sys/fs/cgroup/cpuacct/cpuacct.usage)
• DAGSTER_CPU_CFS_QUOTA_US_PATH: CPU quota in microseconds (default: /sys/fs/cgroup/cpu/cpu.cfs_quota_us)
• DAGSTER_CPU_CFS_PERIOD_US_PATH: CPU period in microseconds (default: /sys/fs/cgroup/cpu/cpu.cfs_period_us)
• DAGSTER_CPU_SHARES_PATH: CPU shares (default: /sys/fs/cgroup/cpu/cpu.shares)

CPU Monitoring (cgroup v2):

• DAGSTER_CPU_STAT_PATH: CPU stats in microseconds (default: /sys/fs/cgroup/cpu.stat)
• DAGSTER_CPU_MAX_PATH: Max CPU quota per period (default: /sys/fs/cgroup/cpu.max)

Memory Monitoring:

• DAGSTER_MEMORY_USAGE_PATH_V1: Memory usage bytes, cgroup v1 (default: /sys/fs/cgroup/memory/memory.usage_in_bytes)
• DAGSTER_MEMORY_USAGE_PATH_V2: Memory usage bytes, cgroup v2 (default: /sys/fs/cgroup/memory.current)
• DAGSTER_MEMORY_LIMIT_PATH_V1: Memory limit, cgroup v1 (default: /sys/fs/cgroup/memory/memory.limit_in_bytes)
• DAGSTER_MEMORY_LIMIT_PATH_V2: Memory limit, cgroup v2 (default: /sys/fs/cgroup/memory.max)

General:

• DAGSTER_CPU_INFO_PATH: CPU core info (default: /proc/cpuinfo)

Quick Install

# No additional installation needed - built into dagster core
# Verify cgroup support in your container:
cat /sys/fs/cgroup/memory.current 2>/dev/null || cat /sys/fs/cgroup/memory/memory.usage_in_bytes 2>/dev/null

Code Evidence

Cgroup path functions with environment variable overrides from container.py:17-99:

def cpu_usage_path_cgroup_v1() -> str:
    """Path to the cgroup file containing the CPU time in nanoseconds.
    We use cgroup files instead of the psutil library because psutil uses
    the host machine's CPU allocation in virtualized environments like
    Docker, K8s, ECS, etc.
    """
    return os.getenv("DAGSTER_CPU_USAGE_PATH",
                      "/sys/fs/cgroup/cpuacct/cpuacct.usage")

Unconstrained memory detection from container.py:8-10:

# When cgroup memory is not constrained, the limit value will be a high
# value close to 2^63 - 1, Ex: AWS ECS returns 2^63 - 2^10
UNCONSTRAINED_CGROUP_MEMORY_LIMIT = 9223372036854000000

Common Errors

Error Message	Cause	Solution
FileNotFoundError: /sys/fs/cgroup/...	Container runtime doesn't mount cgroup filesystem	Ensure cgroup filesystem is mounted; check container security settings
Memory limit shows very large number (~9.2e18)	Cgroup memory is unconstrained	Set container memory limits or ignore (Dagster detects this threshold)

Compatibility Notes

• cgroup v1 vs v2: Dagster auto-detects the cgroup version and reads from appropriate paths. Most modern Linux distributions use cgroup v2.
• macOS/Windows: Container resource monitoring is Linux-only. Not available on macOS or Windows host systems.
• AWS ECS: Returns 2^63 - 2^10 as the memory limit when unconstrained. Dagster uses UNCONSTRAINED_CGROUP_MEMORY_LIMIT to detect this.
• Custom paths: Use the DAGSTER_* environment variables to override cgroup paths in non-standard container configurations.

Related Pages

• Implementation:Dagster_io_Dagster_ML_Pipeline_Assets