Heuristic:Astronomer Astronomer cosmos Memory Optimised Imports
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Performance |
| Last Updated | 2026-02-07 17:00 GMT |
Overview
Reduce Airflow worker memory footprint by enabling lazy imports, avoiding the overhead of loading all Cosmos classes at the top level.
Description
By default, cosmos/__init__.py eagerly imports and re-exports all major Cosmos classes (operators, configs, profiles, graphs, etc.) at the package level. This allows convenient usage like from cosmos import DbtDag but loads all modules into memory even when only a subset is needed. In large Airflow deployments where Cosmos is installed but only used by a few DAGs, this eager loading wastes memory across all worker processes.
The enable_memory_optimised_imports setting disables these eager imports, requiring users to import classes via their full module paths instead. This significantly reduces the memory overhead when Cosmos is installed but not actively used in all workers.
Usage
Use this heuristic when memory is constrained or when Cosmos is installed in a shared Airflow environment where only some DAGs use Cosmos. Enable it by setting [cosmos] enable_memory_optimised_imports = True in Airflow config.
The Insight (Rule of Thumb)
- Action: Set
enable_memory_optimised_imports = Truein the[cosmos]Airflow config section. - Value: Reduces per-worker memory usage by avoiding loading unused Cosmos modules.
- Trade-off: Users must change imports from
from cosmos import DbtDagtofrom cosmos.airflow.dag import DbtDag. - Default: Disabled (
False) for backwards compatibility.
Reasoning
From cosmos/settings.py:48-53:
# Eager imports in cosmos/__init__.py expose all Cosmos classes at the top level,
# which can significantly increase memory usage—even when Cosmos is installed but not
# actively used. This option allows disabling those eager imports to reduce memory
# footprint. When enabled, users must access Cosmos classes via their full module paths,
# avoiding the overhead of importing unused modules and classes.
enable_memory_optimised_imports = conf.getboolean(
"cosmos", "enable_memory_optimised_imports", fallback=False
)
In environments with 50+ DAGs where only 5 use Cosmos, the eager imports in every worker process load operator classes, profile mappings, and graph builders unnecessarily. The memory savings scale with the number of worker processes and the number of unused Cosmos modules.