Implementation:Apache Airflow MetricsRegistry

Knowledge Sources	Apache_Airflow
Domains	Observability, Metrics
Last Updated	2026-02-08 21:00 GMT

Overview

Manages the canonical registry of all Airflow metrics by loading metric definitions from metrics_template.yaml, providing lookup, RST documentation generation, and file output functions.

Description

The metrics_registry.py module serves as the single source of truth for all Airflow metric definitions. It reads metric metadata from a YAML file and provides programmatic access and documentation generation capabilities.

MetricsRegistry Class

The MetricsRegistry class provides in-memory storage and lookup for metric definitions:

• __init__(self) -- Reads the metrics from metrics_template.yaml (located alongside the module) and converts the list into a dictionary keyed by metric name for O(1) lookup.
• get(name: str) -> dict | None -- Retrieves a metric definition by its name. Returns a dictionary with fields such as name, legacy_name, description, type, and name_variables, or None if not found.

Module-Level Functions

• read_metrics_yaml(yaml_path: str) -> list -- Reads and parses a YAML file containing metric definitions. Returns the metrics list from the parsed YAML. Raises FileNotFoundError if the file does not exist.

• convert_to_rst_tables(metrics: list) -> str -- Converts a metrics list to RST-formatted tables, grouped by metric type (counters, gauges, timers). Each table has columns for Name, Legacy Name, and Description. Uses the tabulate library for RST table formatting.

• write_metric_tables_file(rst_tables_str, output_path, yaml_source) -- Writes the generated RST tables to a file, prepending the Apache license header and an auto-generation warning.

• generate_metrics_rst_from_registry() -- Convenience function that chains the full pipeline: reads the YAML registry, converts to RST tables, and writes the output file. Used during documentation builds.

Module-Level Constants

• YAML_REGISTRY_PATH -- The path to metrics_template.yaml, resolved relative to the module's directory.
• RST_TABLE_OUTPUT_PATH -- The default output path for the generated RST documentation file.

Usage

The MetricsRegistry is used by DualStatsManager to look up legacy metric names during dual emission. The RST generation functions are used during documentation builds to produce the metrics reference tables automatically from the YAML source of truth.

Code Reference

Source Location

• Repository: Apache_Airflow
• File: shared/observability/src/airflow_shared/observability/metrics/metrics_registry.py (165 lines)

Signature

YAML_REGISTRY_PATH = Path(__file__).parent / "metrics_template.yaml"
RST_TABLE_OUTPUT_PATH = "airflow-core/docs/administration-and-deployment/logging-monitoring/metric_tables.rst"


def read_metrics_yaml(yaml_path: str) -> list:
    """Read the metrics from a YAML registry file."""
    ...

def convert_to_rst_tables(metrics: list) -> str:
    """Convert a metrics list to RST tables, separated by type."""
    ...

def write_metric_tables_file(rst_tables_str: str, output_path: str, yaml_source: str):
    """Write the RST tables string to a file."""
    ...

def generate_metrics_rst_from_registry():
    """Full pipeline: read YAML, convert to RST, write file."""
    ...


class MetricsRegistry:
    """Class for storing and looking up metrics."""

    def __init__(self): ...
    def get(self, name: str) -> dict | None: ...

Import

from airflow_shared.observability.metrics.metrics_registry import MetricsRegistry
from airflow_shared.observability.metrics.metrics_registry import read_metrics_yaml, generate_metrics_rst_from_registry

I/O Contract

Component	Input	Output	Side Effects / Errors
MetricsRegistry.__init__	(none)	Initialized registry instance	Reads metrics_template.yaml from disk
MetricsRegistry.get	name: str	None	None
read_metrics_yaml	yaml_path: str	list of metric dicts	Raises FileNotFoundError if YAML file missing
convert_to_rst_tables	metrics: list	str (RST-formatted tables)	None
write_metric_tables_file	rst_tables_str: str, output_path: str, yaml_source: str	None	Writes file to output_path
generate_metrics_rst_from_registry	(none)	None	Reads YAML and writes RST file to RST_TABLE_OUTPUT_PATH

Usage Examples

Looking Up a Metric Definition

from airflow_shared.observability.metrics.metrics_registry import MetricsRegistry

registry = MetricsRegistry()
metric = registry.get("dag_processing.import_errors")
if metric:
    print(f"Name: {metric['name']}")
    print(f"Legacy: {metric.get('legacy_name', 'N/A')}")
    print(f"Type: {metric.get('type')}")
    print(f"Description: {metric['description']}")

Reading the YAML Registry Directly

from airflow_shared.observability.metrics.metrics_registry import read_metrics_yaml, YAML_REGISTRY_PATH

metrics = read_metrics_yaml(str(YAML_REGISTRY_PATH))
for m in metrics:
    print(f"{m['name']} ({m['type']}): {m['description']}")

Generating RST Documentation

from airflow_shared.observability.metrics.metrics_registry import generate_metrics_rst_from_registry

# Generates the metrics RST file during a documentation build
generate_metrics_rst_from_registry()

Related Pages

• Principle:Apache_Airflow_Monitoring_Observability