Implementation:Apache Airflow BaseSecretsBackend

Knowledge Sources	Apache Airflow Airflow Secrets Backend
Domains	Security, Secrets_Management
Last Updated	2026-02-08 21:00 GMT

Overview

Abstract base class for secrets backends that defines the interface for retrieving connections, variables, and configuration values from external secret stores such as HashiCorp Vault, AWS SSM, GCP Secret Manager, and others.

Description

BaseSecretsBackend is the foundational ABC that all Airflow secrets backend implementations must extend. It establishes a uniform interface for:

• Connection retrieval: get_conn_value() returns a serialized connection string (URI or JSON), while get_connection() returns a deserialized Connection object. The deserialize_connection() method auto-detects JSON vs URI format and selects the appropriate Connection class based on execution context (SDK Connection for worker processes, core Connection for server processes).
• Variable retrieval: get_variable() returns a string value for a given variable key.
• Configuration retrieval: get_config() returns a string value for a given Airflow configuration key. The default implementation returns None (not all backends support config).
• Path construction: The static build_path() method constructs backend-specific secret paths by joining a prefix and secret ID with a configurable separator.

The class also supports team_name as an optional parameter for multi-tenant Airflow deployments, enabling team-scoped access to connections and variables.

Usage

Provider packages implement concrete subclasses (e.g., HashiCorpVaultBackend, SystemsManagerParameterStoreBackend) by overriding get_conn_value() and get_variable(). The backend is configured via the [secrets] section of airflow.cfg.

Code Reference

Source Location

• Repository: Apache Airflow
• File: shared/secrets_backend/src/airflow_shared/secrets_backend/base.py (121 lines)

Signature

class BaseSecretsBackend(ABC):
    """Abstract base class to retrieve Connection object given a conn_id
    or Variable given a key."""

    @staticmethod
    def build_path(path_prefix: str, secret_id: str, sep: str = "/") -> str:
        """Given conn_id, build path for Secrets Backend."""
        ...

    def get_conn_value(self, conn_id: str, team_name: str | None = None) -> str | None:
        """Retrieve serialized Connection string from Secrets Backend."""
        raise NotImplementedError

    def get_variable(self, key: str, team_name: str | None = None) -> str | None:
        """Return value for Airflow Variable."""
        raise NotImplementedError

    def get_config(self, key: str) -> str | None:
        """Return value for Airflow Config Key."""
        return None

    def deserialize_connection(self, conn_id: str, value: str):
        """Given a serialized representation, return a Connection instance.
        Auto-detects JSON vs URI format and SDK vs core Connection class."""
        ...

    def get_connection(self, conn_id: str, team_name: str | None = None):
        """Return deserialized Connection object with a given conn_id."""
        ...

    @staticmethod
    def _get_connection_class():
        """Detect which Connection class to use based on execution context."""
        ...

Import

from airflow_shared.secrets_backend.base import BaseSecretsBackend

I/O Contract

Core Methods

Method	Parameters	Return Type	Description
build_path	path_prefix: str, secret_id: str, sep: str="/"	str	Construct path: {path_prefix}{sep}{secret_id}
get_conn_value	None	str or None	Retrieve raw connection string (URI or JSON)
get_variable	None	str or None	Retrieve variable value by key
get_config	key: str	str or None	Retrieve Airflow config value (default: None)
get_connection	None	Connection or None	Retrieve and deserialize a Connection object
deserialize_connection	conn_id: str, value: str	Connection	Parse serialized connection into Connection instance

Connection Deserialization Logic

Format Detected	Parsing Method	Description
JSON (starts with {)	Connection.from_json(value, conn_id)	JSON-format connection string
URI (all other formats)	Connection.from_uri(conn_id, uri) or Connection(conn_id, uri=value)	URI-format connection string

Execution Context Detection

Environment Variable	Value	Connection Class Used
_AIRFLOW_PROCESS_CONTEXT	"client"	airflow.sdk.definitions.connection.Connection (worker/triggerer/dag processor)
_AIRFLOW_PROCESS_CONTEXT	(any other value or unset)	airflow.models.connection.Connection (scheduler/API server)

Usage Examples

Implementing a Custom Backend

from airflow_shared.secrets_backend.base import BaseSecretsBackend


class MyCustomBackend(BaseSecretsBackend):
    def __init__(self, prefix="airflow"):
        self.prefix = prefix

    def get_conn_value(self, conn_id: str, team_name: str | None = None) -> str | None:
        path = self.build_path(self.prefix, conn_id)
        # Retrieve from your custom secret store
        return my_store.get(path)

    def get_variable(self, key: str, team_name: str | None = None) -> str | None:
        path = self.build_path(self.prefix, key)
        return my_store.get(path)

Using build_path

from airflow_shared.secrets_backend.base import BaseSecretsBackend

path = BaseSecretsBackend.build_path("airflow/connections", "my_conn_id")
# Returns: "airflow/connections/my_conn_id"

path = BaseSecretsBackend.build_path("secret", "db_password", sep=".")
# Returns: "secret.db_password"

Configuration

# airflow.cfg
[secrets]
backend = my_package.MyCustomBackend
backend_kwargs = {"prefix": "airflow/prod"}

Related Pages

Implements Principle

• Principle:Apache_Airflow_Production_Hardening

Requires Environment

• Environment:Apache_Airflow_Python_Runtime_Environment