Implementation:Spotify Luigi OpenerTarget

Knowledge Sources	Spotify_Luigi
Domains	File_System, URI_Resolution
Last Updated	2026-02-10 08:00 GMT

Overview

OpenerTarget is a Luigi contrib facility that resolves URI strings into the appropriate Luigi target types (local files, S3, mock). It provides a registry-based URI abstraction layer, making it easier to write tasks that work transparently across different storage backends and simplifying testing by allowing URIs to be swapped between real and mock implementations.

Description

The module provides the following key components:

• OpenerRegistry: The central registry that maps URI schemes to Opener implementations. It parses URIs using urllib.parse.urlsplit and delegates target creation to the appropriate opener.
  • __init__(self, openers=None): Initializes with a list of Opener classes. Default opener scheme is 'file'.
  • open(self, target_uri, **kwargs): Parses the URI, resolves the scheme to an opener, conforms query parameters, and returns the constructed target. Also sets target.opener_path to the original URI.
  • get_opener(self, name): Retrieves an opener by scheme name. Raises NoOpenerError if not found.
  • add(self, opener): Registers an opener and all its scheme names.

• Opener: Abstract base class for openers. Subclasses define names (list of URI schemes), allowed_kwargs, and implement get_target().
  • conform_query(cls, query): Parses and validates query string parameters against allowed_kwargs. JSON-loads values for kwargs marked as True.
  • get_target(cls, scheme, path, fragment, username, password, hostname, port, query, **kwargs): Abstract classmethod that constructs and returns the target.

• MockOpener: Opens mock:// URIs as MockTarget instances. Supports is_tmp, mirror_on_stderr, and format query parameters.

• LocalOpener: Opens file:// URIs (or bare paths) as LocalTarget instances. This is the default opener. Supports is_tmp and format query parameters.

• S3Opener: Opens s3:// and s3n:// URIs as S3Target instances. Does not filter unexpected kwargs (filter_kwargs = False), allowing pass-through of AWS credentials and other S3-specific parameters.

• opener: Module-level OpenerRegistry instance pre-configured with MockOpener, LocalOpener, and S3Opener.

• OpenerTarget: Module-level alias for opener.open. This is the primary public interface.

Exception classes:

• OpenerError: Base exception (extends FileSystemException).
• NoOpenerError: No opener registered for the given scheme.
• InvalidQuery: Unexpected query arguments for the opener.

Usage

Call OpenerTarget(uri) with any supported URI to get back the appropriate target. Use mock:// URIs in tests to avoid real I/O.

Code Reference

Source Location

luigi/contrib/opener.py (274 lines)

Signature

class OpenerRegistry:
    def __init__(self, openers=None): ...
    def open(self, target_uri, **kwargs): ...
    def get_opener(self, name): ...
    def add(self, opener): ...

class Opener:
    names = []           # List of URI schemes
    allowed_kwargs = {}  # Dict of allowed query params; True = JSON-load value
    filter_kwargs = True # Filter out unexpected kwargs

    @classmethod
    def conform_query(cls, query): ...

    @classmethod
    def get_target(cls, scheme, path, fragment, username, password, hostname, port, query, **kwargs):
        raise NotImplementedError

class MockOpener(Opener):
    names = ['mock']

class LocalOpener(Opener):
    names = ['file']

class S3Opener(Opener):
    names = ['s3', 's3n']

opener = OpenerRegistry([MockOpener, LocalOpener, S3Opener])
OpenerTarget = opener.open

Import

from luigi.contrib.opener import OpenerTarget

I/O Contract

Inputs

Input	Type	Description
target_uri	str	A URI string. Scheme determines which opener is used: file:// (default), s3://, s3n://, mock://. Bare paths default to file://.
**kwargs	keyword args	Additional arguments forwarded to the opener's get_target() method and merged with query parameters.

Outputs

Output	Type	Description
Return value	Target	A Luigi target instance: LocalTarget, S3Target, or MockTarget depending on the URI scheme. The target has an additional opener_path attribute set to the original URI.

Usage Examples

from luigi.contrib.opener import OpenerTarget

# Local file (default scheme)
local_target = OpenerTarget('/data/output/results.csv')

# Explicit file scheme
local_target = OpenerTarget('file:///data/output/results.csv')

# S3 target
s3_target = OpenerTarget('s3://my-bucket/data/results.parquet')

# S3 with credentials in query string
s3_target = OpenerTarget(
    's3://my-bucket/data/results.parquet?aws_access_key_id=XXX&aws_secret_access_key=YYY'
)

# Mock target for testing
mock_target = OpenerTarget('mock://test/output.txt')

# Using in a task
class FlexibleOutputTask(luigi.Task):
    output_uri = luigi.Parameter()

    def output(self):
        return OpenerTarget(self.output_uri)

    def run(self):
        with self.output().open('w') as f:
            f.write('hello world')

Related Pages

• Spotify_Luigi_URI_Target_Resolution -- Principle governing URI-based target resolution
• luigi.local_target.LocalTarget -- Target returned for file:// URIs
• luigi.contrib.s3.S3Target -- Target returned for s3:// URIs
• luigi.mock.MockTarget -- Target returned for mock:// URIs
• luigi.target.FileSystemException -- Base exception class for opener errors