Implementation:Spotify Luigi RedisTarget

Overview

RedisTarget is a Luigi target class in the luigi.contrib.redis_store module that uses Redis as a marker-based completion store. Rather than storing actual task output data, it records a hash entry in Redis keyed by a configurable prefix and an update_id. This follows the same marker pattern used by Luigi's database targets, where task completion is tracked through a side-channel indicator rather than the output data itself.

Source Location

Property	Value
Source File	`luigi/contrib/redis_store.py`
Lines of Code	102
Module	`luigi.contrib.redis_store`
Domain	Database, Caching

Import Statement

from luigi.contrib.redis_store import RedisTarget

Class: RedisTarget

RedisTarget(Target)

A Luigi target that uses Redis hashes as completion markers.

Class Attributes

Attribute	Type	Default	Description
`marker_prefix`	`Parameter`	`'luigi'`	The prefix used in Redis key construction. Configurable via the `[redis]` section, key `marker-prefix` in `luigi.cfg`.

Constructor

RedisTarget.__init__(self, host, port, db, update_id, password=None, socket_timeout=None, expire=None)

Parameter	Type	Default	Description
`host`	`str`	(required)	Redis server hostname.
`port`	`int`	(required)	Redis server port number.
`db`	`int`	(required)	Redis database index.
`update_id`	`str`	(required)	A unique identifier for this data marker. Used to construct the Redis key and stored as a hash field.
`password`	`str`	`None`	Optional password for Redis authentication.
`socket_timeout`	`int`	`None`	Optional client socket timeout in seconds.
`expire`	`int`	`None`	Optional TTL in seconds. If set, the marker key will expire after this duration.

The constructor creates a redis.StrictRedis client instance stored as self.redis_client.

Methods

Method	Signature	Return Type	Description
`marker_key`	`marker_key(self)`	`str`	Returns the Redis key in the format `'{marker_prefix}:{update_id}'`. For example, with defaults: `'luigi:my_task_20240101'`.
`touch`	`touch(self)`	`None`	Marks the task as complete by setting two hash fields on the marker key: `update_id` (the update identifier) and `date` (current ISO timestamp). If `expire` is set, applies a TTL to the key via `redis_client.expire()`.
`exists`	`exists(self)`	`bool`	Returns `True` if the marker key exists in Redis (using `redis_client.exists() == 1`), `False` otherwise.
`__str__`	`__str__(self)`	`str`	Returns the marker key string.

Redis Key Structure

When touch() is called, the following Redis hash is created:

Key: {marker_prefix}:{update_id}
Fields:
  update_id  -> "{update_id}"
  date       -> "2024-01-15T10:30:00.123456"

If expire is set, the key will have a TTL applied.

Configuration

[redis]
marker-prefix: luigi

Usage Example

from luigi.contrib.redis_store import RedisTarget
import luigi

class LoadToRedis(luigi.Task):
    date = luigi.DateParameter()

    def run(self):
        # Perform the actual data loading work
        self.do_load()

        # Mark the task as complete in Redis
        self.output().touch()

    def output(self):
        return RedisTarget(
            host='redis.example.com',
            port=6379,
            db=0,
            update_id='load_to_redis_%s' % self.date,
            password='secret',
            expire=86400 * 7  # expire marker after 7 days
        )

Marker Pattern

The RedisTarget follows Luigi's marker-based completion pattern, which is also used by PostgresTarget, MySqlTarget, and other database targets. The key characteristics are:

Task completion is tracked separately from the actual task output.
The update_id serves as a unique identifier, typically derived from the task parameters.
The touch() method is called explicitly by the task's run() method after successful execution.
The exists() method checks only whether the marker is present, not whether the actual output data is valid.
Optional TTL support via expire allows automatic cleanup of stale markers.

External Dependencies

redis: The redis Python package (specifically redis.StrictRedis). A warning is logged if not installed; the module will crash at runtime if Redis functionality is used without it.
Luigi core: luigi.target.Target, luigi.parameter.Parameter

Related Principles

Spotify_Luigi_Database_Load_Verification

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment