Environment:Langgenius Dify Redis And Celery Environment

Knowledge Sources	Dify Celery
Domains	Infrastructure, Async_Processing
Last Updated	2026-02-08 11:00 GMT

Overview

Redis 6 cache and message broker environment with Celery 5.5 async task processing for background document indexing, workflow execution, and scheduled maintenance tasks.

Description

This environment defines the Redis and Celery infrastructure required for Dify's asynchronous processing. Redis serves three roles: caching (db 0), Celery message broker (db 1), and pub/sub for real-time SSE event delivery. Celery runs as a separate worker process (and beat scheduler) that shares the same codebase as the API server but operates with Flask application context injection via a custom `FlaskTask` class. The system supports Redis Sentinel and Redis Cluster modes for high availability.

Usage

Use this environment for any asynchronous operations in Dify, including document indexing, workflow background execution, plugin lifecycle operations, and scheduled cleanup tasks. It is an implicit dependency of all implementations that trigger background processing.

System Requirements

Category	Requirement	Notes
Redis	6.x (Alpine image)	Password authentication required
RAM	1GB+ for Redis	Scales with cache size and message queue depth
Network	Port 6379 (Redis)	Internal Docker network only by default

Dependencies

Infrastructure

`redis:6-alpine` Docker image
Redis password authentication enabled

Python Packages

`celery` ~= 5.5.2
`redis[hiredis]` ~= 6.1.0
`gevent` ~= 25.9.1 (for async worker class)

Credentials

`REDIS_PASSWORD`: Redis authentication password (default: difyai123456).
`CELERY_BROKER_URL`: Full Redis connection URL (default: redis://:difyai123456@redis:6379/1).
`REDIS_SENTINEL_PASSWORD`: Sentinel password (optional, for HA setups).
`REDIS_CLUSTERS_PASSWORD`: Cluster password (optional, for cluster mode).

Quick Install

# Redis is included in the Docker Compose stack
docker compose up -d redis

# Verify Redis is running
docker compose exec redis redis-cli -a difyai123456 ping
# Expected: PONG

Code Evidence

Celery Flask integration from `api/extensions/ext_celery.py:46-54`:

class FlaskTask(Task):
    def __call__(self, *args, **kwargs):
        with flask_app.app_context():
            init_request_context()
            return self.run(*args, **kwargs)

Redis health check from `docker/docker-compose.yaml`:

redis:
    image: redis:6-alpine
    healthcheck:
      test: redis-cli -a ${REDIS_PASSWORD} ping | grep PONG

Redis PubSub thread safety from `api/libs/broadcast_channel/redis/_subscription.py:199-206`:

# NOTE: PubSub close() method and message retrieval
# MUST NOT be called concurrently. Always join listener
# threads before returning.

Common Errors

Error Message	Cause	Solution
`ConnectionError: Error 111 connecting to redis:6379`	Redis not running or wrong host	Verify Redis container health: `docker compose ps redis`
`AuthenticationError: NOAUTH`	Missing or wrong Redis password	Check `REDIS_PASSWORD` matches in `.env`
Celery tasks stuck in PENDING	Worker not running or broker connection failed	Start worker: `docker compose up -d worker`
`kombu.exceptions.OperationalError`	Celery broker URL invalid	Verify `CELERY_BROKER_URL` format: `redis://:password@host:port/db`

Compatibility Notes

Redis Sentinel: Supported via `REDIS_USE_SENTINEL=true` with sentinel host list.
Redis Cluster: Supported via `REDIS_USE_CLUSTERS=true` with cluster node list.
SSL/TLS: Configurable via `REDIS_USE_SSL=true` with certificate paths.
Celery Worker Class: Defaults to empty (auto-selects); can be set to `solo` for debugging.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment