Principle:Infiniflow Ragflow Task Queue Consumption
| Knowledge Sources | |
|---|---|
| Domains | RAG, Distributed_Systems, Task_Queue |
| Last Updated | 2026-02-12 06:00 GMT |
Overview
A distributed task consumption pattern using Redis consumer groups to dequeue and hydrate document processing tasks across multiple workers.
Description
Task Queue Consumption is the entry point of the document processing pipeline where background worker processes consume task messages from a Redis stream using consumer groups. Each task message contains a task ID; the worker then hydrates the full task context by joining Task, Document, Knowledgebase, and Tenant records from MySQL. This provides all necessary context (parser config, language, embedding model ID, storage location) for processing. Tasks with retry_count >= 3 are skipped.
Usage
This principle operates automatically in background worker processes (task_executor.py). Workers are started by the entrypoint script and continuously poll the Redis queue.
Theoretical Basis
Uses Redis consumer groups for reliable message delivery:
- At-least-once delivery: Messages are acknowledged only after successful processing
- Consumer groups: Multiple workers share the workload without duplicate processing
- Retry handling: Failed tasks increment retry_count; after 3 failures, tasks are abandoned