Implementation:Infiniflow Ragflow Embedding Function

Knowledge Sources	RAGFlow
Domains	RAG, NLP, Machine_Learning
Last Updated	2026-02-12 06:00 GMT

Overview

Concrete tool for generating text embeddings via pluggable model providers provided by RAGFlow's embedding function and LLMBundle.

Description

The embedding() function in task_executor.py processes chunk dictionaries by calling mdl.encode(texts) on the configured embedding model (resolved via LLMBundle). It handles batching, title/content weighting, and stores results as q_{dim}_vec keys in each chunk dictionary. The base Base.encode interface is implemented by 30+ provider classes (OpenAIEmbed, CohereEmbed, HuggingFaceEmbed, etc.).

Usage

Called automatically during document processing after text chunking. The embedding model is resolved from the tenant's configured embd_id.

Code Reference

Source Location

Repository: ragflow
File: rag/svr/task_executor.py (embedding: L570-621), rag/llm/embedding_model.py (Base: L46-47, OpenAIEmbed: L94-122)

Signature

async def embedding(
    docs: list[dict],
    mdl,
    parser_config: dict = None,
    callback: callable = None
) -> tuple[int, int]:
    """Generate embeddings for document chunks.

    Args:
        docs: list[dict] - Chunk dicts to embed.
        mdl: EmbeddingModel - Embedding model instance (LLMBundle).
        parser_config: dict - Optional parser config.
        callback: callable - Progress callback.

    Returns:
        tuple[int, int] - (token_count, vector_size).
        Side effect: each chunk gets 'q_{dim}_vec' key.
    """

class Base:
    def encode(self, texts: list) -> tuple[np.ndarray, int]:
        """Encode texts to embeddings.
        Returns: (embeddings array, token_count)
        """
        raise NotImplementedError

class OpenAIEmbed(Base):
    def __init__(self, key: str, model_name: str = "text-embedding-ada-002",
                 base_url: str = "https://api.openai.com/v1"):
        ...
    def encode(self, texts: list) -> tuple[np.ndarray, int]:
        ...

Import

from rag.svr.task_executor import embedding
from rag.llm.embedding_model import Base, OpenAIEmbed
from api.db.services.llm_service import LLMBundle

I/O Contract

Inputs

Name	Type	Required	Description
docs	list[dict]	Yes	Chunk dictionaries from parsing
mdl	LLMBundle	Yes	Embedding model
parser_config	dict	No	Parser configuration
callback	callable	No	Progress callback

Outputs

Name	Type	Description
token_count	int	Total tokens processed
vector_size	int	Embedding dimensionality
(side effect)	dict mutation	Each chunk gets q_{dim}_vec key with float32 vector

Usage Examples

from api.db.services.llm_service import LLMBundle
from common.constants import LLMType

# Initialize embedding model
mdl = LLMBundle(tenant_id, LLMType.EMBEDDING, llm_name=embd_id, lang=language)

# Embed chunks
tk_count, vec_size = await embedding(chunks, mdl, parser_config, callback=set_progress)
print(f"Embedded {len(chunks)} chunks: {tk_count} tokens, {vec_size}D vectors")

Related Pages

Implements Principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment