Implementation:BerriAI Litellm RAG Types

Attribute	Value
Sources	litellm/types/rag.py
Domains	RAG (Retrieval Augmented Generation), Ingest Pipeline, Vector Stores, Embeddings, Reranking
Last Updated	2026-02-15 16:00 GMT

Overview

Type definitions for LiteLLM's RAG (Retrieval Augmented Generation) ingest and query APIs, supporting OpenAI, Bedrock, Vertex AI, and S3 Vectors as vector store backends.

Description

This module provides the complete type hierarchy for LiteLLM's unified RAG pipeline. It covers both the ingest (document-to-vector) and query (retrieval-augmented generation) flows. Key type groups include:

• Chunking and preprocessing -- RAGChunkingStrategy (RecursiveCharacterTextSplitter config), RAGIngestOCROptions (OCR model config), RAGIngestEmbeddingOptions (embedding model config).
• Vector store backends -- Provider-specific TypedDicts for each supported backend:
  • OpenAIVectorStoreOptions -- OpenAI vector stores with optional auto-creation and TTL.
  • BedrockVectorStoreOptions -- AWS Bedrock Knowledge Bases with S3, OpenSearch, and IAM auto-creation.
  • VertexAIVectorStoreOptions -- Vertex AI RAG Engine with GCS bucket uploads and corpus imports.
  • S3VectorsVectorStoreOptions -- AWS S3 Vectors with auto-created buckets and indices.
• Union type -- RAGIngestVectorStoreOptions unifying all backend options.
• Pipeline config -- RAGIngestOptions combining OCR, chunking, embedding, and vector store settings.
• Request/Response models -- RAGIngestRequest/RAGIngestResponse for the ingest API, RAGQueryRequest/RAGQueryResponse for the query API.
• Query configuration -- RAGRetrievalConfig (vector store search params) and RAGRerankConfig (reranking model params).

Usage

Import from this module when:

• Configuring a RAG ingest pipeline to index documents into a vector store.
• Building RAG query requests that combine retrieval with LLM generation.
• Implementing custom ingest handlers for specific vector store backends.
• Working with the RAG API endpoints on the LiteLLM proxy.

Code Reference

Source Location

litellm/types/rag.py (271 lines)

Key Types

Type Name	Kind	Description
RAGChunkingStrategy	TypedDict	Chunking config: chunk_size, chunk_overlap, separators
RAGIngestOCROptions	TypedDict	OCR step config with model name
RAGIngestEmbeddingOptions	TypedDict	Embedding step config with model name
OpenAIVectorStoreOptions	TypedDict	OpenAI vector store config (auto-create, TTL, credentials)
BedrockVectorStoreOptions	TypedDict	Bedrock KB config (S3, embedding model, AWS auth, ingestion wait)
VertexAIVectorStoreOptions	TypedDict	Vertex AI RAG Engine config (GCS bucket, corpus ID, import settings)
S3VectorsVectorStoreOptions	TypedDict	S3 Vectors config (bucket name, index, dimension, distance metric)
RAGIngestVectorStoreOptions	Union	Union of all vector store backend options
RAGIngestOptions	TypedDict	Full pipeline config: name, ocr, chunking, embedding, vector_store
RAGIngestResponse	TypedDict	Ingest result: id, status, vector_store_id, file_id, error
RAGIngestRequest	BaseModel	Ingest API request: file_url or file_id, ingest_options
RAGRetrievalConfig	TypedDict	Retrieval config: vector_store_id, provider, top_k, filters
RAGRerankConfig	TypedDict	Rerank config: enabled, model, top_n, return_documents
RAGQueryRequest	BaseModel	Query API request: model, messages, retrieval_config, rerank, stream
RAGQueryResponse	ModelResponse	Query result (extends standard ModelResponse)

Import

from litellm.types.rag import (
    RAGChunkingStrategy,
    RAGIngestOCROptions,
    RAGIngestEmbeddingOptions,
    OpenAIVectorStoreOptions,
    BedrockVectorStoreOptions,
    VertexAIVectorStoreOptions,
    S3VectorsVectorStoreOptions,
    RAGIngestVectorStoreOptions,
    RAGIngestOptions,
    RAGIngestRequest,
    RAGIngestResponse,
    RAGRetrievalConfig,
    RAGRerankConfig,
    RAGQueryRequest,
    RAGQueryResponse,
)

I/O Contract

RAGIngestOptions (Pipeline Configuration)

Field	Type	Description
name	Optional[str]	Optional pipeline name for logging
ocr	Optional[RAGIngestOCROptions]	Optional OCR step (e.g., model: "mistral/mistral-ocr-latest")
chunking_strategy	Optional[RAGChunkingStrategy]	RecursiveCharacterTextSplitter args (chunk_size, chunk_overlap, separators)
embedding	Optional[RAGIngestEmbeddingOptions]	Embedding model config (e.g., model: "text-embedding-3-small")
vector_store	RAGIngestVectorStoreOptions	Vector store backend configuration (required)

RAGIngestRequest (Input)

Field	Type	Default	Description
file_url	Optional[str]	None	URL to fetch the file from
file_id	Optional[str]	None	Existing file ID to ingest
ingest_options	Dict[str, Any]	(required)	RAGIngestOptions as dict

RAGIngestResponse (Output)

Field	Type	Description
id	str	Unique ingest job ID
status	Literal["completed", "in_progress", "failed"]	Job status
vector_store_id	str	The vector store ID (created or existing)
file_id	Optional[str]	The file ID in the vector store
error	Optional[str]	Error message if status is "failed"

RAGQueryRequest (Input)

Field	Type	Default	Description
model	str	(required)	LLM model for generation
messages	List[Any]	(required)	Chat messages
retrieval_config	RAGRetrievalConfig	(required)	Vector store retrieval config
rerank	Optional[RAGRerankConfig]	None	Optional reranking configuration
stream	Optional[bool]	False	Whether to stream the response

Usage Examples

OpenAI vector store ingest

from litellm.types.rag import RAGIngestOptions, OpenAIVectorStoreOptions

options: RAGIngestOptions = {
    "chunking_strategy": {"chunk_size": 1000, "chunk_overlap": 200},
    "embedding": {"model": "text-embedding-3-small"},
    "vector_store": {
        "custom_llm_provider": "openai",
        "vector_store_id": "vs_abc123",
    },
}

Bedrock Knowledge Base ingest with auto-creation

from litellm.types.rag import RAGIngestOptions, BedrockVectorStoreOptions

options: RAGIngestOptions = {
    "ocr": {"model": "mistral/mistral-ocr-latest"},
    "vector_store": {
        "custom_llm_provider": "bedrock",
        "aws_region_name": "us-east-1",
        "wait_for_ingestion": True,
        "ingestion_timeout": 600,
    },
}

RAG query with reranking

from litellm.types.rag import RAGQueryRequest, RAGRetrievalConfig, RAGRerankConfig

query = RAGQueryRequest(
    model="gpt-4",
    messages=[{"role": "user", "content": "What is LiteLLM?"}],
    retrieval_config={
        "vector_store_id": "vs_abc123",
        "custom_llm_provider": "openai",
        "top_k": 10,
    },
    rerank={
        "enabled": True,
        "model": "cohere/rerank-english-v3.0",
        "top_n": 3,
        "return_documents": True,
    },
    stream=False,
)

S3 Vectors ingest

from litellm.types.rag import RAGIngestOptions, S3VectorsVectorStoreOptions

options: RAGIngestOptions = {
    "embedding": {"model": "text-embedding-3-small"},
    "vector_store": {
        "custom_llm_provider": "s3_vectors",
        "vector_bucket_name": "my-embeddings",
        "dimension": 1536,
        "distance_metric": "cosine",
    },
}

Related Pages

• Embedding Types -- Embedding request types used in the RAG ingest pipeline.
• Rerank Types -- Reranking types used for post-retrieval result refinement.
• Vector Store Types -- Vector store CRUD and search types for the underlying storage layer.
• Vector Store File Types -- File management types for vector store file operations.