Principle:Unstructured IO Unstructured Embedding Provider Interface
| Knowledge Sources | |
|---|---|
| Domains | NLP, RAG, Embeddings |
| Last Updated | 2026-02-12 00:00 GMT |
Overview
An abstract interface that defines how document elements are converted to vector embeddings through pluggable embedding providers.
Description
The embedding provider interface establishes a contract for integrating multiple embedding services (OpenAI, AWS Bedrock, HuggingFace, Google Vertex AI, Voyage AI, etc.) into the Unstructured pipeline. By defining a common base class with standard methods, new providers can be added without modifying the chunking or partition pipeline.
The interface requires each provider to implement:
- initialize: Set up the client connection
- embed_documents: Convert a list of elements to elements with embedding vectors
- embed_query: Convert a single query string to an embedding vector
- num_of_dimensions: Report the embedding vector dimensionality
- is_unit_vector: Report whether embeddings are L2-normalized
This abstraction decouples the document processing pipeline from any specific embedding service, enabling users to switch providers by changing configuration.
Usage
Use this principle when selecting or implementing an embedding provider for a RAG pipeline. Understanding the interface is essential for choosing the right provider (based on dimensionality, unit vector normalization, and cost) and for implementing custom providers that integrate with the Unstructured ecosystem.
Theoretical Basis
Vector embeddings map text to dense numerical representations in a high-dimensional space where semantic similarity corresponds to geometric proximity (cosine similarity or dot product).
The provider pattern: Different embedding services have different APIs, authentication mechanisms, and model characteristics. The provider interface normalizes these differences behind a uniform contract:
# Abstract provider interface
class EmbeddingProvider:
def initialize(config):
"""Set up authenticated client."""
def embed_documents(elements) -> elements_with_embeddings:
"""Convert element text to vectors, store in element.embeddings."""
def embed_query(query_text) -> vector:
"""Convert query text to a vector for similarity search."""
def dimensions -> int:
"""Report vector dimensionality (e.g., 1536 for ada-002)."""
Key design decision: The embed_documents method takes Element objects (not raw strings) and returns the same elements with the embeddings field populated. This preserves metadata and enables the embedding step to be a transparent pass-through in the pipeline.