Principle:Unstructured IO Unstructured OpenAI Embedding
| Knowledge Sources | |
|---|---|
| Domains | NLP, RAG, Embeddings |
| Last Updated | 2026-02-12 00:00 GMT |
Overview
A vector embedding generation process that converts document text into dense numerical representations using OpenAI's embedding models for semantic search and retrieval.
Description
OpenAI embedding uses OpenAI's text embedding API to convert document element text into high-dimensional vectors. The default model (text-embedding-ada-002) produces 1536-dimensional vectors that capture semantic meaning, enabling similarity-based retrieval in RAG pipelines.
The Unstructured integration wraps the OpenAI API through LangChain's OpenAIEmbeddings client, providing a consistent interface that operates on Element objects rather than raw strings. This wrapper handles batching, error handling, and the mapping between Element text and embedding vectors.
Usage
Use this principle when you need high-quality semantic embeddings and have access to the OpenAI API. OpenAI embeddings are well-suited for general-purpose English text and produce unit vectors suitable for cosine similarity search. For cost-sensitive or offline scenarios, consider HuggingFace sentence-transformers as an alternative.
Theoretical Basis
Text embeddings map variable-length text to fixed-dimensional vectors using transformer-based models trained on large-scale contrastive learning objectives. The training objective ensures that semantically similar texts have high cosine similarity in the embedding space.
OpenAI ada-002: Uses a 1536-dimensional embedding space. Vectors are L2-normalized (unit vectors), so cosine similarity reduces to dot product. The model handles up to 8,191 tokens per input.
Integration pattern:
# Abstract OpenAI embedding flow
for element in elements:
text = str(element)
vector = openai_client.embed(text) # 1536-dim float vector
element.embeddings = vector
return elements