Principle:Ucbepic Docetl Entity Link Resolution
| Knowledge Sources | |
|---|---|
| Domains | LLM_Data_Processing, Entity_Resolution |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Two-phase entity resolution uses embedding-based blocking followed by LLM-based comparison to resolve variant references (links) in documents to their canonical entity identifiers.
Theoretical Basis
Documents often contain references to entities using inconsistent names, abbreviations, or misspellings. For example, a document might reference "Prof. Smith" while the canonical entity list contains "John Smith, PhD." Entity link resolution maps these variant references to their canonical forms, enabling downstream operations like aggregation and graph construction to work correctly.
The DocETL link resolve operation implements a two-phase approach. In the blocking phase, both the canonical entity identifiers and the unresolved link values are embedded using a text embedding model (defaulting to text-embedding-ada-002, chosen for its stronger performance on short texts). A cosine similarity matrix is computed between link embeddings and entity embeddings, and pairs below a configurable blocking threshold are pruned. This eliminates the vast majority of comparisons -- the operation logs the percentage of comparisons saved. In the comparison phase, each remaining candidate pair is evaluated by an LLM using a user-defined comparison prompt that receives the link value, the candidate entity ID, and the full entity record. The LLM returns a boolean is_same judgment, and confirmed matches are recorded as replacements.
The operation is specifically designed for the link resolution variant of entity resolution, where one side (the entity list) is authoritative and the other side (the links) contains noisy references. This asymmetry distinguishes it from symmetric entity resolution (handled by the resolve operation) and allows for a simpler replacement-based output model where each link value is mapped to at most one canonical entity.
Key Design Decisions
| Decision | Choice | Rationale |
|---|---|---|
| Embedding model | text-embedding-ada-002 by default | Better performance on short text strings compared to text-embedding-3-small; link values and entity names are typically short |
| Blocking strategy | Cosine similarity threshold on full cross-product of embeddings | Simple and effective for moderate-sized entity lists; avoids complex indexing structures while still pruning the majority of non-matches |
| Comparison output | Boolean is_same judgment per pair with replacement dictionary | Matches the asymmetric nature of link resolution where each link maps to at most one canonical entity |