Implementation:Ucbepic Docetl LinkResolveOperation Execute
| Knowledge Sources | |
|---|---|
| Domains | Data_Processing, Entity_Resolution |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Concrete tool for resolving link references in documents by matching them against known entity IDs using embedding similarity and LLM confirmation, provided by DocETL.
Description
The LinkResolveOperation class extends BaseOperation to perform entity linking within document pipelines. It identifies unresolved link values (references in a link field that do not match any known entity ID), computes embeddings for both the unresolved links and the known IDs, uses cosine similarity with a configurable threshold to find candidate matches, and then sends each candidate pair to the LLM via a comparison prompt for confirmation. Confirmed matches result in the link value being replaced with the resolved entity ID in-place.
Usage
Use this operation when documents contain references to other entities (e.g., "related_to", "mentions", "cites") that need to be matched against a canonical set of entity identifiers. Typical scenarios include resolving author name variations in citation networks, linking informal product references to catalog entries, or normalizing cross-references in knowledge bases.
Code Reference
Source Location
- Repository: Ucbepic_Docetl
- File: docetl/operations/link_resolve.py
- Lines: 1-198
Signature
class LinkResolveOperation(BaseOperation):
def __init__(self, *args, **kwargs): ...
def execute(self, input_data: list[dict]) -> tuple[list[dict], float]: ...
def compare(self, link_idx, id_idx, link_value, id_value, item) -> float: ...
Import
from docetl.operations.link_resolve import LinkResolveOperation
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| input_data | List[Dict] | Yes | Documents containing link references to resolve |
| comparison_prompt | str | Yes | Jinja2 template prompt for LLM comparison (has access to link_value, id_value, item) |
| id_key | str | No | Key containing the entity identifier in each document (default "title") |
| link_key | str | No | Key containing the list of link references (default "related_to") |
| blocking_threshold | float | No | Cosine similarity threshold for candidate filtering |
| blocking_conditions | List[str] | No | Code-based conditions for candidate pair generation |
| embedding_model | str | No | Model for embeddings (default "text-embedding-ada-002") |
| comparison_model | str | No | LLM model for comparison (defaults to pipeline default) |
| compare_batch_size | int | No | Number of concurrent comparison threads (default 100) |
Outputs
| Name | Type | Description |
|---|---|---|
| output | Tuple[List[Dict], float] | Input documents modified in-place with resolved links and total cost |
Usage Examples
# YAML pipeline configuration for link resolution
operations:
- name: resolve_references
type: link_resolve
comparison_prompt: |
Is "{{ link_value }}" the same entity as "{{ id_value }}"?
Context from the entity's document: {{ item }}
id_key: title
link_key: related_to
blocking_threshold: 0.7
embedding_model: "text-embedding-ada-002"
comparison_model: "gpt-4o-mini"
# Python API usage
from docetl.operations.link_resolve import LinkResolveOperation
config = {
"name": "resolve_citations",
"type": "link_resolve",
"comparison_prompt": 'Is "{{ link_value }}" referring to "{{ id_value }}"? Context: {{ item }}',
"id_key": "paper_title",
"link_key": "citations",
"blocking_threshold": 0.75,
}
resolve_op = LinkResolveOperation(runner, config, default_model, max_threads)
results, cost = resolve_op.execute(input_data)
# Documents now have their "citations" values replaced with matched paper_title values