Implementation:Recommenders team Recommenders Wikidata Utils
| Knowledge Sources | |
|---|---|
| Domains | Recommender Systems, Knowledge Graphs, Data Enrichment |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Provides utilities for querying Wikipedia and Wikidata APIs to find entity IDs, retrieve linked entities, and get entity descriptions for knowledge graph enrichment of recommendation datasets.
Description
The wikidata module enables knowledge-aware recommendation pipelines by interfacing with the Wikipedia API and Wikidata SPARQL endpoint. get_session manages a shared requests.Session for connection reuse across API calls. find_wikidata_id performs a two-step lookup: first searching Wikipedia's API for a title string to obtain a page ID, then fetching the corresponding Wikidata entity ID from the page's properties. query_entity_links executes a SPARQL query against the Wikidata query service (https://query.wikidata.org/sparql) to retrieve up to 500 linked entities for a given entity ID, including property labels and value labels filtered to English. read_linked_entities parses the SPARQL JSON response into tuples of (entity_id, entity_name). query_entity_description fetches the English-language short description of an entity via SPARQL. search_wikidata orchestrates all functions to produce a DataFrame of search results with columns for name, original entity, linked entities, linked entity names, and optional descriptions. All API-calling functions are decorated with @retry (random backoff, 3 attempts), @lru_cache (up to 1024 entries) for memoization, and @log_retries for logging retry attempts.
Usage
Use this module when building knowledge-aware recommendation algorithms (such as DKN) that leverage Wikidata knowledge graph information to enrich item features with entity embeddings and relationships. Call search_wikidata with a list of item names to obtain structured knowledge graph data as a DataFrame.
Code Reference
Source Location
- Repository: Recommenders
- File: recommenders/datasets/wikidata.py
- Lines: 1-295
Signature
def log_retries(func) -> callable
def get_session(session=None) -> requests.Session
@lru_cache(maxsize=1024)
@retry(wait_random_min=1000, wait_random_max=5000, stop_max_attempt_number=3)
def find_wikidata_id(name, limit=1, session=None) -> str
@lru_cache(maxsize=1024)
@retry(wait_random_min=1000, wait_random_max=5000, stop_max_attempt_number=3)
def query_entity_links(entity_id, session=None) -> dict
def read_linked_entities(data) -> list[tuple[str, str]]
@lru_cache(maxsize=1024)
@retry(wait_random_min=1000, wait_random_max=5000, stop_max_attempt_number=3)
def query_entity_description(entity_id, session=None) -> str
def search_wikidata(names, extras=None, describe=True) -> pd.DataFrame
Import
from recommenders.datasets.wikidata import (
find_wikidata_id,
query_entity_links,
read_linked_entities,
query_entity_description,
search_wikidata,
)
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| name | str | Yes (for find_wikidata_id) | A search string (e.g., "Batman (1989) film") to look up in Wikipedia. |
| limit | int | No (default: 1) | Number of search results to return from Wikipedia. |
| session | requests.Session | No (default: None) | Requests session to reuse connections. If None, uses a global shared session. |
| entity_id | str | Yes (for query functions) | A Wikidata entity ID (e.g., "Q116852"). |
| data | dict | Yes (for read_linked_entities) | JSON dictionary returned by query_entity_links. |
| names | list of str | Yes (for search_wikidata) | List of names to search for in Wikipedia/Wikidata. |
| extras | dict(str: list) or None | No (default: None) | Optional extra fields to attach to results, keyed by field name with lists of values corresponding to each name. |
| describe | bool | No (default: True) | Whether to include the entity description in results. |
Outputs
| Name | Type | Description |
|---|---|---|
| return (find_wikidata_id) | str | Wikidata entity ID (e.g., "Q116852") or "entityNotFound" if no match. |
| return (query_entity_links) | dict | JSON dictionary with SPARQL query results containing linked entities. |
| return (read_linked_entities) | list of tuple(str, str) | List of (entity_id, entity_name) tuples for all linked entities. |
| return (query_entity_description) | str | English description of the entity, or "descriptionNotFound" if unavailable. |
| return (search_wikidata) | pd.DataFrame | DataFrame with columns: name, original_entity, linked_entities, name_linked_entities, and optionally description. |
Usage Examples
Basic Usage
from recommenders.datasets.wikidata import (
find_wikidata_id,
query_entity_links,
read_linked_entities,
query_entity_description,
search_wikidata,
)
# Find the Wikidata entity ID for a movie
entity_id = find_wikidata_id("The Matrix 1999 film")
print(entity_id) # e.g., "Q83495"
# Query all linked entities
links_data = query_entity_links(entity_id)
linked = read_linked_entities(links_data)
for eid, ename in linked[:5]:
print(f"{eid}: {ename}")
# Get the entity description
desc = query_entity_description(entity_id)
print(desc) # e.g., "1999 science fiction action film"
# Batch search for multiple items
movie_names = ["The Matrix", "Inception", "Interstellar"]
results_df = search_wikidata(movie_names, describe=True)
print(results_df.columns.tolist())
# ['name', 'original_entity', 'linked_entities', 'name_linked_entities', 'description']
Dependencies
- requests - HTTP requests for Wikipedia and Wikidata APIs
- pandas - DataFrame construction for search results
- retrying - Automatic retry with random backoff on API failures
- functools.lru_cache - Memoization of API responses (up to 1024 entries per function)
- logging - Logging of retry attempts and warnings