Implementation:Recommenders team Recommenders Covid Utils

Knowledge Sources	Recommenders
Domains	Recommender Systems, Data Loading, COVID-19 Research
Last Updated	2026-02-10 00:00 GMT

Overview

Provides utilities for loading, cleaning, and processing the Azure Open Research COVID-19 (CORD-19) dataset for use in recommendation and NLP experiments.

Description

The covid_utils module contains six functions for working with the CORD-19 dataset hosted on Azure Open Datasets. load_pandas_df constructs a URI to Azure Blob Storage and fetches the metadata CSV into a pandas DataFrame. remove_duplicates iterates over specified columns and drops duplicate rows using np.where with df.duplicated(). remove_nan replaces empty strings with NaN and filters out rows with missing values in specified columns. clean_dataframe orchestrates both cleaning steps, removing duplicates by cord_uid and doi, then dropping rows missing values in cord_uid, doi, title, license, and url. retrieve_text fetches the full body text of a single article from Azure Blob Storage as JSON and concatenates the body_text paragraphs into a single string. get_public_domain_text applies text retrieval across all DataFrame rows, filters out entries with empty text, and returns a curated DataFrame with selected metadata columns plus a full_text column.

Usage

Use this module when building recommendation or NLP experiments on COVID-19 research papers. Start by calling load_pandas_df to fetch the metadata, then clean_dataframe to remove duplicates and invalid rows, and finally get_public_domain_text to retrieve article body text for content-based processing.

Code Reference

Source Location

• Repository: Recommenders
• File: recommenders/datasets/covid_utils.py
• Lines: 1-191

Signature

def load_pandas_df(
    azure_storage_account_name="azureopendatastorage",
    azure_storage_sas_token="",
    container_name="covid19temp",
    metadata_filename="metadata.csv",
) -> pd.DataFrame

def remove_duplicates(df, cols) -> pd.DataFrame

def remove_nan(df, cols) -> pd.DataFrame

def clean_dataframe(df) -> pd.DataFrame

def retrieve_text(
    entry,
    container_name,
    azure_storage_account_name="azureopendatastorage",
    azure_storage_sas_token="",
) -> str

def get_public_domain_text(
    df,
    container_name,
    azure_storage_account_name="azureopendatastorage",
    azure_storage_sas_token="",
) -> pd.DataFrame

Import

from recommenders.datasets.covid_utils import (
    load_pandas_df,
    clean_dataframe,
    get_public_domain_text,
)

I/O Contract

Inputs

Name	Type	Required	Description
azure_storage_account_name	str	No (default: "azureopendatastorage")	Azure storage account name hosting the CORD-19 data.
azure_storage_sas_token	str	No (default: "")	Azure storage SAS token for authentication.
container_name	str	No (default: "covid19temp")	Azure storage container name.
metadata_filename	str	No (default: "metadata.csv")	Name of the metadata CSV file.
df	pd.DataFrame	Yes (for cleaning/text functions)	DataFrame containing CORD-19 metadata.
cols	list of str	Yes (for remove_duplicates/remove_nan)	Column names to check for duplicates or NaN values.
entry	pd.Series	Yes (for retrieve_text)	A single row from the metadata DataFrame.

Outputs

Name	Type	Description
return (load_pandas_df)	pd.DataFrame	Metadata DataFrame loaded from the CORD-19 CSV.
return (clean_dataframe)	pd.DataFrame	Cleaned DataFrame with duplicates and NaN rows removed.
return (retrieve_text)	str	Full text of an article as a single string, or empty string on failure.
return (get_public_domain_text)	pd.DataFrame	DataFrame with cord_uid, doi, title, publish_time, authors, journal, url, abstract, and full_text columns.

Usage Examples

Basic Usage

from recommenders.datasets.covid_utils import (
    load_pandas_df,
    clean_dataframe,
    get_public_domain_text,
)

# Load the CORD-19 metadata
metadata_df = load_pandas_df()

# Clean the dataset (remove duplicates and rows with missing fields)
clean_df = clean_dataframe(metadata_df)

# Filter for public domain articles
public_df = clean_df[clean_df["license"] == "cc0"]

# Retrieve full text for public domain articles
full_text_df = get_public_domain_text(public_df, container_name="covid19temp")
print(full_text_df.columns.tolist())
# ['cord_uid', 'doi', 'title', 'publish_time', 'authors', 'journal', 'url', 'abstract', 'full_text']

Dependencies

• pandas - DataFrame construction and manipulation
• numpy - Numeric operations and NaN handling
• requests - HTTP requests for fetching article JSON from Azure Blob Storage

Related Pages

Requires Environment

• Environment:Recommenders_team_Recommenders_Python_Core_Dependencies