Implementation:Recommenders team Recommenders NCF NNI Utils

Knowledge Sources	Recommenders
Domains	Hyperparameter Tuning, Evaluation, NNI
Last Updated	2026-02-10 00:00 GMT

Overview

Utility functions for evaluating trained NCF models and combining metrics results during NNI hyperparameter tuning experiments.

Description

This module provides two helper functions for post-experiment analysis of NCF models tuned with NNI. The compute_test_results function takes a trained NCF model and computes both rating predictions (per user-item pair from the test set) and ranking predictions (all user-item combinations with training items filtered out via an outer merge). It evaluates each configured metric by dynamically calling the metric function using eval().

The combine_metrics_dicts function takes multiple metrics dictionaries (as variable arguments) and concatenates them into a single pandas DataFrame using successive append calls. This is useful for aggregating results across multiple tuning trials for comparison and reporting.

Usage

Use these functions after an NNI hyperparameter tuning experiment completes to evaluate the best model on a held-out test set and to aggregate metrics from multiple trials into a single DataFrame for analysis.

Code Reference

Source Location

• Repository: Recommenders
• File: recommenders/tuning/nni/ncf_utils.py
• Lines: 1-78

Signature

def compute_test_results(model, train, test, rating_metrics, ranking_metrics, k=DEFAULT_K)

def combine_metrics_dicts(*metrics)

Import

from recommenders.tuning.nni.ncf_utils import compute_test_results, combine_metrics_dicts

I/O Contract

Inputs

compute_test_results

Name	Type	Required	Description
model	object	Yes	Trained NCF (TensorFlow) model with a predict() method
train	pandas.DataFrame	Yes	Training set DataFrame with userID, itemID, and rating columns
test	pandas.DataFrame	Yes	Test set DataFrame with userID, itemID, and rating columns
rating_metrics	list	Yes	List of rating metric function names (e.g., ["rmse", "mae"])
ranking_metrics	list	Yes	List of ranking metric function names (e.g., ["ndcg_at_k"])
k	int	No	Top-K value for ranking metrics (default: DEFAULT_K)

combine_metrics_dicts

Name	Type	Required	Description
*metrics	dict (varargs)	Yes	One or more dictionaries of metric name to value mappings

Outputs

compute_test_results

Name	Type	Description
return	dict	Dictionary mapping metric names to their computed values

combine_metrics_dicts

Name	Type	Description
return	pandas.DataFrame	DataFrame with one row per metrics dictionary and columns for each metric

Usage Examples

Basic Usage

from recommenders.tuning.nni.ncf_utils import compute_test_results, combine_metrics_dicts

# Evaluate a trained NCF model on the test set
test_results = compute_test_results(
    model=trained_ncf_model,
    train=train_df,
    test=test_df,
    rating_metrics=["rmse", "mae"],
    ranking_metrics=["ndcg_at_k", "precision_at_k"],
    k=10,
)

# Combine metrics from multiple trials
trial1_metrics = {"rmse": 0.95, "mae": 0.72}
trial2_metrics = {"rmse": 0.91, "mae": 0.68}
combined_df = combine_metrics_dicts(trial1_metrics, trial2_metrics)
print(combined_df)