Implementation:Recommenders team Recommenders LightGBM Utils

Knowledge Sources	Recommenders
Domains	Feature Engineering, Gradient Boosting, Data Preprocessing
Last Updated	2026-02-10 00:00 GMT

Overview

Provides the NumEncoder class for comprehensive feature engineering that converts mixed categorical and numerical features into dense numerical representations suitable for LightGBM and similar gradient boosting models.

Description

This module contains the NumEncoder class and a helper function unpackbits. NumEncoder implements a multi-step encoding pipeline for feature engineering: (1) it filters low-frequency categorical values below a configurable threshold or bottom percentage, replacing them with the <LESS> sentinel; (2) it fills missing values using <UNK> for categoricals and the column mean for numericals; (3) it applies ordinal encoding via the category_encoders library; (4) it performs dynamic sequential target encoding that computes a running mean and count of labels per category to prevent target leakage; (5) it converts ordinal-encoded categoricals to binary bit representations using the unpackbits helper. The fit_transform method learns all encoding parameters from training data and returns (features, labels) numpy arrays. The transform method applies the saved encodings to test or validation data without re-learning parameters. The unpackbits function converts decimal integer arrays into multi-bit binary arrays for compact categorical representation.

Usage

Use this module when preparing features for a LightGBM-based recommendation model or any gradient boosting approach that requires dense numerical input. Initialize NumEncoder with your categorical columns, numerical columns, and label column, then call fit_transform on the training set and transform on the test/validation set to obtain consistently encoded feature matrices.

Code Reference

Source Location

• Repository: Recommenders
• File: recommenders/models/lightgbm/lightgbm_utils.py
• Lines: 1-207

Signature

def unpackbits(x, num_bits)

class NumEncoder:
    def __init__(self, cate_cols, nume_cols, label_col, threshold=10, thresrate=0.99)
    def fit_transform(self, df)
    def transform(self, df)

Import

from recommenders.models.lightgbm.lightgbm_utils import NumEncoder, unpackbits

I/O Contract

Inputs

NumEncoder.__init__

Name	Type	Required	Description
cate_cols	list	Yes	Column names of categorical features
nume_cols	list	Yes	Column names of numerical features
label_col	object	Yes	Column name of the label
threshold	int	No	Categories with frequency below this are replaced with <LESS> (default 10)
thresrate	float	No	Top percentage of categories to keep; bottom (1 - thresrate) are filtered (default 0.99)

NumEncoder.fit_transform

Name	Type	Required	Description
df	pd.DataFrame	Yes	Training DataFrame containing categorical columns, numerical columns, and the label column

NumEncoder.transform

Name	Type	Required	Description
df	pd.DataFrame	Yes	Test or validation DataFrame with the same columns as the training set

unpackbits

Name	Type	Required	Description
x	numpy.ndarray	Yes	Decimal integer array to convert to binary representation
num_bits	int	Yes	Maximum bit length for the binary conversion

Outputs

NumEncoder.fit_transform

Name	Type	Description
trn_x	numpy.ndarray	Encoded feature matrix combining numerical features, target-encoded features, and binary-encoded categorical features
trn_y	numpy.ndarray	Label array reshaped to (-1, 1)

NumEncoder.transform

Name	Type	Description
vld_x	numpy.ndarray	Encoded feature matrix using parameters learned during fit_transform
vld_y	numpy.ndarray	Label array reshaped to (-1, 1)

unpackbits

Name	Type	Description
return	numpy.ndarray	Binary representation array with an additional dimension of size num_bits

Usage Examples

Basic Usage

import pandas as pd
from recommenders.models.lightgbm.lightgbm_utils import NumEncoder

# Define column types
categorical_cols = ["user_gender", "item_category", "user_occupation"]
numerical_cols = ["user_age", "item_price", "user_activity_count"]
label_col = "rating"

# Initialize the encoder
encoder = NumEncoder(
    cate_cols=categorical_cols,
    nume_cols=numerical_cols,
    label_col=label_col,
    threshold=10,      # Filter categories appearing fewer than 10 times
    thresrate=0.99,    # Keep top 99% of categories by frequency
)

# Encode training data (learns encoding parameters)
train_x, train_y = encoder.fit_transform(train_df)

# Encode test data (applies saved parameters)
test_x, test_y = encoder.transform(test_df)

# Use with LightGBM
import lightgbm as lgb
dtrain = lgb.Dataset(train_x, label=train_y.ravel())
dtest = lgb.Dataset(test_x, label=test_y.ravel(), reference=dtrain)