Implementation:Microsoft DeepSpeedExamples Focal Loss

Knowledge Sources	Microsoft_DeepSpeedExamples
Domains	Deep Learning, Loss Functions, Classification
Last Updated	2026-02-07 12:00 GMT

Overview

A PyTorch implementation of Focal Loss that down-weights well-classified examples to focus training on hard, misclassified samples in imbalanced classification tasks.

Description

The FocalLoss class implements the focal loss function from the paper "Focal Loss for Dense Object Detection" (Lin et al., 2017). The loss is defined as Loss(x, class) = -alpha * (1 - softmax(x)[class])^gamma * log(softmax(x)[class]), where the (1 - p_t)^gamma modulating factor reduces the contribution of easy examples (those with high predicted probability for the correct class) and concentrates the loss on hard-to-classify instances.

The alpha parameter is a per-class weighting factor provided as a 1D tensor, defaulting to uniform weights of 1.0 for all classes. The gamma parameter controls the strength of the focusing effect: gamma=0 recovers standard cross-entropy, while higher values (default gamma=2) more aggressively down-weight easy examples. The forward method computes softmax probabilities, constructs a one-hot class mask via scatter_, gathers the predicted probability for the true class, and applies the focal modulation. The loss supports both mean and sum reduction via the size_average flag.

The implementation handles automatic CUDA device placement for the alpha tensor when inputs are on GPU.

Usage

Use Focal Loss as a drop-in replacement for cross-entropy loss when training on class-imbalanced datasets. It is particularly effective for object detection, multi-class classification, and any task where easy negatives dominate the loss gradient. In the BingBertSquad pipeline it is available as an alternative loss function for fine-tuning.

Code Reference

Source Location

• Repository: Microsoft_DeepSpeedExamples
• File: training/BingBertSquad/turing/loss.py
• Lines: 1-60

Signature

class FocalLoss(nn.Module):
    def __init__(self, class_num, alpha=None, gamma=2, size_average=True):
    def forward(self, inputs, targets) -> Tensor:

Import

from turing.loss import FocalLoss

I/O Contract

Inputs

Name	Type	Required	Description
class_num	int	Yes	Number of classes in the classification task
alpha	Tensor or None	No	Per-class weighting factors of shape (class_num, 1); defaults to ones
gamma	float	No	Focusing parameter controlling down-weighting of easy examples (default: 2)
size_average	bool	No	If True, return mean loss; if False, return sum (default: True)
inputs	Tensor	Yes	Raw logits of shape (N, C) where N is batch size and C is number of classes
targets	Tensor	Yes	Ground-truth class indices of shape (N,)

Outputs

Name	Type	Description
loss	Tensor	Scalar focal loss value, either mean or sum over the batch

Usage Examples

from turing.loss import FocalLoss

# Initialize Focal Loss for 10 classes
criterion = FocalLoss(class_num=10, gamma=2, size_average=True)

# Forward pass
logits = model(inputs)           # shape: (batch_size, 10)
targets = batch["labels"]        # shape: (batch_size,)
loss = criterion(logits, targets)
loss.backward()