Implementation:Microsoft Onnxruntime CUDA ReductionAll
| Knowledge Sources | |
|---|---|
| Domains | Training, CUDA_Kernels |
| Last Updated | 2026-02-10 04:00 GMT |
Overview
Concrete tool for computing the global L2 norm across multiple tensors in the ONNX Runtime CUDA training framework.
Description
Implements the ReduceAllL2 operator for CUDA that computes the L2 norm (Frobenius norm) across all elements of multiple input tensors, producing a single scalar output. The implementation supports two paths: (1) non-deterministic path using MultiTensorReduceL2 functor via the multi-tensor framework with chunk size 2048*32, computing a squared sum across all tensors followed by a scalar square root; (2) deterministic path that performs individual reduce_square_sum per tensor, followed by a reduce_sum of the per-tensor norms, and a final square root. The deterministic path uses accumulation types for numerical stability and separate reduction buffers. Registered for multiple input/output type combinations: float/float, MLFloat16/float, float/MLFloat16, MLFloat16/MLFloat16, BFloat16/float, float/BFloat16, BFloat16/BFloat16.
Usage
Used during training for global gradient norm computation, required by gradient clipping, LAMB optimizer trust ratio, and mixed-precision loss scaling.
Code Reference
Source Location
- Repository: Microsoft_Onnxruntime
- File: orttraining/orttraining/training_ops/cuda/reduction/reduction_all.cc
- Lines: 1-114
Signature
template <typename TIn, typename TOut>
class ReduceAllL2 : public CudaKernel {
Status ComputeInternal(OpKernelContext* ctx) const;
};
Import
#include "orttraining/training_ops/cuda/reduction/reduction_all.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| tensors | Tensor(TIn)... | Yes | Variadic input tensors to compute L2 norm over |
Outputs
| Name | Type | Description |
|---|---|---|
| norm | Tensor(TOut) | Scalar L2 norm of all input elements |
Usage Examples
REGISTER_REDUCE_ALL_KERNEL_TYPED(ReduceAllL2, float, float)
REGISTER_REDUCE_ALL_KERNEL_TYPED(ReduceAllL2, MLFloat16, float)
REGISTER_REDUCE_ALL_KERNEL_TYPED(ReduceAllL2, BFloat16, float)