Implementation:NVIDIA TransformerEngine Debug Log NVFP4 Stats
| Field | Value |
|---|---|
| Sources | TransformerEngine |
| Domains | Deep_Learning, PyTorch, Debug, Quantization |
| Last Updated | 2026-02-07 14:00 GMT |
Overview
Logs statistics of NVFP4 quantized tensors, providing underflow percentage and mean squared error metrics for 4-bit quantization debugging.
Description
LogNvfp4TensorStats collects statistics on NVFP4-quantized tensors during training. It supports two statistics: underflows% (percentage of non-zero elements clipped to zero from packed FP4 data) and mse (mean squared error between quantized and original tensors). The feature requires an NVFP4Quantizer and validates that quantized tensors are NVFP4TensorStorage instances. Like other logging features, it is micro-batch aware and supports distributed reduction.
Usage
Enable via YAML config under transformer_engine.LogNvfp4TensorStats. Requires NVFP4-capable hardware and recipe. Use freq > 1 to reduce overhead.
Code Reference
Source Location
- Repository
NVIDIA/TransformerEngine- File
transformer_engine/debug/features/log_nvfp4_tensor_stats.py- Lines
- 1--225
Signature
@Registry.register_feature(namespace="transformer_engine")
class LogNvfp4TensorStats(BaseLogTensorStats):
def check_if_stat_is_supported(self, stat: str) -> bool: ...
def get_stat_with_prefix(self, stat: str) -> str: ...
def inspect_tensor_enabled(self, config, layer_name, tensor_name, iteration) -> Tuple[bool, Optional[int]]: ...
def inspect_tensor(self, config, layer_name, tensor_name, iteration, tp_group, tensor, rowwise_quantized_tensor=None, columnwise_quantized_tensor=None, quantizer=None) -> None: ...
Import
from transformer_engine.debug.features.log_nvfp4_tensor_stats import LogNvfp4TensorStats
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| config | Dict | Yes | Must contain stats list; optionally freq, start_step, end_step
|
| tensor | torch.Tensor | Yes | Original high-precision tensor |
| quantizer | NVFP4Quantizer | Yes | Must be an NVFP4Quantizer instance |
| rowwise_quantized_tensor | NVFP4TensorStorage | Yes | The NVFP4 quantized tensor |
Outputs
| Name | Type | Description |
|---|---|---|
| (none) | None | Statistics are buffered and logged at the next debug_api.step()
|
Usage Examples
# YAML configuration:
# example_nvfp4_tensor_stat_collection:
# enabled: True
# layers:
# layer_types: [layernorm_linear]
# transformer_engine:
# LogNvfp4TensorStats:
# enabled: True
# tensors_struct:
# - tensor: activation
# stats: [underflows%, mse]
# freq: 1
# - tensor: gradient
# stats: [underflows%, mse]
# freq: 5
# start_step: 0
# end_step: 80