Implementation:NVIDIA TransformerEngine Activation C API

Field	Value
Sources	TransformerEngine
Domains	Deep_Learning, Optimization
Last Updated	2026-02-07 14:00 GMT

Overview

Declares the C API for GPU-accelerated activation functions (GeLU, SiLU, ReLU, QuickGeLU, SquaredReLU) and their gated variants (GeGLU, SwiGLU, ReGLU, etc.), including both forward and backward passes.

Description

activation.h exposes activation functions as extern "C" functions operating on NVTETensor and NVTEGroupedTensor types via CUDA streams. An NVTE_Activation_Type enum enumerates all supported activations.

Supported activations:

• Standard: GeLU, SiLU, ReLU, QuickGeLU, SquaredReLU
• Gated variants: GeGLU, SwiGLU, ReGLU, QGeGLU, SReGLU, Clamped SwiGLU
• Backward passes: Corresponding gradient functions (nvte_dgelu, nvte_dsilu, etc.)
• Grouped variants: Batched processing via nvte_group_gelu, nvte_group_silu, etc.

All output tensors support MXFP8 block quantization when the scaling mode is configured accordingly.

Usage

Use these functions when implementing forward and backward passes for Transformer feed-forward network layers with FP8/MXFP8 quantization support.

Code Reference

Source Location

Repository

NVIDIA/TransformerEngine

File

transformer_engine/common/include/transformer_engine/activation.h

Lines

1--424

Signature

enum class NVTE_Activation_Type {
  GELU, GEGLU, SILU, SWIGLU, RELU, REGLU,
  QGELU, QGEGLU, SRELU, SREGLU, CLAMPED_SWIGLU
};

void nvte_gelu(const NVTETensor input, NVTETensor output, cudaStream_t stream);
void nvte_silu(const NVTETensor input, NVTETensor output, cudaStream_t stream);
void nvte_relu(const NVTETensor input, NVTETensor output, cudaStream_t stream);
void nvte_qgelu(const NVTETensor input, NVTETensor output, cudaStream_t stream);
void nvte_srelu(const NVTETensor input, NVTETensor output, cudaStream_t stream);

// Gated variants
void nvte_geglu(const NVTETensor input, NVTETensor output, cudaStream_t stream);
void nvte_swiglu(const NVTETensor input, NVTETensor output, cudaStream_t stream);
void nvte_reglu(const NVTETensor input, NVTETensor output, cudaStream_t stream);

// Backward passes
void nvte_dgelu(const NVTETensor grad, const NVTETensor input,
                NVTETensor output, cudaStream_t stream);

Import

#include "transformer_engine/activation.h"

I/O Contract

Inputs

Name	Type	Required	Description
input	NVTETensor	Yes	Input tensor for activation
stream	cudaStream_t	Yes	CUDA stream for the operation

Outputs

Name	Type	Description
output	NVTETensor	Result of the activation function, optionally quantized to FP8/MXFP8

Usage Examples

#include "transformer_engine/activation.h"

// Forward pass: apply GeLU activation
nvte_gelu(input_tensor, output_tensor, cuda_stream);

// Backward pass: compute GeLU gradient
nvte_dgelu(grad_output, input_tensor, grad_input, cuda_stream);

// Gated variant: SwiGLU (used in LLaMA-style models)
nvte_swiglu(input_tensor, output_tensor, cuda_stream);

Related Pages

• Environment:NVIDIA_TransformerEngine_CUDA_Toolkit_Requirements