Principle:FMInference FlexLLMGen Fused Transformer Layer CUDA
| Knowledge Sources | |
|---|---|
| Domains | CUDA, Transformer, Kernel Fusion, GPU Optimization |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
Fusing an entire transformer encoder layer into a single GPU execution context eliminates inter-kernel launch overhead and enables workspace memory reuse across the attention, normalization, and feed-forward sub-layers.
Description
A standard transformer encoder layer consists of many discrete operations: layer normalization, QKV projection, attention score computation, softmax, dropout, context multiplication, output projection, residual addition, another normalization, and a two-layer feed-forward network with activation. When each operation is a separate GPU kernel launch, the overhead from kernel launch latency, memory allocation, and redundant global memory round-trips becomes significant, especially for small batch sizes or short sequences.
Kernel fusion addresses this by combining multiple operations into fewer, larger kernels or by orchestrating them within a single C++ class that manages a shared workspace buffer. Key fusion strategies include:
- Fused QKV projection: Computing Q, K, and V matrices in a single GEMM call with 3x hidden_size output, then splitting via pointer arithmetic. This saves two additional GEMM kernel launches.
- Fused bias-add and reshape: Combining bias addition with the (batch, seq, 3*hidden) to (3, batch, heads, seq, head_dim) transpose in a single custom kernel.
- Fused dropout with residual addition: Combining the dropout mask application and residual connection into one pass, avoiding an extra global memory read/write cycle.
- Workspace memory reuse: Pre-allocating a single large workspace buffer and partitioning it across operations, so intermediate results are written to and read from the same memory region without separate allocations.
Activation checkpointing trades compute for memory by not saving certain intermediate values (e.g., GELU inputs) during the forward pass and recomputing them during backward. The gelu_checkpoint and attn_dropout_checkpoint flags control this tradeoff.
Pre-LN vs. Post-LN ordering affects both numerical stability and the data flow through residual connections. Pre-LN applies normalization before the attention/FFN sub-layers, while Post-LN applies it after.
Usage
Apply this principle when designing high-performance transformer implementations for training, where reducing kernel launch overhead and memory traffic through fusion can yield significant speedups, especially at smaller batch sizes where GPU compute is not the bottleneck.
Theoretical Basis
Kernel launch overhead on NVIDIA GPUs is typically 5-10 microseconds per kernel. A standard transformer layer may involve 20-30 separate kernel launches; fusing these into fewer launches can save 100+ microseconds per layer, which compounds across many layers and training steps.
Memory bandwidth savings from fusion arise because intermediate results (e.g., the output of bias addition that feeds into a reshape) stay in registers or L1/L2 cache rather than being written to and read from global memory (HBM). Global memory bandwidth on modern GPUs is 1-3 TB/s, but L1/L2 can be 10-30x faster.
Workspace pre-allocation avoids the overhead of CUDA memory allocator calls (which require driver interaction) by computing the maximum workspace size upfront: workSpaceSize = 4 * (bsz * seq * hidden) + additional buffers for attention scores and intermediate activations.
Stochastic mode skips cudaStreamSynchronize calls between operations, allowing the GPU to overlap kernel execution and scheduling. This improves throughput but makes debugging harder since errors may be detected later than they occur.