Implementation:Turboderp org Exllamav2 ExLlamaV2PosEmbedding

Knowledge Sources	Turboderp_org_Exllamav2
Domains	Positional_Encoding, Model_Architecture
Last Updated	2026-02-15 00:00 GMT

Overview

ExLlamaV2PosEmbedding implements learned (absolute) positional embeddings that are added to hidden states based on token position within the sequence.

Description

This module wraps a standard PyTorch nn.Embedding layer containing learned positional vectors. During the forward pass, it slices the embedding table at the appropriate position range and adds the positional vectors to the input hidden states in-place. The module handles both simple batch processing (no attention parameters) and streaming/batched inference where each sample in the batch may have a different past length and position offset.

By default, the module enforces that max_seq_len does not exceed the native context size of the learned embeddings, since learned positional embeddings cannot be trivially extended. However, an override_max_seq_len flag allows bypassing this check when the caller takes responsibility for handling out-of-range positions.

Position offsets from attn_params.position_offsets are respected, and negative starting positions are clamped to zero to support partial context scenarios. The module supports both standard ExLlamaV2Attention.Params and PagedParams for paged attention.

Usage

Use ExLlamaV2PosEmbedding with model architectures that use learned absolute positional embeddings (e.g., GPT-2 style) rather than rotary or relative positional encodings. The module is instantiated by the model loader and invoked as part of the embedding layer stack.

Code Reference

Source Location

• Repository: Turboderp_org_Exllamav2
• File: exllamav2/pos_embedding.py
• Lines: 1-126

Signature

class ExLlamaV2PosEmbedding(ExLlamaV2Module):

    name: str = "Pos. Embedding"

    embedding: nn.Embedding | None
    native_ctx_size: int | None

    def __init__(
        self,
        model: ExLlamaV2,
        key: str,
        override_max_seq_len: bool = False,
    ): ...

    def load(self): ...
    def unload(self): ...
    def get_weight(self) -> torch.Tensor: ...
    def weight_footprint(self) -> int: ...

    def forward(
        self,
        hidden_states: torch.Tensor,
        cache=None,
        attn_params: ExLlamaV2Attention.Params = None,
        past_len=None,
        intermediates: bool = False,
        loras=None,
        **kwargs
    ) -> torch.Tensor | dict[str: torch.Tensor]: ...

Import

from exllamav2.pos_embedding import ExLlamaV2PosEmbedding

I/O Contract

forward()

Parameter	Type	Description
hidden_states	torch.Tensor	Input hidden states of shape (batch, seq_len, hidden_size)
cache	None	Cache object (unused directly, passed for API compatibility)
attn_params	None	Attention parameters providing past_len and position_offsets per batch element; if None, positions start at 0
past_len	None	Fallback past length when attn_params is None
intermediates	bool	If True, return a dict wrapping hidden_states
loras	None	Unused; present for API compatibility

Return	Type	Description
hidden_states	torch.Tensor	Hidden states with positional embeddings added in-place (when intermediates=False)
result dict	dict	{"hidden_states": tensor} (when intermediates=True)

__init__()

Parameter	Type	Description
model	ExLlamaV2	Parent model reference
key	str	Weight key for the positional embedding tensor
override_max_seq_len	bool	If True, skip the assertion that max_seq_len <= native_ctx_size

Usage Examples

from exllamav2.pos_embedding import ExLlamaV2PosEmbedding

# Construction (typically handled by model loader)
pos_emb = ExLlamaV2PosEmbedding(
    model=model,
    key="transformer.wpe",
    override_max_seq_len=False,
)
pos_emb.load()

# Simple forward pass without attention params (positions start at 0)
# hidden_states shape: (batch, seq_len, hidden_size)
output = pos_emb.forward(hidden_states)

# Forward pass with attention params during streaming
output = pos_emb.forward(
    hidden_states,
    attn_params=attn_params,  # contains past_len per batch element
)

# Access the raw embedding weights
weight_tensor = pos_emb.get_weight()
print(f"Embedding table shape: {weight_tensor.shape}")
# e.g. torch.Size([2048, 2560])

Related Pages

• Implementation:Turboderp_org_Exllamav2_RoPE - Rotary position embeddings (alternative positional encoding approach)
• Implementation:Turboderp_org_Exllamav2_ExLlamaV2ParallelDecoder - Decoder block that may follow positional embedding in the model stack