Implementation:Facebookresearch Audiocraft RotaryEmbedding

Knowledge Sources	Facebookresearch_Audiocraft RoFormer xPos
Domains	Transformer, Positional_Encoding
Last Updated	2026-02-14 01:00 GMT

Overview

Concrete tool for applying Rotary Position Embeddings (RoPE) and its length-extrapolatable variant xPos to transformer attention.

Description

This module implements RotaryEmbedding for standard RoPE and XPos for the xPos variant. RoPE encodes position by rotating query and key vectors in pairs of dimensions using sinusoidal frequencies. xPos extends this with exponential decay factors that improve extrapolation to longer sequences than seen during training.

Usage

Import these classes when configuring positional encoding for StreamingTransformer layers. RoPE is used by the MusicGen and MAGNeT language models.

Code Reference

Source Location

• Repository: Facebookresearch_Audiocraft
• File: audiocraft/modules/rope.py
• Lines: 1-125

Signature

class RotaryEmbedding(nn.Module):
    def __init__(self, dim: int, max_period: float = 10000.0, xpos: bool = False,
                 scale: float = 512, device=None, dtype=None): ...
    def rotate(self, x: torch.Tensor, start: int = 0, invert: bool = False) -> torch.Tensor: ...

Import

from audiocraft.modules.rope import RotaryEmbedding

I/O Contract

Inputs

Name	Type	Required	Description
x	torch.Tensor	Yes	Query or key tensor [..., T, D]
start	int	No	Starting position offset (for streaming)
invert	bool	No	Apply inverse rotation (for xPos keys)

Outputs

Name	Type	Description
rotated	torch.Tensor	Position-encoded tensor, same shape as input

Related Pages

• Principle:Facebookresearch_Audiocraft_Autoregressive_Token_Generation