Implementation:Huggingface Peft PrefixTuningConfig

Overview

PrefixTuningConfig is the configuration dataclass for prefix tuning in the Hugging Face PEFT library. It stores the parameters needed to set up a prefix tuning adapter, including whether to use an MLP-based reparameterization of the prefix vectors and the hidden size of that MLP. This class inherits from PromptLearningConfig, which provides shared fields common to all prompt-learning methods.

Signature

from peft import PrefixTuningConfig

@dataclass
class PrefixTuningConfig(PromptLearningConfig):
    encoder_hidden_size: int = None
    prefix_projection: bool = False

Parameters

Own Parameters

Parameter	Type	Default	Description
`encoder_hidden_size`	`int`	`None`	The hidden size of the MLP encoder used for reparameterization of the prefix vectors. Only relevant when `prefix_projection` is `True`. When `None`, defaults are inferred from the base model's configuration.
`prefix_projection`	`bool`	`False`	Whether to project the prefix embeddings through an MLP before injecting them into the transformer layers. When `True`, a two-layer MLP is used during training to reparameterize the prefix vectors for more stable optimization. The MLP can be discarded after training.

Inherited Parameters (from PromptLearningConfig)

Parameter	Type	Default	Description
`num_virtual_tokens`	`int`	`None`	The number of virtual prefix tokens prepended to the key and value matrices at each transformer layer.
`token_dim`	`int`	`None`	The hidden embedding dimension of the base transformer model.
`num_transformer_submodules`	`Optional[int]`	`None`	The number of transformer submodules in the base model.
`num_attention_heads`	`Optional[int]`	`None`	The number of attention heads in the base model.
`num_layers`	`Optional[int]`	`None`	The number of layers in the base transformer model.
`task_type`	`str`	(from PeftConfig)	The task type (e.g., SEQ_CLS, CAUSAL_LM, SEQ_2_SEQ_LM, TOKEN_CLS).

Usage Example

from peft import PrefixTuningConfig, get_peft_model, TaskType

# Basic prefix tuning without reparameterization
config_basic = PrefixTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=20,
)

# Prefix tuning with MLP reparameterization for stable training
config_mlp = PrefixTuningConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    num_virtual_tokens=20,
    prefix_projection=True,
    encoder_hidden_size=512,
)

model = get_peft_model(base_model, config_mlp)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment