Implementation:Huggingface Peft PromptEncoderConfig

Overview

PromptEncoderConfig is the configuration dataclass for P-tuning in the Hugging Face PEFT library. It stores the parameters needed to set up a P-tuning adapter, including the type of prompt encoder (MLP or LSTM), the encoder's hidden size, depth, and dropout rate. This class inherits from PromptLearningConfig, which provides shared fields common to all prompt-learning methods.

Signature

from peft import PromptEncoderConfig

@dataclass
class PromptEncoderConfig(PromptLearningConfig):
    encoder_reparameterization_type: Union[str, PromptEncoderReparameterizationType] = PromptEncoderReparameterizationType.MLP
    encoder_hidden_size: int = None
    encoder_num_layers: int = 2
    encoder_dropout: float = 0.0

Parameters

Own Parameters

Parameter	Type	Default	Description
`encoder_reparameterization_type`	`Union[str, PromptEncoderReparameterizationType]`	`PromptEncoderReparameterizationType.MLP`	The type of reparameterization (encoder architecture) used to generate prompt embeddings. Accepted values are MLP (a multi-layer perceptron) and LSTM (a long short-term memory network). The MLP encoder applies feedforward transformations, while the LSTM encoder captures sequential dependencies between prompt positions.
`encoder_hidden_size`	`int`	`None`	The hidden size of the prompt encoder. Determines the dimensionality of the intermediate representations within the encoder network. When `None`, defaults are inferred from the base model's configuration.
`encoder_num_layers`	`int`	`2`	The number of layers in the prompt encoder. Controls the depth of the MLP or LSTM network used to generate the prompt embeddings.
`encoder_dropout`	`float`	`0.0`	The dropout probability applied within the prompt encoder. A value of `0.0` means no dropout is applied.

Inherited Parameters (from PromptLearningConfig)

Parameter	Type	Default	Description
`num_virtual_tokens`	`int`	`None`	The number of virtual (soft) tokens whose embeddings are generated by the prompt encoder and prepended to the input.
`token_dim`	`int`	`None`	The hidden embedding dimension of the base transformer model.
`num_transformer_submodules`	`Optional[int]`	`None`	The number of transformer submodules in the base model.
`num_attention_heads`	`Optional[int]`	`None`	The number of attention heads in the base model.
`num_layers`	`Optional[int]`	`None`	The number of layers in the base transformer model.
`task_type`	`str`	(from PeftConfig)	The task type (e.g., SEQ_CLS, CAUSAL_LM, SEQ_2_SEQ_LM, TOKEN_CLS).

Usage Example

from peft import PromptEncoderConfig, get_peft_model, TaskType

# P-tuning with MLP encoder (default)
config_mlp = PromptEncoderConfig(
    task_type=TaskType.SEQ_CLS,
    num_virtual_tokens=20,
    encoder_reparameterization_type="MLP",
    encoder_hidden_size=256,
    encoder_num_layers=2,
    encoder_dropout=0.1,
)

# P-tuning with LSTM encoder
config_lstm = PromptEncoderConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=20,
    encoder_reparameterization_type="LSTM",
    encoder_hidden_size=256,
    encoder_num_layers=2,
    encoder_dropout=0.0,
)

model = get_peft_model(base_model, config_mlp)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment