Implementation:Huggingface Peft CPTConfig

Knowledge Sources	Huggingface_Peft
Domains	Deep_Learning, Parameter_Efficient_Finetuning
Last Updated	2026-02-07 14:00 GMT

Overview

Concrete configuration dataclass for the CPT (Context-aware Prompt Tuning) PEFT method provided by the Huggingface PEFT library.

Description

CPTConfig controls the hyperparameters for Context-aware Prompt Tuning, introducing parameters for token type masks, prompt tuning initialization, loss weighting, and projection settings. It extends PromptLearningConfig and is restricted to CAUSAL_LM task types. The configuration automatically derives num_virtual_tokens from the length of cpt_token_ids.

Usage

Import and use CPTConfig when you want to apply Context-aware Prompt Tuning to a causal language model, particularly when you need fine-grained control over prompt token types, weighted loss decay, and projection epsilon values for input and format projections.

Code Reference

Source Location

• Repository: Huggingface_Peft
• File: src/peft/tuners/cpt/config.py
• Lines: 22-100

Signature

@dataclass
class CPTConfig(PromptLearningConfig):
    cpt_token_ids: Optional[list[int]] = None
    cpt_mask: Optional[list[int]] = None
    cpt_tokens_type_mask: Optional[list[int]] = None
    opt_weighted_loss_type: Optional[Literal["none", "decay"]] = "none"
    opt_loss_decay_factor: Optional[float] = 1.0
    opt_projection_epsilon: Optional[float] = 0.1
    opt_projection_format_epsilon: Optional[float] = 0.1
    tokenizer_name_or_path: Optional[str] = None

Import

from peft import CPTConfig

I/O Contract

Inputs

Name	Type	Default	Description
cpt_token_ids	Optional[list[int]]	None	Token IDs used for CPT prompts. Defaults to [0] if not set.
cpt_mask	Optional[list[int]]	None	Mask applied to CPT tokens. Defaults to all-ones if not set.
cpt_tokens_type_mask	Optional[list[int]]	None	Mask indicating the type of each CPT token. Defaults to all-ones if not set.
opt_weighted_loss_type	Optional[Literal["none", "decay"]]	"none"	Type of weighted loss: 'none' or 'decay'.
opt_loss_decay_factor	Optional[float]	1.0	Factor for exponential decay in loss weighting.
opt_projection_epsilon	Optional[float]	0.1	Epsilon value for input projection.
opt_projection_format_epsilon	Optional[float]	0.1	Epsilon value for format projection.
tokenizer_name_or_path	Optional[str]	None	Tokenizer for prompt tuning initialization, used only if prompt_tuning_init is TEXT.

Note: CPTConfig inherits additional fields from PromptLearningConfig, including task_type (must be CAUSAL_LM).

Outputs

Name	Type	Description
instance	CPTConfig	Configuration object to pass to get_peft_model

Usage Examples

Basic Configuration

from peft import CPTConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("model-name")
config = CPTConfig(
    task_type="CAUSAL_LM",
    cpt_token_ids=[0, 1, 2, 3, 4],
    opt_weighted_loss_type="decay",
    opt_loss_decay_factor=0.9,
)
model = get_peft_model(model, config)

Related Pages

• Environment:Huggingface_Peft_Python_Core_Dependencies