Implementation:Microsoft BIPIA VicunaWithSpecialToken

Field	Value
Sources	Repo
Domains	NLP, Security, Evaluation
Last Updated	2026-02-14

Overview

Concrete tool for evaluating finetuned defense models with special data boundary tokens provided by the BIPIA defense module.

Description

VicunaWithSpecialToken extends the Vicuna model wrapper class to override process_fn() with custom prompt construction that inserts <data> and </data> markers around external content.

The process_fn constructs a Vicuna chat template:

A chat between a curious user and an artificial intelligence assistant.
The assistant gives helpful, detailed, and polite answers to the user's questions.
USER: ... <data> [context] </data> ... ASSISTANT:

The method finds the context string within the formatted prompt and wraps it with the special tokens.

Usage

Instantiate with a config pointing to the finetuned model checkpoint. Use for both attacked and clean evaluation.

Code Reference

Source: BIPIA repo, File: defense/white_box/eval.py, Lines: L134–159

Signature

class VicunaWithSpecialToken(Vicuna):
    """Inherits from bipia.model.llama.Vicuna"""
    require_system_prompt = False

    def process_fn(self, example: dict, prompt_construct_fn: Callable) -> dict:
        ...

Also: inference() function at L162–343 orchestrates the full evaluation pipeline.

Import: from defense.white_box.eval import VicunaWithSpecialToken (or instantiated within eval.py)

I/O Contract

Inputs

Parameter	Type	Required	Description
config	dict	Yes	With `model_name` pointing to finetuned checkpoint path, `load_8bit=False`
example	dict	Yes	Dataset row with context, question, etc.
prompt_construct_fn	Callable	Yes	Partial of `construct_prompt`

Outputs

Modified example dict with a "message" field containing the Vicuna-format prompt with <data>/</data> wrapped context.

Usage Examples

Instantiation with a finetuned model path:

from omegaconf import OmegaConf

config = OmegaConf.create({
    "model_name": "/path/to/finetuned",
    "load_8bit": False,
})

model = VicunaWithSpecialToken(config)

CLI command for running the full evaluation pipeline:

python defense/white_box/eval.py \
    --model_name_or_path /path/to/finetuned \
    --dataset_name qa \
    --add_special_context_token \
    --context_data_file /path/to/context_data.jsonl \
    --attack_data_file /path/to/attack_data.jsonl \
    --output_path /path/to/output/

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment