Principle:Microsoft BIPIA Few Shot Example Construction

Field	Value
Sources	BIPIA paper
Domains	NLP, In_Context_Learning, Defense
Last Updated	2026-02-14

Overview

An in-context learning technique that constructs demonstration examples from training data to teach LLMs the expected behavior of ignoring injected attacks in external content.

Description

Few-shot example construction selects random examples from the training set where the model should have responded to the original task while ignoring the injected attack. Each example pairs a user prompt (with bordered external content containing an attack) with the correct assistant response (the original task answer, not the attack response). These examples are prepended to test prompts to prime the model.

The construction process operates as follows:

1. A configurable number of training examples (num_examples) are sampled at random using a seeded RNG for reproducibility.
2. For each sampled example, the user prompt is generated via prompt_construct_fn and then processed through add_border(), which wraps the external content (context) with delimiter strings.
3. The expected assistant response is generated via response_construct_fn, which extracts the ground-truth answer (the ideal field) from the training example.
4. The user prompt and assistant response are stored as paired messages that will be inserted into the final prompt between the system prompt and the actual test query.

This approach ensures that the demonstration examples faithfully represent the desired behavior: answering the user's original question while treating the injected attack as inert content within the external data.

Usage

Use when building few-shot defense prompts for API-based LLMs. Requires a training split with known correct responses. The technique is applicable to any task domain supported by BIPIA (email, QA, table, abstract, code) and works with both chat-mode and completion-mode models. The number of examples and the random seed are configurable via command-line arguments (num_examples and fewshot_seed).

Theoretical Basis

In-context learning allows LLMs to learn patterns from demonstration examples without parameter updates. By showing examples where attacks are present but ignored, the model learns the behavioral pattern of attending to the user question rather than injected instructions.

The core insight is that few-shot demonstrations provide behavioral priming: the model observes a consistent pattern of (attack-containing input, correct-task-output) pairs and generalizes this pattern to new inputs. This leverages the LLM's ability to perform pattern completion over its context window.

Pseudocode of the construction flow:

for i in random_sample(training_set, num_examples):
    user_prompt   = add_border(prompt_construct_fn(example_i), example_i["context"])
    assistant_resp = response_construct_fn(example_i)
    store_pair(user_prompt, assistant_resp)

The resulting pairs act as implicit instructions: rather than telling the model to ignore attacks (an explicit instruction that may be circumvented), the examples show the model what ignoring attacks looks like in practice. This demonstration-based approach complements explicit defenses such as ignore-guidance prompts and border delimiters.

Related Pages

• Implementation:Microsoft_BIPIA_FewShotChatGPT35Defense_Construct_Example