Implementation:Openai Evals FewShotSolver

Knowledge Sources	Openai_Evals
Domains	Evaluation, Solvers
Last Updated	2026-02-14 10:00 GMT

Overview

Concrete tool for few-shot in-context learning via example prepending provided by the evals library.

Description

FewShotSolver is a subclass of NestedSolver that enhances a base solver by prepending example input-output interactions to the prompt. It loads training data from a JSONL file (specified by train_jsonl), randomly samples n_shots examples using a deterministic seed, and inserts them as user-assistant message pairs before the current test input.

During initialization, FewShotSolver validates that the training data contains at least n_shots examples and that the data conforms to one of two supported formats: either a list of message dictionaries (each with "role" and "content" keys) or a dictionary with "input" and "ideal" keys. The training data is converted into pairs of Message objects (user input and assistant response).

At solve time, the _modify_task_state method performs a contamination check to ensure none of the sampled training examples match the current test input. It then constructs a new TaskState with the few-shot examples prepended to the message history. If repeat_task_description is enabled, the task description is re-inserted between examples as a system message to reinforce the instruction.

The name property delegates to the base solver's name, reflecting that FewShotSolver is a transparent wrapper.

Usage

Import FewShotSolver when you need to provide in-context examples to a solver before it answers evaluation questions. This is particularly useful for tasks where models benefit from seeing example input-output pairs, such as classification, translation, or structured output generation.

Code Reference

Source Location

• Repository: Openai_Evals
• File: evals/solvers/nested/fewshot_solver.py
• Lines: 1-110

Signature

class FewShotSolver(NestedSolver):
    def __init__(
        self,
        train_jsonl: str,
        n_shots: int,
        base_solver: SolverSpec,
        repeat_task_description: bool = False,
        postprocessors: list[str] = [],
        registry: Any = None,
        seed: int = 121123,
    ):
        ...

    @property
    def base_solver(self) -> Solver:
        ...

    def _solve(self, task_state: TaskState, **kwargs) -> SolverResult:
        ...

    def _modify_task_state(self, task_state: TaskState) -> TaskState:
        ...

    @property
    def name(self) -> str:
        ...

Import

from evals.solvers.nested.fewshot_solver import FewShotSolver

I/O Contract

Inputs

Name	Type	Required	Description
train_jsonl	str	Yes	Path to a JSONL file containing training examples. Each line must be either a list of message dicts or a dict with "input" and "ideal" keys.
n_shots	int	Yes	Number of few-shot examples to sample and prepend to the prompt.
base_solver	SolverSpec	Yes	Specification for the nested solver that will process the augmented prompt.
repeat_task_description	bool	No	If True, re-inserts the task description as a system message between each few-shot example (except before the first). Defaults to False.
postprocessors	list[str]	No	List of postprocessor names to apply to solver output. Defaults to an empty list.
registry	Any	No	Registry object for resource lookup.
seed	int	No	Random seed for deterministic example sampling. Defaults to 121123.
task_state	TaskState	Yes	The current evaluation task state, passed to _solve.

Outputs

Name	Type	Description
SolverResult	SolverResult	The result from the base solver after processing the few-shot-augmented prompt.
name	str	Delegates to base_solver.name, making the wrapper transparent.

Usage Examples

from evals.solvers.nested.fewshot_solver import FewShotSolver
from evals.solvers.solver import SolverSpec

# Define FewShotSolver via YAML-style config (typical usage)
# solver:
#   class: evals.solvers.nested.fewshot_solver:FewShotSolver
#   args:
#     train_jsonl: /path/to/train.jsonl
#     n_shots: 5
#     base_solver:
#       class: evals.solvers.openai_solver:OpenAISolver
#       args:
#         model: gpt-4

# Programmatic usage
base_spec = SolverSpec(
    class_name="evals.solvers.openai_solver:OpenAISolver",
    args={"model": "gpt-4"},
)

solver = FewShotSolver(
    train_jsonl="/path/to/train.jsonl",
    n_shots=5,
    base_solver=base_spec,
    repeat_task_description=False,
    seed=42,
)

result = solver(task_state)
print(result.output)  # answer from base solver with few-shot context

Related Pages

• Environment:Openai_Evals_Python_Runtime