Implementation:Princeton nlp SimPO Decontaminate Humaneval

Knowledge Sources	Princeton_nlp_SimPO SimPO
Domains	Data_Quality, NLP, Evaluation
Last Updated	2026-02-08 04:30 GMT

Overview

Concrete tool for filtering HumanEval benchmark contamination from training datasets provided by the SimPO alignment package.

Description

The decontaminate_humaneval function removes training samples that contain content from the OpenAI HumanEval benchmark. It loads HumanEval docstrings and canonical solutions as reference strings, then checks each training sample for substring matches using whitespace-normalized, case-insensitive comparison. Trivially simple solutions (e.g., return x + y) are excluded from the filter to avoid over-aggressive removal. The module also provides helper functions for docstring extraction, dataset column loading, and whitespace normalization.

Usage

Import this module when preparing training data for supervised fine-tuning or preference optimization and you need to ensure the dataset does not contain HumanEval benchmark content. This prevents inflated evaluation scores caused by benchmark leakage into training data.

Code Reference

Source Location

• Repository: Princeton_nlp_SimPO
• File: alignment/decontaminate.py
• Lines: 1-91

Signature

HUMAN_EVAL_STRINGS_OK: List[str]
# Trivially simple solutions kept in training data

FILTER_OUT: Dict[str, List[str]]
# Pre-loaded HumanEval docstrings and canonical solutions

def extract_docstring(prompt: str) -> str:
    """Extract docstring from a HumanEval prompt."""

def human_eval_docstrings() -> List[str]:
    """Load all HumanEval docstrings from the openai_humaneval dataset."""

def load_dataset_column(dataset: str, column: str, split: str, name=None) -> List[str]:
    """Load a single column from a HuggingFace dataset, returning non-empty strings."""

def normalize_whitespace(text: str) -> str:
    """Collapse all whitespace in text to single spaces."""

def decontaminate_humaneval(
    samples: List[Dict[str, Any]],
    text_column: str = "text",
    filter_out: Dict[str, List[str]] = FILTER_OUT,
) -> List[Dict[str, Any]]:
    """
    Filter training samples that contain HumanEval benchmark content.

    Args:
        samples: Batch of training samples (HuggingFace datasets batch format).
        text_column: Column name containing the text to check.
        filter_out: Dict mapping benchmark name to list of filter strings.

    Returns:
        List of booleans: True if sample should be kept, False if contaminated.
    """

Import

from alignment.decontaminate import decontaminate_humaneval

I/O Contract

Inputs

Name	Type	Required	Description
samples	List[Dict[str, Any]]	Yes	Batch of training samples in HuggingFace datasets batch format (dict of lists)
text_column	str	No	Column name containing text to check (default: "text")
filter_out	Dict[str, List[str]]	No	Benchmark name to filter strings mapping (default: FILTER_OUT with HumanEval docstrings and solutions)

Outputs

Name	Type	Description
result	List[bool]	Boolean mask: True for clean samples, False for contaminated samples

Usage Examples

Filtering a HuggingFace Dataset

from datasets import load_dataset
from alignment.decontaminate import decontaminate_humaneval

# 1. Load your training dataset
dataset = load_dataset("your_org/your_dataset", split="train")

# 2. Apply decontamination filter as a batch operation
filtered_dataset = dataset.filter(
    decontaminate_humaneval,
    batched=True,
    batch_size=1000,
    input_columns=["text"],
)

# 3. filtered_dataset now excludes samples containing HumanEval content
print(f"Original: {len(dataset)}, Filtered: {len(filtered_dataset)}")

Using with a Custom Text Column

from alignment.decontaminate import decontaminate_humaneval

# If your dataset uses a different column name for text content
filtered_dataset = dataset.filter(
    decontaminate_humaneval,
    batched=True,
    fn_kwargs={"text_column": "content"},
)

Inspecting the Pre-loaded Filter Strings

from alignment.decontaminate import FILTER_OUT, HUMAN_EVAL_STRINGS_OK

# See what benchmarks are filtered
print("Benchmarks:", list(FILTER_OUT.keys()))
# Output: ['human_eval_docstrings', 'human_eval_solutions']

# See which trivial solutions are excluded from filtering
print("Allowed simple solutions:", HUMAN_EVAL_STRINGS_OK)
# Output: ['return x + y', 'return len(string)', 'return n**2', 'return " ".join(strings)']

Related Pages

This implementation has no environment or heuristic dependencies. It is a pure Python text-processing utility using only the datasets library.