Principle:Iamhankai Forest of Thought Dataset Loading
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Preprocessing |
| Last Updated | 2026-02-14 03:00 GMT |
Overview
A data ingestion pattern that loads, filters, and slices benchmark datasets for structured evaluation of reasoning systems.
Description
Dataset Loading handles the transformation of raw benchmark data (JSONL, Parquet) into structured HuggingFace Dataset objects ready for tree-search evaluation. The pattern supports multiple math reasoning benchmarks (GSM8K, MATH500, AIME) with dataset-specific field mappings (question/problem/Problem for queries, answer/Answer for ground truth). It provides difficulty-level filtering for MATH datasets and range-based slicing for distributed processing.
Usage
Use this principle at the start of any FoT benchmark evaluation workflow, after argument parsing and model loading. Dataset loading is required before the forest construction step can begin iterating over problems.
Theoretical Basis
The loading pattern implements dataset abstraction: heterogeneous benchmark formats are normalized to a uniform interface. Key design:
- Format detection: Automatically handles JSONL and Parquet input formats
- Field mapping: Dataset-specific accessors (e.g., GSM8K uses question/answer, MATH uses problem/answer, AIME uses Problem/Answer)
- Level filtering: MATH benchmark problems are stratified by difficulty (Levels 1-5); filtering enables targeted evaluation
- Range slicing: start_id/end_id parameters support parallel evaluation across multiple GPUs or jobs