Principle:Sail sg LongSpec Benchmark Data Preparation
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Benchmarking, NLP |
| Last Updated | 2026-02-14 05:00 GMT |
Overview
Principle for loading and preparing benchmark evaluation datasets (LongBench and AIME) with task-specific prompt templates and context length filtering.
Description
Benchmark Data Preparation transforms raw evaluation datasets into model-ready inputs for measuring speculative decoding performance. Two benchmark suites are supported:
- LongBench: Long-context NLP tasks stored as JSONL files. Five tasks are evaluated: government report summarization (gov_report), meeting Q&A (qmsum), multi-source news summarization (multi_news), code completion (lcc), and repository-level code completion (repobench-p).
- AIME: Mathematical reasoning problems from the "AI-MO/aimo-validation-aime" HuggingFace dataset, filtered by problem ID range. Used with the QwQ-32B model for chain-of-thought reasoning evaluation.
The preparation involves:
- Loading raw data from JSONL files or HuggingFace datasets
- Filtering by context length (1200 < len <= context_length for LongBench)
- Formatting prompts using task-specific templates
- Tokenizing and padding for model input
Usage
Use when setting up evaluation runs for GLIDE speculative decoding. The benchmark data is loaded once per evaluation session and iterated for all samples.
Theoretical Basis
Benchmark evaluation requires consistent data preparation to ensure fair comparisons across methods (tree, sequential, vanilla). The filtering step removes samples that are too short (< 1200 tokens) or exceed the model's context window, ensuring all methods are evaluated on the same data subset.
LongBench task templates:
# Abstract template patterns (not actual implementation)
dataset2prompt = {
"gov_report": "Summarize the report: {context}",
"qmsum": "Answer the question: {input}\nBased on: {context}",
"multi_news": "Summarize the news articles: {context}",
"lcc": "Complete the code: {context}",
"repobench-p": "Complete the code: {context}",
}