Workflow:Testtimescaling Testtimescaling github io Adding a New Paper
| Knowledge Sources | |
|---|---|
| Domains | Academic_Tooling, Content_Curation, LLMs |
| Last Updated | 2025-04-13 00:00 GMT |
Overview
End-to-end process for adding a new test-time scaling research paper to the curated survey, including taxonomy classification and automated citation tracking integration.
Description
This workflow describes the manual process of incorporating a new research paper into the Awesome Test-Time Scaling survey. A contributor identifies a relevant paper, classifies it according to the four-dimensional taxonomy (What, How, Where, How Well), adds its entry to the comparison table in the README, and registers its arXiv ID in the papers configuration file. Once registered, the automated citation tracking pipeline picks up the new paper and includes its citations in the aggregate badge count.
Goal: A new paper entry in the README comparison table with proper taxonomy classification and automatic citation tracking.
Scope: From paper discovery through taxonomy analysis, README table entry, citation registration, and verification.
Strategy: Manual taxonomy classification following the survey framework, combined with automated citation integration via the existing daily pipeline.
Usage
Execute this workflow when a new test-time scaling paper is published or discovered that should be included in the survey. The paper must be relevant to test-time scaling in LLMs and ideally available on arXiv for citation tracking.
Execution Steps
Step 1: Paper Discovery and Relevance Assessment
Identify a new paper related to test-time scaling in LLMs. Verify that it fits within the survey scope by checking whether it addresses at least one dimension of the taxonomy: what to scale, how to scale, where to scale, or how well it scales.
Key considerations:
- The paper should be publicly available (preferably on arXiv)
- It must relate to test-time scaling or test-time compute in LLMs
- Check that the paper is not already listed in the comparison table
Step 2: Taxonomy Classification
Analyze the paper and classify its contributions along the four taxonomy dimensions defined by the survey:
What to Scale: Determine whether the paper uses Parallel, Sequential, Hybrid, or Internal scaling.
How to Scale: Identify which techniques are used across six categories: Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), Stimulation (STI), Search (SEA), Verification (VER), and Aggregation (AGG).
Where to Scale: Note the application domains (Math, Code, Science, Open-Ended, etc.).
How Well: Record the evaluation metrics used (Pass@1, accuracy, token cost, etc.).
Step 3: Add Paper to Comparison Table
Insert a new row in the README comparison table with the paper title, arXiv badge link, and taxonomy classifications. Follow the existing HTML table format with proper column alignment.
Key considerations:
- Match the exact HTML structure of existing table rows
- Include an arXiv badge image linking to the paper
- Use checkmarks or specific technique names in taxonomy columns
- Mark inapplicable categories with the cross symbol
Step 4: Register for Citation Tracking
Add the paper's arXiv ID to the papers.json configuration file (both the root copy and the .github/scripts/ copy). This registers the paper with the automated citation tracking pipeline.
Key considerations:
- Both
papers.jsonfiles must be updated consistently - Each entry requires a title and arxiv_id field
- The daily cron job will automatically pick up the new paper on next run
Step 5: Verify Integration
After committing the changes, verify that the paper appears correctly in the rendered README table and that the citation tracking workflow can fetch its citation count from Semantic Scholar.
Key considerations:
- Check the rendered GitHub Pages site for correct table formatting
- Verify the Semantic Scholar API returns data for the arXiv ID
- The citation badge will update on the next daily cron run