Principle:Princeton nlp Tree of thought llm Experiment Configuration
| Knowledge Sources | |
|---|---|
| Domains | Experiment_Management, CLI_Design |
| Last Updated | 2026-02-14 03:30 GMT |
Overview
A structured approach to defining and parsing all hyperparameters for a Tree of Thoughts experiment through command-line arguments.
Description
Experiment Configuration ensures that every parameter controlling an experiment run—LLM backend, temperature, task selection, search method variants, and sample counts—is explicitly declared, validated, and accessible through a single namespace object. This avoids hard-coded values scattered through the codebase and makes experiments reproducible by encoding the full configuration in the command-line invocation (typically via shell scripts).
The configuration controls two distinct experiment modes:
- ToT BFS mode (default): Uses method_generate, method_evaluate, and method_select to configure the tree search.
- Naive baseline mode (--naive_run): Bypasses tree search, using only prompt_sample and n_generate_sample for direct sampling.
Usage
Use this principle at the entry point of any experiment runner. It should be the first step after program launch, before task instantiation or any computation. The resulting configuration object is passed unchanged to all downstream functions.
Theoretical Basis
Experiment configuration follows the principle of externalizing hyperparameters:
# Abstract pattern
config = parse_arguments(cli_args)
# config is immutable namespace passed to all downstream functions
task = create_task(config.task)
result = run_experiment(config, task)
log_results(result, config)
Key design decisions:
- Enumerated choices: Task names, method types, and model names are constrained to valid options via choices parameter in argparse, catching invalid inputs early.
- Sensible defaults: Most parameters have reasonable defaults, so minimal scripts only need to specify the task name.
- Mode flag: The --naive_run flag acts as a discriminator between two experiment modes that share the same entry point.