Heuristic:Princeton nlp Tree of thought llm Global State Token Counting
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, Optimization |
| Last Updated | 2026-02-14 04:00 GMT |
Overview
Uses module-level global variables to accumulate OpenAI API token counts across all calls for cost estimation without modifying function signatures.
Description
The src/tot/models.py module declares two global variables at the top level: completion_tokens and prompt_tokens, both initialized to 0. Every call to chatgpt() increments these counters from the API response's usage data. The gpt_usage() function reads these globals to compute a cost estimate based on per-model pricing. This global accumulator pattern avoids passing a counter object through the entire call stack.
Usage
The token counters are consulted at the end of each task iteration via gpt_usage(args.backend) in run.py, and the running totals are stored in the JSON log file. This allows tracking cumulative API cost across an entire experiment.
The Insight (Rule of Thumb)
- Action: Declare `completion_tokens = prompt_tokens = 0` at module level. Increment in every API call via `global completion_tokens, prompt_tokens`.
- Value: Zero-overhead cost tracking with no changes to function signatures or return types.
- Trade-off: Global mutable state makes the module non-reentrant. Cannot run two experiments in parallel within the same process and get separate token counts. Counters never reset between solve() calls within a single run() invocation (which is the intended behavior — cumulative tracking).
Reasoning
In a research codebase where experiments run as single-threaded scripts, global state is an acceptable simplification. The alternative — passing a usage tracker object through gpt → chatgpt → solve → run — would add complexity to every function signature for a cross-cutting concern. The global pattern ensures every API call is counted, including those from task-specific code like TextTask.test_output and MiniCrosswordsEnv.prompt_status that call gpt() directly.
Code Evidence
Global declaration from `src/tot/models.py:5`:
completion_tokens = prompt_tokens = 0
Accumulation from `src/tot/models.py:27-36`:
def chatgpt(messages, model="gpt-4", temperature=0.7, max_tokens=1000, n=1, stop=None) -> list:
global completion_tokens, prompt_tokens
outputs = []
while n > 0:
cnt = min(n, 20)
n -= cnt
res = completions_with_backoff(model=model, messages=messages, temperature=temperature, max_tokens=max_tokens, n=cnt, stop=stop)
outputs.extend([choice.message.content for choice in res.choices])
# log completion tokens
completion_tokens += res.usage.completion_tokens
prompt_tokens += res.usage.prompt_tokens
return outputs
Cost calculation from `src/tot/models.py:39-47`:
def gpt_usage(backend="gpt-4"):
global completion_tokens, prompt_tokens
if backend == "gpt-4":
cost = completion_tokens / 1000 * 0.06 + prompt_tokens / 1000 * 0.03
elif backend == "gpt-3.5-turbo":
cost = completion_tokens / 1000 * 0.002 + prompt_tokens / 1000 * 0.0015
elif backend == "gpt-4o":
cost = completion_tokens / 1000 * 0.00250 + prompt_tokens / 1000 * 0.01
return {"completion_tokens": completion_tokens, "prompt_tokens": prompt_tokens, "cost": cost}