Principle:Openai Evals Self Consistency Prompting
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Prompting Strategy, Ensemble Methods |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A principle that improves answer reliability by generating multiple independent reasoning paths and selecting the most consistent answer through consensus, building on chain-of-thought prompting with ensemble-style aggregation.
Description
Self-consistency prompting addresses a fundamental limitation of single-sample generation: any individual reasoning chain may contain errors, hallucinations, or suboptimal reasoning paths. By sampling multiple independent reasoning chains for the same question and aggregating their answers, self-consistency leverages the statistical property that correct reasoning paths tend to converge on the same answer while incorrect paths tend to diverge.
The Openai Evals framework implements self-consistency through the SelfConsistencySolver class, which supports two distinct consensus modes:
- Majority voting mode: The most straightforward aggregation strategy. Multiple reasoning chains are generated, each producing a final answer. The answer that appears most frequently across all chains is selected as the final output. This mode is computationally efficient and works well when answers are discrete and easily comparable (e.g., multiple-choice letters, numerical values).
- Judge-based mode: A more sophisticated approach where an LLM judge evaluates the set of generated reasoning chains and answers, then determines which answer is best supported. This mode handles cases where simple string matching is insufficient -- for example, when answers are semantically equivalent but differently worded, or when the quality of reasoning matters as much as the final answer.
The process follows these steps:
- Sample generation -- The underlying solver (typically a CoT solver) is invoked n times with sampling (temperature > 0) to produce diverse reasoning chains.
- Answer extraction -- Each reasoning chain yields a final answer.
- Consensus determination -- Either majority voting or judge-based evaluation selects the final answer.
Key design considerations:
- Sample count (n): More samples increase robustness but also increase computational cost. Typical values range from 5 to 40.
- Temperature: Higher temperatures produce more diverse reasoning paths, which is essential for self-consistency to be effective. If temperature is too low, all paths will be nearly identical, defeating the purpose.
- Answer normalization: Before voting, answers may need normalization (lowercasing, stripping whitespace, resolving equivalent forms) to ensure correct counting.
Usage
Apply self-consistency prompting when:
- The task involves complex reasoning where a single chain of thought may err, but the correct answer is likely to emerge across multiple attempts.
- You need higher confidence in the model's output and can afford the additional computational cost.
- The task has discrete, comparable answers (ideal for majority voting) or requires nuanced judgment (use judge-based mode).
- You are working with high-stakes evaluations where accuracy matters more than latency or cost.
- The underlying solver already uses chain-of-thought prompting, and you want to further boost its accuracy.
Avoid self-consistency when:
- The task is simple enough that a single sample is reliably correct.
- Computational budget is tightly constrained.
- The task requires creative or open-ended generation where there is no single correct answer.
Theoretical Basis
Self-consistency is grounded in ensemble theory and the wisdom of crowds principle from statistics. Wang et al. (2022) demonstrated that marginalizing over diverse reasoning paths significantly outperforms greedy decoding of a single chain.
Formal framework:
Given question q, sample n reasoning chains:
(r_1, a_1), (r_2, a_2), ..., (r_n, a_n) ~ P(reasoning, answer | q)
where r_i is the reasoning chain and a_i is the extracted answer.
Majority voting:
final_answer = argmax_a count(a_i == a for i in 1..n)
This is equivalent to a maximum a posteriori (MAP) estimate under the assumption that each sample is an independent draw from the model's answer distribution:
P(answer = a | q) ≈ (1/n) * sum(1[a_i == a] for i in 1..n)
Judge-based consensus:
final_answer = judge_model.evaluate(q, [(r_1, a_1), ..., (r_n, a_n)])
The judge considers both the reasoning quality and answer frequency to make a determination. This is particularly valuable when:
a_1 = "approximately 3.14"
a_2 = "pi"
a_3 = "3.14159"
These are semantically identical but would be counted as three different answers by majority voting.
Why self-consistency works:
The key assumption is that correct reasoning paths are more likely to converge than incorrect ones. If the model has a probability p > 0.5 of producing the correct answer on any single attempt, then by the law of large numbers, majority voting with n samples will converge to the correct answer as n increases:
P(majority_vote_correct) = sum(C(n,k) * p^k * (1-p)^(n-k) for k in ceil(n/2)..n)
Even when p < 0.5, self-consistency can still help if incorrect answers are distributed across multiple wrong options while the correct answer is the single most common one.