ACE Journal

Test-Time Compute Scaling via Self-Consistency and Chain-of-Thought Aggregation

Who this is for: ML engineers and applied researchers deploying large language models on reasoning-heavy tasks (math, code generation, formal verification, multi-step QA) who want to improve output quality without retraining. You should be comfortable with sampling APIs (OpenAI, Hugging Face, or similar) and the idea of chain-of-thought prompting. You do not need to train reward models to use the basic version; best-of-N with a verifier is an optional extension discussed later.

The problem with the single best guess

When you call a language model with temperature=0 (greedy decoding), you get one answer. For many tasks, that is fine. For tasks where reasoning is the hard part, it is an unnecessarily thin slice of what the model knows.

Think about a hard multi-step arithmetic problem. The model has seen a distribution of solution strategies during training. On any given forward pass, the probability mass over reasoning paths is spread: some paths lead to the right answer via clean algebraic steps, others make sign errors in step three, others conflate variables. Greedy decoding picks the single highest-probability token at each step, which commits to one path and one path only. If that path contains a subtle error, the final answer is wrong and nothing rescues it.

The insight behind self-consistency, formalized by Wang et al. [1] in 2022, is that the correct answer tends to be reached by many different reasoning paths while incorrect answers tend to be idiosyncratic. Errors are noisy; correctness is convergent. If you sample 20 independent reasoning traces, count the final answers, and take the plurality, you are effectively marginalizing over the space of reasoning strategies the model knows. The noise in individual traces cancels, and the signal concentrates on the right answer.

This is not a new idea in probabilistic inference, but its application to language model decoding is both simple and surprisingly effective.

Mechanics: sampling, aggregation, and the compute curve

The self-consistency procedure has three steps, shown in Figure 1.

flowchart LR
    P[Prompt + CoT instruction] --> S1[Sample 1\ntemp=0.7]
    P --> S2[Sample 2\ntemp=0.7]
    P --> SN[Sample N\ntemp=0.7]
    S1 --> A1[Answer A]
    S2 --> A2[Answer B]
    SN --> AN[Answer A]
    A1 --> V[Majority vote\naggregator]
    A2 --> V
    AN --> V
    V --> OUT[Final answer: A]

Figure 1. The self-consistency loop: N independent chain-of-thought completions are sampled at non-zero temperature, their final answers are extracted and tallied, and the plurality answer is returned. Note that two paths here reached answer A via different intermediate steps; a single path that produced B is overruled.

Sampling. You run N forward passes, all from the same prompt, at a non-zero temperature (typically 0.5 to 0.8). Temperature above zero is essential: with temperature=0 every sample is identical and voting is pointless. The stochasticity in the sampling process is what generates the diversity of reasoning paths.

Marginalizing over reasoning paths. The formal interpretation is that you are approximating the marginalized probability of each final answer by integrating out the intermediate reasoning steps. If the model assigns high probability to the correct answer across many reasoning routes, that answer will dominate the vote even if some individual routes are wrong. The vote is a Monte Carlo estimator of that marginalized probability.

Majority vote vs. weighted vote. The simplest aggregator counts votes. A more principled variant weights each candidate answer by the model’s log-probability of the full completion that produced it, so higher-confidence paths count more. Wang et al. [1] found that the simple plurality vote was competitive with the weighted variant in most settings, which is useful because computing log-probabilities is not always available through production APIs.

Best-of-N with a verifier. A distinct but related strategy generates N candidates and uses a separate verifier or reward model to score them, returning the top-scoring one rather than the plurality. This is more expensive (you need the verifier) and more powerful when the verifier is calibrated, because it can distinguish among candidates that all received one vote. It is also the approach used in more advanced systems such as the process reward model (PRM) work described in [4].

The compute-accuracy curve. The gain from more samples is roughly logarithmic. Going from one sample to eight usually yields a noticeable improvement. Going from 32 to 64 yields a smaller increment, and from 64 to 128 smaller still. The practical consequence is that most of the available gain is captured at 16 to 32 samples for typical tasks, and beyond 64 you are mostly paying for diminishing marginal accuracy. Figure 2 captures this shape qualitatively.

Samples (N) Relative accuracy gain
1 baseline
2 small jump
4 moderate gain
8 noticeable gain
16 good gain
32 diminishing returns
64 marginal gain
128 near-plateau

Figure 2. Qualitative shape of the accuracy-vs-sample-count curve; actual values depend on model and task. The characteristic logarithmic compression means the cost doubles between each labeled x-axis point while accuracy gains shrink. This curve is illustrative, not derived from specific benchmark numbers.

Worked example: a Python self-consistency loop

The following snippet shows a minimal self-consistency implementation against an OpenAI-compatible chat API. It samples N completions, extracts the final numerical answer from each, and returns the majority. The extraction logic is intentionally simple; real deployments need something more robust, which the next section addresses.

import re
from collections import Counter
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from env

def sample_cot(prompt: str, n: int = 16, temperature: float = 0.7) -> list[str]:
    response = client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[{"role": "user", "content": prompt}],
        n=n,
        temperature=temperature,
        max_tokens=512,
    )
    return [choice.message.content for choice in response.choices]

def extract_answer(completion: str) -> str | None:
    """Pull the last number that appears after 'answer' or at end of text."""
    match = re.search(r"(?:answer is|answer:|=)\s*([-\d,.]+)", completion, re.I)
    if match:
        return match.group(1).strip().replace(",", "")
    numbers = re.findall(r"[-\d]+(?:\.\d+)?", completion)
    return numbers[-1] if numbers else None

def self_consistent_answer(prompt: str, n: int = 16) -> str:
    completions = sample_cot(prompt, n=n)
    answers = [extract_answer(c) for c in completions]
    valid = [a for a in answers if a is not None]
    if not valid:
        return "no_answer"
    majority, count = Counter(valid).most_common(1)[0]
    print(f"Vote distribution: {Counter(valid).most_common(5)}")
    return majority

Two things to notice. First, the n parameter in the chat completions API requests all samples in one API call, so the network round-trip overhead is not multiplied by N. Second, extract_answer is the fragile part: see the next section.

The hard part

Self-consistency sounds almost too simple, and in some ways it is. The real complexity lives in three places practitioners discover after the first working demo.

Answer extraction brittleness. Majority voting only works if you can reliably parse the final answer out of free-form reasoning text and normalize it into a comparable form. “The answer is 42”, “= 42”, “42.0”, and “forty-two” should all count as the same answer. When the model produces structured output or follows a strict format string (“Final answer: …”), extraction is tractable. When it produces prose reasoning that trails off without a clean delimiter, you will lose votes to parse failures and skew the distribution. A common fix is to add a format instruction to the prompt (“end your response with ‘Answer: '") and validate compliance before counting.

Diminishing returns and cost. At 16 samples of 512 tokens each, you are spending roughly 8,000 tokens of output for one final answer. If your task requires many such calls, the economics degrade fast. For a pipeline that calls a model hundreds of times, a 16x cost multiplier is not a rounding error. The practical approach is to apply self-consistency selectively: classify queries as easy (low uncertainty, greedy decoding is probably right) or hard (high uncertainty or verifiable, worth sampling), and spend the budget on the latter.

When self-consistency helps vs. hurts. Self-consistency assumes errors are stochastic: each wrong path fails in a different way, so errors don’t cluster. This assumption holds well for arithmetic and symbolic reasoning, where the mistake space is large and diverse. It fails for tasks where the model has a systematic wrong belief. If the model consistently misapplies a rule (say, it always treats “not A or B” as “not A and not B”), all 20 samples produce the same wrong answer and the majority vote confidently returns it. More samples do not help; the model needs information it does not have. Self-consistency is not a substitute for capability.

Calibration of confidence. The plurality count is sometimes mistaken for a calibrated probability. It is not. A 12/20 vote for an answer does not mean 60% confidence in any well-founded sense, because the samples are correlated (they come from the same model with the same priors) and the answer space is not uniformly covered. Teams that use the vote fraction as a reliability signal in downstream logic often find it poorly calibrated in the tails.

Integration and day-to-day use

The most common practical pattern is task routing: apply cheap greedy decoding by default and trigger self-consistency when a task meets criteria for high-value, verifiable output. Figure 3 shows this routing architecture.

flowchart TD
    Q[Inbound query] --> CL{Classifier:\nreasoning-heavy?}
    CL -- No --> G[Greedy decode\n1 sample]
    CL -- Yes --> SC[Self-consistency\n16-32 samples]
    G --> OUT[Response]
    SC --> AE[Answer extraction\n+ majority vote]
    AE --> OUT
    OUT --> VER{Verifiable?\n e.g. code, math}
    VER -- Yes --> CHECK[Run verifier /\nunit test]
    CHECK -- Pass --> DONE[Deliver]
    CHECK -- Fail --> ESC[Escalate or\nresample]
    VER -- No --> DONE

Figure 3. A practical routing architecture: cheap greedy decoding for the common case, self-consistency sampling reserved for reasoning-heavy queries, and an optional verification step for tasks where correctness can be checked programmatically.

For code generation, the verifier can be a unit test suite, which is more reliable than vote counting because it actually checks correctness rather than popularity. This is a useful asymmetry: on code tasks, pass@k (does any of k samples pass the tests?) is often more informative than majority voting, and it pairs naturally with self-consistency by giving you k chances to find a passing solution.

At scale, the token budget matters. A simple safeguard is to cap total output tokens per query rather than capping samples, since some queries will produce short completions and you can afford more of them. Logging the vote distribution (as the worked example does) is worth keeping in production: a flat distribution where every sample produces a different answer is a signal that the model is confused, not uncertain, and you should surface that for review rather than returning a low-confidence plurality.

Self-consistency occupies a specific position in the broader test-time compute landscape and it is worth being clear about what it is not.

Greedy decoding. The baseline it replaces. Greedy decoding is optimal when you want the single highest-probability completion and the task is one where the first-rank token at each step is reliable. It becomes suboptimal on multi-step reasoning tasks precisely because the argmax at each step can commit to a subtree of low overall probability. Self-consistency avoids this commitment by sampling.

Best-of-N with a verifier or reward model. A stronger variant when you have a scorer. Best-of-N generates N candidates and returns the one with the highest verifier score rather than the plurality answer. When the verifier is accurate (a trained reward model, a formal checker, or a test suite), this outperforms majority vote because it can distinguish among candidates that happen to agree on the wrong answer. Cobbe et al. [4] showed that training a verifier on outcome-labeled math solutions significantly outperforms majority voting alone at the same N. The cost is that you need the verifier, which requires either a separately trained model or a task with a built-in correctness signal.

Process reward models (PRMs). Rather than scoring final answers, a PRM assigns credit to individual reasoning steps. This enables guided tree search: at each intermediate step, the PRM scores candidate next steps and the search expands the most promising branch. The approach underlying the OpenAI o1 technical report [5] relies on this idea. PRMs are substantially more expensive to train and to run than majority voting, but they can catch errors early in the reasoning chain rather than only at the output, and they generalize better to tasks where many different final answers could be acceptable.

Tree of Thoughts. Yao et al. [6] formalized a general framework for deliberate reasoning via tree search over thought steps, where a model evaluates each partial solution and decides whether to continue, backtrack, or explore a different branch. Tree of Thoughts subsumes self-consistency (breadth-first sampling) as a special case and adds backtracking. The tradeoff is complexity: implementing a coherent tree search over model completions requires either a strong value model or human evaluation at each node, which limits practical applicability outside research settings.

Chain-of-thought prompting. The underlying technique that makes self-consistency possible [2]. Without chain-of-thought prompting, most language models produce answers too short and too direct for there to be meaningful diversity across samples. Chain-of-thought creates the reasoning trace that carries the variance self-consistency exploits.

Where it breaks

Self-consistency is a useful tool with clear failure modes. Here are six.

Systematic model errors. As discussed above: when the model consistently applies a wrong rule or has a wrong fact, more samples reinforce the wrong answer. This is the most important failure mode because it is invisible without ground-truth labels.

Low answer cardinality. On tasks where there are only two or three plausible answers, self-consistency provides little benefit because the vote is split among a small set and noise has limited room to cancel. Binary classification tasks are a near-degenerate case.

Tasks without extractable final answers. Majority voting requires a discrete, comparable answer. Open-ended generation (summarization, style transfer, creative writing) does not produce a natural aggregation target. You cannot majority-vote a paragraph. Some researchers have explored clustering embeddings of completions and returning the centroid, but this is a looser notion of consistency and the quality benefit is less established.

High output variance and long contexts. At very high temperature or on tasks requiring very long reasoning chains, the variance across samples can be so large that the vote distribution is nearly flat. The plurality then reflects statistical noise, not signal. Lowering temperature partially helps but reduces the diversity that makes the method work in the first place.

Correlated failures. The statistical argument for self-consistency assumes approximate independence among samples. In practice, samples from the same model share the same training biases, the same knowledge gaps, and the same systematic tendencies. The correlation structure can concentrate failures in ways that simple vote counting does not account for.

Cost at production scale. At N=32 and an average completion length of 400 tokens, you are generating 12,800 output tokens per query. At typical API pricing as of early 2026, this is roughly 25 to 64 times the cost of a greedy decode, depending on the model tier. For internal tools or batch workflows, this is often acceptable. For real-time user-facing applications, it requires careful task selection and budget caps.

Further reading

Start with the original self-consistency paper [1] and the chain-of-thought prompting paper it builds on [2]; together they establish the theoretical motivation and the empirical baseline. The Let’s Verify Step by Step paper from OpenAI [10] is the right next read if you want to understand when a verifier outperforms majority voting. For tree search approaches, the Tree of Thoughts paper [6] is the primary reference and is readable without heavy prerequisites. The Big-Bench Hard benchmark [3] is useful as a task suite that specifically targets multi-step reasoning and is frequently used to evaluate self-consistency methods. Scaling laws references [7] provide context for how test-time scaling compares to training-time scaling in the broader landscape. The Hugging Face blog post on sampling strategies [8] is a practical complement for anyone working with open-source models rather than closed APIs.

Written with AI assistance, editorially reviewed.

References

[1] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-Consistency Improves Chain of Thought Reasoning in Language Models,” arXiv:2203.11171, 2022. https://arxiv.org/abs/2203.11171

[2] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” arXiv:2201.11903, 2022. https://arxiv.org/abs/2201.11903

[3] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei, “Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them,” arXiv:2210.09261, 2022. https://arxiv.org/abs/2210.09261

[4] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training Verifiers to Solve Math Word Problems,” arXiv:2110.14168, 2021. https://arxiv.org/abs/2110.14168

[5] OpenAI, “Learning to Reason with LLMs,” OpenAI Research Blog, 2024. https://openai.com/research/learning-to-reason-with-llms

[6] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of Thoughts: Deliberate Problem Solving with Large Language Models,” arXiv:2305.10601, 2023. https://arxiv.org/abs/2305.10601

[7] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buckaite, E. Cai, L. Rutherford, A. Bousquet, G. Bricken, M. Reing, and A. Mensch, “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022. https://arxiv.org/abs/2203.15556

[8] Hugging Face, “How to Generate Text: Using Different Decoding Methods for Language Generation with Transformers.” https://huggingface.co/blog/how-to-generate

[9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language Models are Few-Shot Learners,” arXiv:2005.14165, 2020. https://arxiv.org/abs/2005.14165

[10] L. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s Verify Step by Step,” arXiv:2305.20050, 2023. https://arxiv.org/abs/2305.20050