Score Versus Cost Per Task: Measuring the Returns to Reasoning Effort in Frontier Language Models

1. Introduction

For most of the period since large language models (LLMs) became commercial products, their cost was discussed in terms of a single number: the price per million tokens of input and output. That number has fallen quickly. The 2025 AI Index reported that the inference cost of reaching GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024 [1], and Epoch AI found that the price of reaching fixed benchmark milestones fell by between 9-fold and 900-fold per year, depending on the milestone [2]. A later analysis using the same kinds of data found that while the price of a given level of performance falls by roughly 5 to 10 times per year, the price of running the newest frontier models has risen [3].

The reason the two trends diverge is that the amount of work a model does on a task is no longer fixed. Current models from every major developer can reason at length before they answer, and the user can choose how long, through settings named low, medium, high, xhigh or max. Each step up generates more reasoning tokens, which are billed at the output price. The relevant economic quantity for a business is therefore not the price per token but the cost per task, and how that cost trades against the quality of the result. Kapoor et al. argued that agent benchmarks should report cost alongside accuracy for exactly this reason [4], and Erol et al. formalized the expected cost of obtaining one correct answer as an evaluation metric [5].

This paper asks a narrow, practical question: when a firm pays for more reasoning effort, how much better does the model get? In economic terms, it measures the returns to reasoning effort. It does not attempt to rank models overall, to predict future prices, or to evaluate safety or latency. Its readers are assumed to be managers and engineers in small and medium-sized enterprises who must choose a model and a setting for a defined workload.

The paper is organized as follows. Section 2 explains how cost per task arises and how effort settings differ by vendor. Section 3 describes the data, the cost definition and the treatment of uncertainty. Section 4 presents results from a controlled effort study, a coding benchmark, a reasoning benchmark near its ceiling, and an index that links list prices to task costs. Section 5 discusses where extra effort pays, how vendor charts should be read, and the limitations of the evidence. Section 6 turns the findings into a procedure for firms, and Section 7 concludes.

2. Background

2.1 From token prices to cost per task

Vendors price input and output tokens separately, usually with a discount for input that is served from a cache. Reasoning is billed as output: Anthropic states that thinking tokens are billed as output tokens [6], OpenAI bills reasoning tokens as output tokens [7], and Google's output price for Gemini models includes thinking tokens [8]. The cost of one task is the sum over token types of the tokens consumed multiplied by their price:

where N are token counts per task and p are prices per token. A higher effort setting mainly raises N_reason; the prompt and the answer change much less. Because p_out is typically three to five times p_in for the models in this study (Section 4.4), reasoning tokens quickly dominate the bill. Figure 1 illustrates the mechanism schematically.

Two consequences follow from Eq. (1). First, two models with identical list prices can differ several-fold in cost per task, because they spend different numbers of tokens on the same work. Second, tokens are not a common unit: tokenizers differ between vendors, and Anthropic states that its newer models produce approximately 30% more tokens for the same text than earlier ones [6]. Per-token prices can therefore mislead even within one vendor's product line.

2.2 Reasoning-effort settings across vendors

Every vendor in this study exposes some control over reasoning, but the names and number of levels differ (Table 1). Anthropic and OpenAI offer five levels from low to max, and some OpenAI models add a sixth, none; Google offers three for its Gemini 4 models; DeepSeek, Moonshot and Zhipu offer low, high and max; and Meta adds a minimal level. Defaults also differ: a model called through its default setting may already be at medium, high or max. A setting with the same name is not the same amount of work across vendors, so comparisons in this paper are made between steps within one model, not between equally named settings of different models. Table 1 summarizes the settings as published.

2.3 Prior work on cost-aware evaluation

The idea of trading cost against quality is not new. FrugalGPT showed that cascading queries from cheaper to more expensive models could match a strong model's accuracy at much lower cost [9]. Studies of inference scaling found that spending more computation at inference time, by sampling repeatedly or generating longer solutions, can raise accuracy along a predictable curve [10, 11]. The Holistic Agent Leaderboard standardized agent evaluation across models and scaffolds, and found that higher reasoning effort reduced accuracy in most of its runs [12].

Other work points to limits. Reasoning models can spend many tokens on trivial problems with little benefit [13], shorter reasoning chains can be more accurate than longer ones for the same question [14], disabling explicit thinking can remain competitive at matched token budgets [15], and on some tasks longer reasoning lowers accuracy [16]. Accuracy and token use also change in different ways as problems become harder [17]. A counterpoint is that small gains in per-step accuracy compound over long, multi-step tasks, so a gain that looks negligible on a single-answer benchmark may matter for long-running agents [18]. This paper adds current, cross-vendor measurements to that discussion.

3. Data and Method

3.1 Sources

The analysis uses only sources that publish both a quality measure and a measured cost for each model and effort setting, and that make the underlying values available (Table 2). The primary source for the effort question is a study by Vals AI, which ran seven models on the same tasks from its Vals Index at three effort settings and published accuracy, standard errors and cost per test [19, 20]. Vals AI does not sell models, and because the tasks are identical across settings, differences between settings are not confounded by task selection. The study used about half of the tasks in each Vals Index benchmark, so its scores differ slightly from the main leaderboard.

For comparison across models, the paper uses FrontierCode 2.0, a coding benchmark run by Cognition, which publishes a data file with scores and cost per attempt for 31 models at their available settings, 29 of them at two or more settings [21]. ARC-AGI-2, run by the non-profit ARC Prize Foundation on a 120-task semi-private set, provides a reasoning benchmark at retail prices [22–24]. The Artificial Analysis Intelligence Index supplies cost per index task and blended list prices for 27 models [25, 26]. Official price pages supply list prices [6–8], and Arena supplies human-preference ratings with confidence intervals, which measure preference rather than correctness [27]. Arena publishes no prices; its Agent Arena table includes a median cost per task whose method is not documented, so it is not used here.

3.2 Defining cost per task

The cost per task in this paper is the operator's own figure, computed in each case from tokens consumed and list prices as in Eq. (1), with one exception explained in Section 5.2: for Claude Fable 5.2 on FrontierCode, the system-card costs are used. The operators differ in detail: Cognition reports mean spend per attempt including public discounts [21], ARC Prize uses retail pricing [24], and Artificial Analysis weights task costs by the same weights as its index [25]. Because these definitions are not identical, costs from different sources are never placed on the same axis; each figure and table uses one source. Where an accuracy rate is available, the paper also reports cost per correct answer, following Erol et al. [5]:

where a is the fraction of tasks answered correctly. Eq. (2) is applied only to the Vals data, whose score is an accuracy. FrontierCode scores are rubric averages, and dividing cost by them would not give a cost per success.

3.3 Uncertainty

Among the sources that report cost, only Vals AI publishes an uncertainty interval, and none publishes task-level results. The paper therefore approximates the standard error of a score s (in percent) on n independent tasks by the binomial formula:

with a 95% interval of ±1.96 SE. For ARC-AGI-2, n = 120. FrontierCode runs each of its 132 tasks up to five times; the paper reports two bounds, a conservative one with n = 132 tasks and an optimistic one with n = 660 attempts, since attempts on the same task are not independent. The data file suggests that some models were run fewer than five times per task, so the attempt bound is optimistic for those rows. For Vals AI, the published standard errors are used directly. Figure 2 shows how the interval narrows with the number of tasks: near a score of 50% on 132 tasks, the 95% interval is about ±8.5 points, so only large differences can be detected. A change between two effort settings is treated as distinguishable when

Eq. (4) treats the two settings as independent samples, which is conservative because the same tasks are used at each setting; a paired test on task-level data would detect smaller differences. Vals AI reports such paired tests in its own analysis [19]. The cost multiple of a step is the ratio of the costs per task at the two settings, r = C₂ / C₁.

3.4 Collection and verification

Data were collected on November 11, 2026, in four independent streams: Arena ratings, benchmarks that publish cost per task, official vendor prices, and independent indexes. Each recorded value carries the address of the page or data file it was read from. Values were taken from operators' data files by script wherever these existed, and no value was read off a chart image. A separate verifier, who had not seen the collectors' notes, then re-fetched the primary sources and compared 3,573 fields one by one. The check covered every row of the Vals study, ARC-AGI-2, FrontierCode, the price table and the Arena ratings, 65 of the 181 Artificial Analysis rows, and samples of the other benchmarks. It found four errors, three in price data and one in a reference title, which were corrected; no checked benchmark or rating value differed from its source. A second reviewer examined the method, and its findings shaped the choice of anchor benchmarks and the treatment of uncertainty described above. The full dataset, with sources, corrections and both reviews, accompanies this paper.

4. Results

4.1 The same tasks at three effort settings

Figure 3 and Table 3 show the Vals AI study. For every model, accuracy at the second setting tested is higher than at the lowest, but the size of the change differs greatly: for DeepSeek V4.1 Flash, Grok 4.7 and Gemini 4 Argon the first step is within noise, and for Gemini 4 Argon the cost per test fell. Opus 5.5 rose 12.2 points from low to high, Muse Spark 1.3 rose 13.1 points from minimal to high, and GLM-5.3 rose 10.7 points from low to high; each of these gains is distinguishable at 95%. The next step bought much less: Opus 5.5 gained 1.5 points from high to max while its cost per test rose 2.46 times, from $10.76 to $26.49, and GLM-5.3 gained 1.1 points for 1.31 times the cost.

Table 4 lists all fourteen steps. Five are distinguishable at 95%. The final step, from the second-highest to the highest setting tested, was within noise for six of the seven models; the exception was Grok 4.7, which gained 5.8 points from high to xhigh. Two final steps lowered the score slightly (Muse Spark 1.3 and Gemini 4 Argon), and in both cases the change is well within noise. Vals AI's own paired analysis reaches a consistent conclusion: in 22 of 56 combinations of model and benchmark, a lower setting scored as high as the leaderboard setting at lower cost [19].

Cost per correct answer, Eq. (2), makes the trade-off concrete. For Opus 5.5 it rises from $7.62 at low to $15.81 at high and $38.06 at max. At its lowest setting Opus 5.5 retained 80% of its maximum-setting accuracy at 16% of the cost. DeepSeek V4.1 Flash, the cheapest model in the study, reached 61.3% at max for $0.59 per test, about one forty-fifth of the cost of Opus 5.5 at max for an accuracy 8.3 points lower.

Figure 4 plots every step in the two controlled data sets by its cost multiple and score change. In both, most steps cost 1.2 to 2.7 times the previous setting, and the distinguishable gains are concentrated in the lower steps of each model.

4.2 Comparison across models on FrontierCode 2.0

Figure 5 shows FrontierCode 2.0 for five models. Claude Fable 5.2 reached the highest score of any configuration, 49.1% at xhigh, at a cost of $14.19 per task (system-card costs; see Section 5.2). Its max setting scored 45.7% at $44.70, 3.15 times the cost of xhigh. The fall of 3.4 points is not distinguishable: z is −0.56 on the task bound and −1.26 on the attempt bound. Anthropic's system card also notes that 22 of the 660 attempts at max timed out and that the score would be 44.4% if those attempts were counted as zero [28]. The supportable conclusion is that max bought no measurable gain over xhigh at three times the cost, not that it performed worse.

Table 5 tests the eighteen steps of the five models. Only one is distinguishable on the conservative bound, Opus 5.5 from low to medium (+15.4 points for 2.28 times the cost). Five more are distinguishable only on the optimistic attempt bound. Across the 29 models in the data file with two or more settings, 5 of 90 adjacent steps are distinguishable on the task bound and 32 on the attempt bound, and 12 steps lowered the score, none of them distinguishably on the task bound. Between generations, the difference is large: Fable 5.2 at xhigh scored 16.7 points above Fable 5.1 at xhigh (z = 2.8), at 1.63 times the cost per task.

Gemini 4 Carbon shows how a price change can move a model's position. Its price of $2 / $10 per million tokens is introductory and rises to $4 / $20 on December 1, 2026 [8]. Cognition does not state the price date of its costs. If they use the introductory price, the same workload will cost twice as much from December 1, because every token price doubles, moving the best configuration from $4.04 to about $8.09 per task (hollow markers in Figure 5).

4.3 A benchmark near its ceiling: ARC-AGI-2

ARC-AGI-2 is the second version of the ARC-AGI benchmark of abstract reasoning puzzles [23], and current frontier models now score 90% or more on its semi-private set: 18 of the 86 configurations in the data do so. At those scores the 95% interval on 120 tasks is about ±3.9 to ±5.4 points, so the remaining differences cannot be resolved. Figure 6 plots error rate rather than score, on a logarithmic axis, to make the last few points visible. GPT-6 Astra at max scored 95.0% at $1.12 per task, while GPT-6.1 Sol at high scored 91.7% at $0.13; the difference of 3.3 points (z ≈ 1.0) cost 8.3 times as much. In addition, 27 of the 86 costs are lower bounds, because ARC Prize notes that they include recorded usage only [22]. ARC-AGI-2 remains informative for smaller models, whose steps are large, but it cannot locate a point of diminishing returns for frontier models.

4.4 What list prices hide

Figure 7 places each model's blended list price, as computed by Artificial Analysis from a 7:2:1 mix of cached, input and output tokens [25], against its cost per index task at each effort setting. Price and cost are correlated, but within a single model the cost per task varies widely with effort: across the 17 models with two or more reasoning settings (non-reasoning modes excluded), the highest setting cost a median of 3.2 times the lowest, and up to 29.8 times. Artificial Analysis already prices the Gemini 4 models at their post-introductory $4 / $20. Claude Fable 5.2 cost $3.10 per task at low and $26.54 at max on this index, for scores of 53.9 and 64.6. Gemini 4 Carbon cost more at low ($7.19) than at medium ($6.59) while scoring lower, so cost is not always monotonic in effort.

List prices also carry conditions that a comparison must record (Table 6). Several are temporary: Gemini 4 Carbon, Gemini 4 Argon and Gemini 3.8 Flash are at introductory prices, and GPT-5.6 Sol at a promotional price until November 30, 2026 [7, 8]. Some double above a prompt-length threshold, and DeepSeek charges half its peak rate off-peak. A cost comparison is therefore valid only at a stated price date.

5. Discussion

5.1 Where extra effort pays and where it does not

The evidence supports a consistent pattern. Moving up from the lowest setting often produces a large and distinguishable gain, especially for models whose lowest setting reasons very little; the gains of 10 to 15 points for Opus 5.5, Muse Spark 1.3 and GLM-5.3 in the Vals study, and of 15.4 points for Opus 5.5 on FrontierCode, are of this kind. The highest settings rarely produce a gain that can be measured on benchmarks of this size, yet they multiply cost. Across sources, the highest setting cost a median of 2.9 to 4.6 times the lowest (2.9 on FrontierCode, 3.2 on the Artificial Analysis index, 3.4 in the Vals study and 4.6 on ARC-AGI-2).

This pattern is consistent with the literature on overthinking and on tasks where longer reasoning does not help [13, 14, 16], but it should not be read as a general law. The size of the benchmarks limits what can be detected: a real gain of two or three points is invisible at n = 132. Grok 4.7's distinguishable gain at its highest setting shows that the last step can pay for some models. Gains that are small on single-answer benchmarks may also compound over long agentic tasks [18]. The safe conclusion is that the return to the highest setting is uncertain and must be measured, not assumed.

5.2 Reading vendor charts: the Fable 5.2 case

The chart that prompted this study, a score-against-cost plot for FrontierCode 2.0, was shared on social media without its source. It was traced to Figure 8.4.A of Anthropic's system card for Claude Fable 5.2, which states that Cognition ran the evaluation for every model and that each score is a single result without an uncertainty interval [28]. Two features of the published data deserve attention.

First, there are two cost series for Fable 5.2 (Table 7). Cognition's data file lists costs exactly 1.25 times those in the system card at every setting, while the costs of other models agree between the two sources. The ratio equals the ratio of the list prices of Fable 5.1 and Fable 5.2 [6], which suggests, but does not establish, that the data file was priced at the older rate. This paper uses the system-card costs, which match the current price, and shows the alternative in Figure 5.

Second, launch claims depend on which series is used (Table 8). The claim that Fable 5.2 at high matches the best score of Opus 5.5 at roughly half the cost holds on the system-card costs (0.52 times) but becomes about two-thirds on the data file (0.66 times) [29]. The claimed lead over GPT-6 Astra at similar cost is 2.5 points, well within the interval. Neither claim is wrong on its own terms, but both are more precise than the data allow. Readers should treat single-run benchmark charts from any vendor as indicative and look for the operator's own data.

5.3 Limitations

• Comparability. Each operator defines cost per task differently, applies prices at different dates, and runs each vendor's models in a different harness. FrontierCode runs Anthropic, OpenAI and Google models in different tools, so cross-model differences mix model and harness effects.
• Independence. Cognition, which runs FrontierCode, lists its own models on the same benchmark. Artificial Analysis includes a FrontierCode-derived component in its index [26], so the two are not independent confirmations.
• Noise. Without task-level data, the intervals here are approximations, and paired tests could detect smaller differences than Eq. (4).
• Coverage. Low-cost and open-weight models are thinly represented on the cost-reporting benchmarks, and models are tested at different settings in different sources.
• Transfer. Benchmark cost per task is a lower bound for a business deployment, which adds retries, longer contexts, tool calls and human review. Caching also differs: in Artificial Analysis's run for Fable 5.2 at max, cache reads and writes accounted for about half of the total cost [30].

6. Practical Guidance for Small and Medium-Sized Enterprises

The results suggest that the effort setting should be managed as a cost parameter, chosen by measurement on the firm's own work. Figure 8 sets out a procedure that a small team can run in a few days.

1. Build a test set from real work. Public benchmarks indicate direction, not the size of a gain on a particular workload. A set of 100 to 300 representative tasks with known answers gives an interval of roughly ±5 to ±10 points (Figure 2), enough to detect large effects.
2. Start low and step up. Run two or three candidate models, including at least one low-cost model, at a low setting, then raise the setting one step at a time on the same tasks.
3. Keep a step only if it pays. Accept a step when the gain exceeds the interval and its value exceeds the extra cost per task. Compare cost per correct answer, Eq. (2), not price per token.
4. Budget from measured tokens. Record tokens by type for each task and multiply by a dated price sheet, Eq. (1). A price change then becomes a recalculation rather than a new test.
5. Watch price dates. Note introductory and promotional prices and their end dates, and repeat the comparison when they expire or a new model is released.

Firms that use the AI Models page on this site to shortlist candidates should note that it is a separate, dated snapshot built from overlapping but not identical sources; the procedure above applies whichever shortlist is used.

7. Conclusion

Reasoning effort has made the cost of using a language model a variable that the user controls. On current evidence, moving from the lowest to the highest setting multiplies cost per task by about three to five, and much more for some models. The lower steps often buy large, measurable improvements; the highest steps usually buy changes too small to detect on public benchmarks, at 1.1 to 3.2 times the cost of the step below. Frontier models now cluster so closely on some benchmarks that they cannot be ranked, and list prices, which are often temporary and differ in tokenization, are a weak guide to task cost.

These conclusions are bounded by the data: benchmarks of 120 to 132 tasks cannot detect small gains, operators define cost differently, and vendor charts rest on single runs. For a firm, the practical implication is straightforward. The effort setting should be chosen by testing on representative tasks, kept at the lowest level whose results are good enough, and revisited when prices or models change. Future work should use task-level data with paired tests and a single dated price sheet, and should extend the comparison to the low-cost and open-weight models that matter most to smaller firms.