Tīrtha Bench: the data coders need · tool calling · updated Oct 18, 2026
Every model we have tool-call tested, ranked only where the test was the same
Accuracy by task type, speed, tokens and cost per right answer, with every raw row kept. Each card holds models that ran the same test and were scored the same way. A model's place on one card says nothing
about a model that only appears on another.
Kind
Maker
No model matches these filters.
68models tested
749result rows, Jun 12 to Oct 18
8most models on one shared test
38models with only one test
The bottom line · 8 models on the same 800 tasks and 120 traps
Most accurate
grok-4.7 and opus-5.5 tie at 95.4% of 800 tasks. Next: sonnet-5, 95.1%.
Rule: Highest overall accuracy on the shared 800-task test (same tasks, same scorer, n=800 each).
Best value
Tīrthaour product: 93.6% at $0.00076 per right answer (proposed level pricing). Next: grok-4.7, 95.4% at $0.0015.
Rule: Highest accuracy divided by cost per right answer, among models at or above the median accuracy, same test.
Best at traps
Tīrthaour product: held back on 119 of 120 traps. Next: grok-4.5, 118 of 120.
Rule: Highest trap pass rate on the same 120 traps (min n=40).
What only a shared test showssame tasks · same scorer
Same model, different hosts: gpt-oss-120b
4 hosts, 14.1 points apart on the same 800 tasks. Most of the gap is parallel calls: Cerebras, Groq, DeepInfra scored 0% on the 200 tasks that need several calls at once.
Same model, different hosts: Llama 3.3 70B
2 hosts, 23.9 points apart on the same 800 tasks. Most of the gap is parallel calls: SambaNova scored 0% on the 200 tasks that need several calls at once.
Price per right answer, not sticker price
Dollars per 1,000 correct tool calls: 65x between the highest and the lowest.
Version to version: DeepSeek V4 Flash to V4.1 Flash 1017
Same overall score (92.0% vs 91.6%), far fewer traps fallen for. Oct 18, both on DeepInfra.
Checked by an outside run
An outside tester re-ran our 800 requests on his own machine with Berkeley's official checker (Oct 18): within 3 tasks.
What the numbers mean
Accuracy
Share of tasks where the model made the right tool call with the right arguments, scored by our harness. Only compare models that ran the same test.
Traps
Tasks where no tool fits or a decoy is offered. Passing means holding back instead of calling something wrong. Higher is safer.
Cost per right answer
What the whole run cost at the vendor's list price, divided by the number of right answers. It folds price and accuracy into one number a buyer can use.
Typical speed
Median seconds per call: half of calls were faster.
VOID
A run that broke its own pre-registered rules. We keep it and mark it, and never count it as a score.
Level pricing
How Tīrtha is priced: each request is billed at the level that served it (Base, Plus or Max) plus a margin. The margin shown is proposed and under review.
We publish misses and VOID runs on our own product too.
What the data says
1
The top is a tie.
On the 800-task test, opus-5.5, grok-4.7 and sonnet-5 land within 0.3 points of each other. The bottom of the same table, gpt-5.1, is at 81%. The leaders are interchangeable; the tail is not.
The whole picture: accuracy against priceEach dot is one model's best scored run where cost per right answer was logged; tap a dot or a name to open that model. The low-cost open models sit far left at the same height as the frontier. Tīrtha is shown at its proposed level pricing.
Swipe the chart sideways to see every model.
Meta
Anthropic
OpenAI
xAI
Qwen (Alibaba)
Z.ai (GLM)
Tīrtha (ours)
Best bang for the buckAccuracy divided by cost per right answer, on the same runs as the chart above, scaled so the best is 100. Top five, plus our two products wherever they land (rank of 15 on the left). Tīrtha is priced at its proposed level pricing.
Tīrtha Frontier96.6% at $0.0037 per right answer · our product (measured the same way)
2
The 8 models on one shared test800 tasks · 120 traps · same scorer
Sorted by accuracy; tap a header to sort by it. Every model here ran the identical tasks, so this is the one table where every column compares like with like. Hover a label for the rule that gave it.
overall weighted from its task-type rows; drops to 61.0% on parallel calls
Rank by task typeThe 8 models on the shared 800-task test, ranked within each task type (1 is best). Follow a line to see where a model rises or falls; a tie shares a rank number. Hover a dot for the score.
Single, several, parallelsame 800 tasks · same scorer
Parallel calls means one request that needs several tool calls at once, like checking the weather in three cities. Every model drops there, and it is where they separate most: one fell to 61%. Speed is the typical seconds per call (half of calls were faster), with the slow end in brackets; shorter bar is faster. Tokens out is the average answer length, thinking included. Cost per right answer is the comparable money number: each vendor counts tokens its own way, so token counts are a guide to how much a model talks, not a price. Sorted by parallel.
The outlined row is Tīrtha itself (Base, checked, escalating to Plus), run Oct 17 through api.tirtha.ai on the same 800 tasks. Against bars Grok sealed before the run it was the fastest here and best on traps, and lower cost per right answer than Sonnet 5 ($0.0017 vs $0.0031), but not as accurate as Sonnet 5 overall (93.6% vs 95.1%), mostly on single calls; parallel and several tools are within noise of Sonnet. Its accuracy is its Base model's: the checker sent 9 calls up (all then right) and caught none of the 51 wrong answers. Its cost is shown at level pricing: each request billed at its own level's measured cost plus a 70% margin (proposed, under review).
A decision model does not write the tool call. It reads one and answers a yes/no question with a probability, P(yes). We use Microsoft-Decision-1 that way, as the model layer of our tool-call checker: plain code checks the facts first, then the decision model gets one clear-cut question. These numbers are not on the same scale as the accuracy cards above. Each number shows its ledger row id.
Same 80 checks (muse-round)
Wrong calls let through
Right calls wrongly blocked
Sent up
$ / check
Speed (median, all 3 asks)
1. typesafe/jev-1.13 decision model · TypeSafe via OpenRouter
0 of 60
0 of 20
75%
$0.00011
0.26s hosted
2. gpt-oss-120b chat model as checker · Cerebras
0 of 60
0 of 20
75%
$0.0015
0.55s hosted
3. muse-glimmer-30b chat model as checker · DeepInfra
0 of 60
0 of 20
75%
$0.0027
3.67s hosted
4. x-ai/grok-4.5 chat model as checker · xAI via OpenRouter
0 of 60
0 of 20
75%
$0.0060
4.56s hosted
5. openai/gpt-5.6-luna chat model as checker · OpenAI via OpenRouter
0 of 60
3 of 20
79%
$0.000774
2.48s hosted
6. openai/gpt-oss-safeguard-20b chat model as checker · Groq via OpenRouter
0 of 60
4 of 20
80%
$0.000441
0.99s hosted
7. Qwen/Qwen3-Coder-480B-A35B-Instruct-Turbo chat model as checker · DeepInfra
0 of 60
6 of 20
82%
$0.000461
12.99s hosted
8. deepseek-ai/DeepSeek-V4-Flash chat model as checker · DeepInfra
3 of 60
0 of 20
71%
$0.00011
2.58s hosted
9. microsoft/microsoft-decision-1 decision model · Azure via OpenRouter
3 of 60
3 of 20
75%
$0.000079
0.40s hosted
10. openai/gpt-6-luna-decisions decision model · OpenAI via OpenRouter
3 of 60
4 of 20
76%
$0.000196
0.31s hosted
11. Qwen/Qwen3-235B-A22B-Instruct-2507 descriptive-only chat model as checker · DeepInfra
4 of 58
4 of 20
74%
$0.000273
2.38s hosted
12. openai/gpt-5.4-nano descriptive-only chat model as checker · OpenAI via OpenRouter
4 of 59
10 of 20
82%
$0.000583
1.43s hosted
13. google/gemma-4-26B-A4B-it chat model as checker · DeepInfra (hosted; local-able)
5 of 60
0 of 20
69%
$0.000182
1.36s hosted
14. z-ai/glm-4.7-flash (reasoning off) descriptive-only chat model as checker · Novita via OpenRouter
25 of 55
0 of 16
42%
$0.000651
11.00s hosted
muse-round-2026-10-13, checker arm: 60 wrong calls and 20 right ones, the same checklist question asked 3 times, approved only if all 3 say yes. Ranked by wrong calls let through, then right calls wrongly blocked, then speed, then cost. The bar to beat was Decision-1 and GPT-6 Luna (decisions) at 3 of 60. Speeds marked hosted include the network and the host queue; nothing ranks higher for being local. An easy synthetic set, kept internal. Hover a number for its ledger row (70 rows, r1097 to r4447).
Scaled up to real traffic (Oct 18)
The checkers with a clean sheet on the 80 checks, run on the two real-traffic sets Decision-1 was measured on, with the same frozen labels. Bars were written blind by Grok before any call. Not like-for-like: the candidates got the 80-check checklist prompt; Decision-1 ran inside checker v2.7 (Set A) or on a bare question at a 0.95 bar (Set B). Approving at the default yes/no line, every candidate approved nearly everything and let through far more wrong calls than Decision-1.
microsoft/microsoft-decision-1 · Azure via OpenRouter · Oct 10 to 14
Returns P(yes) for one yes/no question. Inside checker v2.7 it asks about coverage, tool choice and each argument; any doubt sends the call up to a stronger model.
Checker v2.7 on 452 fresh ToolACE calls, Oct 12
Approved265 of 452 (58.6%)r0871, r3774
Wrong among approved1 of 265 (0.38%)r0877, r3775 · 95% upper bound 1.78%
Sent up for a stronger model187 of 452 (41.4%)r0872, r3776
Right calls sent up anyway146 of 452 (32.3%)r0874, r3777 · 117 of the 146 were UNSURE (r0880)
Can it set its own approve bar? Oct 14, pre-registered
Approved at P(yes) of 0.95 or more674 of 900r3881
Wrong among those approved11 of 674 (1.63%)r3882, r3883 · 95% upper bound 2.69% (r3884)
VerdictFAILr3880 · the bar needed 1% or less, so no threshold was certified and every call still goes up
Clear-cut checks, speed and cost, Oct 10
Verdict flips on clear-cut tool checks0 of 120r2036
Median time per call303 msr0370 · p90 398 ms, 775 calls, exploratory
Spend for those 775 calls$0.024r0369
Not deterministic. Six byte-identical requests came back between 0.999089 and 0.99929 (r0811). Across two runs the same question moved by up to 0.37, and about 1% of verdicts crossed the 0.9 approve bar (FINDING-decision1-is-not-deterministic-2026-10-11). A gate on one call near the bar can flip on a retry, so we say "steadier", never "deterministic".
Give it a fact, not a judgement. Asked a judgement ("is this note relevant?"), identical requests differed by a median 0.0069 and up to 0.166, and that gate was voided (r0662). With a fact from code in front of it, a matched pair scored 0.93 vs 0.15, as it should (r0672, n=2). FINDING-decision-models-want-a-fact-not-a-judgement-2026-10-10.
Labels. The Oct 12 run was labelled by GPT-6.1 Sol on low effort, which disagreed with careful labels about 5.7% of the time, and used one corpus (ToolACE). Many right calls go UNSURE, and that is where the cost sits.
GPT-6 Luna (decisions)
openai/gpt-6-luna-decisions · Oct 13
A decisions mode of GPT-6 Luna, tried as a swap for Decision-1 in the same checker seat.
Checker arm, muse-round, Oct 13 (80 checks)
Wrong calls let through3 of 60r1107 · 95% upper bound 12.4%
Right calls approved16 of 20r1108
Sent up76%r1109 · 61 of 80
Cost per check$0.00020r1110 · about 2.5x Decision-1 on the same checks
Median time per check0.31 sr1111
Small and easy. 80 checks on a synthetic set, bar missed, internal only. Its Oct 12 bake-off rows are VOID (the judge prompt saw the expected calls), so they are left out here.
Jev-1.13
typesafe/jev-1.13 · TypeSafe via OpenRouter · Oct 18
A decision model from TypeSafe: returns P(yes) for one yes/no question, like Decision-1. Re-run Oct 18 on the exact 80 checks Decision-1 and GPT-6 Luna (decisions) took, same checklist, same 3 asks, same scoring.
Checker arm, muse-round, re-run Oct 18 (80 checks)
Wrong calls let through0 of 60r4393 · Decision-1 let through 3 (r1102); 95% upper bound 4.9%
Right calls approved20 of 20r4394 · Decision-1 approved 17
Sent up75%r4395 · exactly the 60 wrong calls
Cost per check$0.00011r4396 · 3 asks; Decision-1 $0.000079
Median time per check0.27 sr4397 · hosted
Oct 12 bake-off (VOID run, kept for the record)
Median time per check322 msr0961
Small and easy. 80 synthetic checks, internal only. The Oct 12 bake-off was voided for reasons that were ours, not Jev's (another arm's prompt saw the expected calls, and one arm had missing answers), so this is its first clean comparison.
Laya
Jev-style decision model · run locally · Sep 26
Local, so its speed is a local-machine number; that alone does not help production. Its one test was a different job: deciding whether a hook's note was useful or noise, not checking tool calls.
Hook gate, zero-shot, Sep 26
Useful vs noise (AUROC, 0.5 is chance)0.471r1654 · n=29, CI about plus or minus 0.2
Median time per decision168 msr1654
No better than chance on this job, on a small sample. Fast because it runs on our own machine.
Not on the checker test yet (Oct 18). The laya package is not installed in any environment on this Mac (only its weights are cached), so it could not be run on the 80 checks without installing something (r4449).
Source: Tirtha results ledger (ALL-RESULTS-2026-10-14.json). VOID runs left out. Scores are pass rates as stored; tests differ
in difficulty, so never compare across cards. Trap scores before 2026-09-24 read only the first call, so a served decoy could still pass.
Cost is per right answer where the run logged it, else per task; time is median seconds per call.
VOID runs are left out of the ranked cards; the Decision models section shows one VOID run's speed and cost, marked. Row ids (r0000) point into the ledger. Logos: simple-icons (CC0) where one exists, else a monogram.
