TīrthaModels

← Leaderboard · every tool-calling result, by model

All 66 models and 2 Tīrtha systems, every result in one place

Pick a model to open its page: its headline numbers and where each one ranks, a one-line read of what they mean, how it does by task type and against the field, then every result we hold, grouped by test, VOID runs included and marked. A model name on the leaderboard opens its page here, already open.

At a glance

Accuracy against priceEach model's best overall score where cost per right answer was logged, at its own list price. Tap a dot or a name to open that model below. Tīrtha and Tīrtha Frontier are shown at level pricing: every request billed at its own level's cost plus a 70% margin (proposed, under review).

Swipe the chart sideways to see every model.

80%85%90%95%$0.00005$0.0001$0.0002$0.0005$0.001$0.002$0.005$0.01cost per right answer, log scale (lower cost to the left)↖ Upper-left is bestmore right answers for less moneyyellow band: the efficient frontier (nothing is both lower cost and more accurate)glm-5.2: 97.5% at $0.0033 per right answer. Open its page.Tīrtha Frontier: 96.6% at $0.0037 per right answer. Open its page.qwen3-235b: 95.7% at $0.00007 per right answer. Open its page.grok-4.7: 95.7% at $0.0015 per right answer. Open its page.sonnet-4.5: 95.5% at $0.0049 per right answer. Open its page.gpt-5.1-codex-mini: 95.5% at $0.0026 per right answer. Open its page.sonnet-5: 95.0% at $0.0031 per right answer. Open its page.opus-5.5: 94.0% at $0.0056 per right answer. Open its page.grok-4.5: 94.0% at $0.0015 per right answer. Open its page.Tīrtha: 93.6% at $0.00076 per right answer. Open its page.glm-4.7: 93.5% at $0.0033 per right answer. Open its page.muse-spark-open: 93.2% at $0.0020 per right answer. Open its page.qwen3-coder-480b: 91.3% at $0.00024 per right answer. Open its page.gpt-5.2-codex: 89.9% at $0.0013 per right answer. Open its page.gpt-5.1-codex: 81.8% at $0.0010 per right answer. Open its page.Tīrtha FrontierTīrthaglm-5.2qwen3-235bgrok-4.7sonnet-4.5gpt-5.1-codex-minisonnet-5opus-5.5grok-4.5glm-4.7muse-spark-openqwen3-coder-480bgpt-5.2-codexgpt-5.1-codex
  • Meta
  • Anthropic
  • OpenAI
  • xAI
  • Qwen (Alibaba)
  • Z.ai (GLM)
  • Tīrtha (ours)
Best bang for the buckAccuracy divided by cost per right answer, on the same runs as the chart above, scaled so the best is 100. Top five, plus our two products wherever they land (rank of 15 on the left). Tīrtha is priced at its proposed level pricing.
  1. 1
    qwen3-235b95.7% at $0.00007 per right answer
    100
  2. 2
    qwen3-coder-480b91.3% at $0.00024 per right answer
    27
  3. 3
    Tīrtha93.6% at $0.00076 per right answer · our product (measured the same way)
    9
  4. 4
    gpt-5.1-codex81.8% at $0.0010 per right answer
    6
  5. 5
    gpt-5.2-codex89.9% at $0.0013 per right answer
    5
  6. 13
    Tīrtha Frontier96.6% at $0.0037 per right answer · our product (measured the same way)
    2
Rank by task typeThe 8 models on the shared 800-task test, ranked within each task type (1 is best). Follow a line to see where a model rises or falls; a tie shares a rank number. Hover a dot for the score.
Single callSeveral toolsParallel callsTrapssonnet-5, single call: 97.5% (1 of 8)1sonnet-5, several tools: 93.5% (4 of 8)4sonnet-5, parallel calls: 92.0% (4 of 8)4sonnet-5, traps: 94.2% (4 of 8)4sonnet-5sonnet-5opus-5.5, single call: 97.0% (3 of 8)3opus-5.5, several tools: 94.0% (2 of 8)2opus-5.5, parallel calls: 93.5% (1 of 8)1opus-5.5, traps: 85.0% (8 of 8)8opus-5.5opus-5.5gpt-5.1-codex, single call: 86.8% (8 of 8)8gpt-5.1-codex, several tools: 88.0% (8 of 8)8gpt-5.1-codex, parallel calls: 61.0% (8 of 8)8gpt-5.1-codex, traps: 90.0% (5 of 8)5gpt-5.1-codexgpt-5.1-codexgpt-5.2-codex, single call: 92.5% (7 of 8)7gpt-5.2-codex, several tools: 89.0% (7 of 8)7gpt-5.2-codex, parallel calls: 87.5% (7 of 8)7gpt-5.2-codex, traps: 86.7% (7 of 8)7gpt-5.2-codexgpt-5.2-codexgrok-4.5, single call: 94.0% (5 of 8)5grok-4.5, several tools: 93.5% (4 of 8)4grok-4.5, parallel calls: 92.0% (4 of 8)4grok-4.5, traps: 98.3% (2 of 8)2grok-4.5grok-4.5grok-4.7, single call: 97.5% (1 of 8)1grok-4.7, several tools: 94.0% (2 of 8)2grok-4.7, parallel calls: 92.5% (3 of 8)3grok-4.7, traps: 97.5% (3 of 8)3grok-4.7grok-4.7muse-spark-open, single call: 95.2% (4 of 8)4muse-spark-open, several tools: 93.0% (6 of 8)6muse-spark-open, parallel calls: 89.5% (6 of 8)6muse-spark-open, traps: 90.0% (5 of 8)5muse-spark-openmuse-spark-openTīrtha, single call: 93.5% (6 of 8)6Tīrtha, several tools: 94.5% (1 of 8)1Tīrtha, parallel calls: 93.0% (2 of 8)2Tīrtha, traps: 99.2% (1 of 8)1TīrthaTīrtha
How we price what we measureTīrtha is priced per level: every request is billed at the level that served it, so no request is sold below its cost. Real traffic so far (10,440 requests, all ours: no outside customers yet). Price shown at a 70% margin, proposed and under review.
LevelRequestsShareOur cost / requestPrice / request
Base10,30498.7%$0.000494$0.001646
Plus1301.2%$0.000134$0.000448
Max60.1%$0.005964$0.019879
Blended10,440100%$0.000493$0.001642
Tīrtha Frontier (agreement-checked, one live night)800$0.001077$0.003590

Source: scripts/pricing-page-data.py, the same numbers as the pricing page.

About the ledger itself

Most-tested modelsDistinct tests per model, top 15, in maker colours.
gpt-oss-120bgpt-oss-120b: 1919sonnet-5sonnet-5: 1111qwen3-coder-480bqwen3-coder-480b: 1010grok-4.5grok-4.5: 99qwen3-235bqwen3-235b: 99sonnet-4.5sonnet-4.5: 99llama-3.3-70bllama-3.3-70b: 88deepseek-v4-flashdeepseek-v4-flash: 77grok-4.7grok-4.7: 77muse-glimmer-30bmuse-glimmer-30b: 66glm-5.2glm-5.2: 55gpt-5.2gpt-5.2: 55opus-5.5opus-5.5: 55qwen3-coderqwen3-coder: 55decision-1decision-1: 44
MakersModels we have tool-call results for, by maker.
Qwen (Alibaba): 12OpenAI: 10Anthropic: 8DeepSeek: 7Z.ai (GLM): 5Meta: 5xAI: 3Google: 3NVIDIA: 3Moonshot (Kimi): 2Kwaipilot (Kuaishou): 2Maker not recorded: 2Microsoft: 1Mistral: 1MiniMax: 1Pair of makers: 166models
  • Qwen (Alibaba)12
  • OpenAI10
  • Anthropic8
  • DeepSeek7
  • Z.ai (GLM)5
  • Meta5
  • xAI3
  • Google3
  • NVIDIA3
  • Moonshot (Kimi)2
  • Kwaipilot (Kuaishou)2
  • Maker not recorded2
  • Microsoft1
  • Mistral1
  • MiniMax1
  • Pair of makers1
Testing over timeResult rows logged per week.
week of Jul 13: 36 resultsJul 13week of Jul 20: 21 resultsweek of Jul 27: 131 resultsweek of Aug 3: 68 resultsAug 3week of Aug 10: 29 resultsweek of Aug 17: 0 resultsweek of Aug 24: 0 resultsAug 24week of Aug 31: 0 resultsweek of Sep 7: 0 resultsweek of Sep 14: 0 resultsSep 14week of Sep 21: 148 resultsweek of Sep 28: 0 resultsweek of Oct 5: 268 results268Oct 5week of Oct 12: 241 results
How results were judgedValid means scored against pre-registered bars; VOID means the run broke its own rules.
  • valid392 42%
  • descriptive-only332 35%
  • other114 12%
  • VOID91 10%
  • inconclusive13 1%
Sort

OpenAI · Model

gpt-oss-120b

19tests145results25valid#2of 3 on Value rung per task shape: code vs tools

accuracy64.0%5 of 5n=800 · Frontier refresh BFCL v4 pinned arms
traps passed98.0%5 of 5n=50 · Trap set per shape
typical speed0.36 s1 of 5median per call · Frontier refresh BFCL v4 latency

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on single calls (90.2%), weakest on parallel calls (0.0%); behind Sonnet 5 on all 4 task types; one of the fastest (1 of 5).

Accuracy by task typegpt-oss-turbo-vs-plain-deepinfra-2026-10-18 (auto-logged run). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
90.2%
several toolsn=200
86.0%
parallel callsn=200
0.0%
trapsn=120
81.7%
0%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyMistake atlas: first-call tool accuracy across families
4
worse88.3% · 4 of 5better
Speed, typical seconds per callFrontier refresh BFCL v4 latency (from Mac)
1
worse0.36 s · 1 of 5better
All 145 results28 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
gpt-oss-turbo-vs-plain-deepinfra-2026-10-18 (auto-logged run) 9 results · 2026-10-18
2026-10-18trap success (all calls scored)81.7%deepinfra-plainauto
2026-10-18tokens out per call (mean)116.6 tokensdeepinfra-plainauto
2026-10-18latency p956.03 sdeepinfra-plainauto
2026-10-18latency p502.25 sdeepinfra-plainauto
2026-10-18cost per taskNonedeepinfra-plainauto
2026-10-18accuracy, single call90.2%deepinfra-plainauto
2026-10-18accuracy, several tools to choose from86.0%deepinfra-plainauto
2026-10-18accuracy, parallel calls0.0%deepinfra-plainauto
2026-10-18accuracy66.6%deepinfra-plainauto
gpt-oss-groq-or-together-2026-10-18 (auto-logged run) 18 results · 2026-10-18
2026-10-18trap success (all calls scored)69.2%togetherauto
2026-10-18trap success (all calls scored)83.3%Groqauto
2026-10-18tokens out per call (mean)378.4 tokenstogetherauto
2026-10-18tokens out per call (mean)123.7 tokensGroqauto
2026-10-18latency p959.97 stogetherauto
2026-10-18latency p951.07 sGroqauto
2026-10-18latency p502.51 stogetherauto
2026-10-18latency p500.5 sGroqauto
2026-10-18cost per taskNonetogetherauto
2026-10-18cost per task$0.00011Groqauto
2026-10-18accuracy, single call82.2%togetherauto
2026-10-18accuracy, single call89.5%Groqauto
2026-10-18accuracy, several tools to choose from79.5%togetherauto
2026-10-18accuracy, several tools to choose from86.5%Groqauto
2026-10-18accuracy, parallel calls68.0%togetherauto
2026-10-18accuracy, parallel calls0.0%Groqauto
2026-10-18accuracy78.0%togetherauto
2026-10-18accuracy66.4%Groqauto
gpt-oss-groq-cerebras-direct-2026-10-18 (auto-logged run) 18 results · 2026-10-18
2026-10-18trap success (all calls scored)83.3%cerebrasauto
2026-10-18trap success (all calls scored)3.3%groqINVALID:
2026-10-18tokens out per call (mean)112.5 tokenscerebrasauto
2026-10-18tokens out per call (mean)71.3 tokensgroqINVALID:
2026-10-18latency p950.51 scerebrasauto
2026-10-18latency p955.47 sgroqINVALID:
2026-10-18latency p500.28 scerebrasauto
2026-10-18latency p502.22 sgroqINVALID:
2026-10-18cost per taskNonecerebrasauto
2026-10-18cost per taskNonegroqINVALID:
2026-10-18accuracy, single call85.2%cerebrasauto
2026-10-18accuracy, single call87.8%groqINVALID:
2026-10-18accuracy, several tools to choose from85.0%cerebrasauto
2026-10-18accuracy, several tools to choose from81.0%groqINVALID:
2026-10-18accuracy, parallel calls0.0%cerebrasauto
2026-10-18accuracy, parallel calls0.0%groqINVALID:
2026-10-18accuracy63.9%cerebrasauto
2026-10-18accuracy64.1%groqINVALID:
Soft two-low-cost agreement re-score + strict tool calls on gpt-oss-120b 12 results · 2026-10-15
2026-10-15strict wins / losses on the same item-3/100 (-3.0%)cerebras direct · paireddescriptive-only
2026-10-15strict honoured (schema-violating probe args blocked)0.0%openrouter->cerebras (require_parameters, no fallbacks) · B strict via OpenRouterpreflight-fail
2026-10-15strict honoured (schema-violating probe args blocked)100.0%cerebras direct · B strict directpreflight-pass
2026-10-15malformed args (bad JSON) / unknown tool0/100 (0.0%)cerebras direct · bothdescriptive-only
2026-10-15latency p50 ms319 mscerebras direct · strict ON vs OFFdescriptive-only
2026-10-15exact correct vs BFCL truth73.0%cerebras direct · strict ONdescriptive-only
2026-10-15exact correct vs BFCL truth76.0%cerebras direct · strict OFFdescriptive-only
2026-10-15enum-trap probe (lion vs enum zebra/horse)Noneopenrouter->cerebras · B strict via OpenRouternon-discriminating
2026-10-15canonical correct vs BFCL truth76.0%cerebras direct · strict ONdescriptive-only
2026-10-15canonical correct vs BFCL truth79.0%cerebras direct · strict OFFdescriptive-only
2026-10-15HTTP 429 (retried ok)3/100 (3.0%)cerebras direct · bothdescriptive-only
2026-10-15HTTP 400 schema rejected3/100 (3.0%)cerebras direct · strict ONdescriptive-only
Ladder M2: which second low-cost model / pair / anchored trio should check Qwen3-Coder-480B on tool calls (GROK-prereg-ladder-m2) 3 results · 2026-10-14
2026-10-14rho_err = P(candidate wrong | Qwen wrong), K=19 confirmed15.8%DeepInfra direct · gptoss120bvalid
2026-10-14latency p500.91 sDeepInfra direct · gptoss120bvalid
2026-10-14$/item$0.00010DeepInfra direct · gptoss120bvalid
Gateway catch across families (Together) 12 results · 2026-10-09
2026-10-09trigger fired overall39/1500 (2.6%)Together · catch triggerVOID
2026-10-09run cost (usage)NoneTogether · all armsVOID
2026-10-09retry rows carrying a lookup_contact-only harmony-text call (repair)30/31 (96.8%)Together · SPECIFICdescriptive-only
2026-10-09T trap success (success/all)262/300 (87.3%)TogetherVOID
2026-10-09T trap success (success/all)262/300 (87.3%)Together · SPECIFICVOID
2026-10-09T trap success (success/all)262/300 (87.3%)Together · GENERICVOID
2026-10-09P parallel success (success/all)401/600 (66.8%)TogetherVOID
2026-10-09P parallel success (success/all)401/600 (66.8%)Together · SPECIFICVOID
2026-10-09P parallel success (success/all)401/600 (66.8%)Together · GENERICVOID
2026-10-09M BFCL-multiple success (success/all)458/600 (76.3%)TogetherVOID
2026-10-09M BFCL-multiple success (success/all)458/600 (76.3%)Together · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)458/600 (76.3%)Together · GENERICVOID
Gateway catch across families: mistake atlas (Together) 4 results · 2026-10-09
2026-10-09success rate (answered denominator)88.3%Together · ATLAS NONE, T trapsdescriptive-only
2026-10-09success rate (answered denominator)79.4%Together · ATLAS NONE, M bfcl-multipledescriptive-only
2026-10-09success rate (answered denominator)78.4%Together · ATLAS NONE, P bfcl-paralleldescriptive-only
2026-10-09api_error VOIDs100 countTogether · ATLAS NONEdescriptive-only
Gateway catch, Qwen family + gpt-oss (DeepInfra) 11 results · 2026-10-09
2026-10-09trigger fired overall0/1500 (0.0%)DeepInfra · catch triggerVOID
2026-10-09run cost (usage)$0.0451DeepInfra · all armsVOID
2026-10-09T trap success (success/all)298/300 (99.3%)DeepInfraVOID
2026-10-09T trap success (success/all)298/300 (99.3%)DeepInfra · SPECIFICVOID
2026-10-09T trap success (success/all)298/300 (99.3%)DeepInfra · GENERICVOID
2026-10-09P parallel success (success/all)0/600 (0.0%)DeepInfraVOID
2026-10-09P parallel success (success/all)0/600 (0.0%)DeepInfra · SPECIFICVOID
2026-10-09P parallel success (success/all)0/600 (0.0%)DeepInfra · GENERICVOID
2026-10-09M BFCL-multiple success (success/all)514/600 (85.7%)DeepInfraVOID
2026-10-09M BFCL-multiple success (success/all)514/600 (85.7%)DeepInfra · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)514/600 (85.7%)DeepInfra · GENERICVOID
Mistake atlas: first-call tool accuracy across families 4 results · 2026-10-09
2026-10-09success rate (answered denominator)88.3%Together · ATLAS_ONLY T (traps)descriptive-only
2026-10-09success rate (answered denominator)79.4%Together · ATLAS_ONLY M (bfcl-multiple)descriptive-only
2026-10-09success rate (answered denominator)78.4%Together · ATLAS_ONLY P (bfcl-parallel)descriptive-only
2026-10-09api_error VOIDs100 countTogether · ATLAS_ONLY all setsdescriptive-only
Per-model mistake atlas, first calls (Together) 8 results · 2026-10-08
2026-10-08run costNoneTogether · NONE (first call)descriptive-only
2026-10-08mistake shape: wrong_tool4/1500 (0.3%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: missed_call68/1500 (4.5%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: decoy_extra_call97/1500 (6.5%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: arg_wrong_VALUE98/1500 (6.5%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: answered_in_text58/1500 (3.9%)Together · NONE (first call)descriptive-only
2026-10-08latency p506.78 sTogether · NONE (first call)descriptive-only
2026-10-08first-call success (T+M+P sets)1133/1500 (75.5%)Together · NONE (first call)descriptive-only
Cascade vs arms vs oracle on HumanEval+ / BFCL 1 result · 2026-08-16
2026-08-16HumanEval+ plus pass91.5% · single armvalid
Frontier refresh BFCL v4 pinned arms 1 result · 2026-08-06
2026-08-06accuracy (our harness)512/800 (64.0%)Cerebras · provider-pinnedvalid
Frontier refresh BFCL v4 per category 3 results · 2026-08-06
2026-08-06accuracy (our harness)343/400 (85.8%)Cerebras · simplevalid
2026-08-06accuracy (our harness)169/200 (84.5%)Cerebras · multiplevalid
2026-08-06accuracy (our harness)0/200 (0.0%)Cerebras · parallelvalid
Frontier refresh BFCL v4 latency (from Mac) 1 result · 2026-08-06
2026-08-06p50 latency0.36 sCerebras · provider-pinneddescriptive-only
Trap set per shape (frontier refresh arms) 4 results · 2026-08-06
2026-08-06trap success49/50 (98.0%) · S1 forecastvalid
2026-08-06trap success49/50 (98.0%) · S2 destinationvalid
2026-08-06trap success0/50 (0.0%) · S3 poundsvalid
2026-08-06trap success50/50 (100.0%) · S4 no-toolvalid
Live ladder rungs: parallel tool-call capability 4 results · 2026-08-06
2026-08-06parallel tool calls emitted0/9 (0.0%)Cerebras · low-cost-1 codedescriptive-only
2026-08-06parallel tool calls emitted0/6 (0.0%)DeepInfra · low-cost-3 bothdescriptive-only
2026-08-06parallel tool calls emitted0/15 (0.0%)Cerebras (parallel_tool_calls:true) · hypothesis testdescriptive-only
2026-08-06parallel tool calls emitted12/15 (80.0%)Together · hypothesis testdescriptive-only
Parallel tool calls by provider (gpt-oss-120b) 5 results · 2026-07-29 to 2026-08-06
2026-08-06multi-call responses12/15 (80.0%)Together (direct) · re-derivation BFCL-paralleldescriptive-only
2026-08-06multi-call responses13/15 (86.7%)OpenRouter pinned Together · re-derivation BFCL-paralleldescriptive-only
2026-07-29exact call-count match0/8 (0.0%)Cerebras · provider pinneddescriptive-only
2026-07-29exact call-count match0/8 (0.0%)DeepInfra/SambaNova (OpenRouter unpinned) · unpinneddescriptive-only
2026-07-29exact call-count match5/8 (62.5%)Together · provider pinneddescriptive-only
Provider lottery: gpt-oss-120b pinned vs unpinned 2 results · 2026-08-02
2026-08-02tool-call accuracy65.2% · pinned (prod order)valid
2026-08-02tool-call accuracy65.8%OpenRouter (unpinned) · unpinnedvalid
Provider lottery: pinned vs unpinned same model 2 results · 2026-08-02
2026-08-02trap success75.0%OpenRouter pinned Cerebras,Groq,Together · pinned prod order trapsvalid
2026-08-02BFCL accuracy65.2%OpenRouter pinned Cerebras,Groq,Together · pinned prod ordervalid
Frontier refresh benchmark, BFCL-parallel 1 result · 2026-07-30
2026-07-30BFCL-parallel accuracy0.0%valid
Parallel tool-call capability per provider x model 3 results · 2026-07-30
2026-07-30parallel tool-call probe passed5/8 (62.5%)Together (via OpenRouter) · provider pinned: Togetherdescriptive-only
2026-07-30parallel tool-call probe passed0/8 (0.0%)Groq (via OpenRouter) · provider pinned: Groqdescriptive-only
2026-07-30parallel tool-call probe passed0/8 (0.0%)DeepInfra (via OpenRouter) · provider pinned: DeepInfradescriptive-only
Frozen ladder BFCL v4 N=800: cost/latency 3 results · 2026-07-29
2026-07-29p50 latency0.39 sapi.tirtha.ai · overallvalid
2026-07-29escalation rate1.8%api.tirtha.ai · overallvalid
2026-07-29cost per successful task$0.00066api.tirtha.ai · overallVOID
Frozen ladder trap set 2 results · 2026-07-29
2026-07-29wrong tool served25.0%api.tirtha.ai · trapsvalid
2026-07-29trap success30/40 (75.0%)api.tirtha.ai · trapsvalid
Drafter candidates: tool accuracy vs generation rate 4 results · 2026-07-29
2026-07-29trap accuracy30/40 (75.0%)Cerebras · solodescriptive-only
2026-07-29generation rate (900 tok)1410.8 tok/sCerebras · solodescriptive-only
2026-07-29BFCL parallel accuracy2.0%Cerebras · solodescriptive-only
2026-07-29BFCL overall accuracy65.8%Cerebras · solodescriptive-only
Provider sweep from box: parallel calls + latency 1 result · 2026-07-29
2026-07-29parallel tool calls emitted0/6 (0.0%)Cerebras · provider sweepdescriptive-only
Solo BFCL: gpt-oss-120b vs qwen3-235b 2 results · 2026-07-29
2026-07-29latency p503.03 s · solo, identical harnessdescriptive-only
2026-07-29accuracy65.8% · solo, identical harnessdescriptive-only
Tools repair loop vs cold re-ask on arg errors 4 results · 2026-07-27
2026-07-27rescued27.3% · repair (rejection forwarded)valid
2026-07-27rescued22.7% · cold re-ask (control)valid
2026-07-27rescued22.7% · follow-up bare rejectionvalid
2026-07-27rescued22.7% · follow-up schema-vocabulary rejectionvalid
Value rung per task shape: code vs tools 3 results · 2026-07-26
2026-07-26success84.5% · tools (synthetic, paired)descriptive-only
2026-07-26pass@179.4% · code (HumanEval+ + MBPP+)descriptive-only
2026-07-26accuracy26/42 (61.9%) · escalated subset (witness disagrees)descriptive-only
  1. Sample sizes run from n=1 to n=1500; 41 of 145 results carry a confidence interval or p-value.
  2. Results dated 2026-07-26 to 2026-10-18; hosts: Cerebras, Cerebras (parallel_tool_calls:true), DeepInfra, DeepInfra (via OpenRouter), DeepInfra direct, DeepInfra/SambaNova (OpenRouter unpinned) and others.
  3. 23 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  5. Also logged as: gpt-oss-120b, gpt-oss-120b (tools tier), gpt-oss-120b-Turbo, gpt-oss-120b-Turbo (reasoning low), openai/gpt-oss-120b, openrouter:openai/gpt-oss-120b. Hosts: Cerebras, Cerebras (parallel_tool_calls:true), DeepInfra, DeepInfra (via OpenRouter), DeepInfra direct, DeepInfra/SambaNova (OpenRouter unpinned), Groq, Groq (via OpenRouter), OpenRouter (unpinned), OpenRouter pinned Cerebras,Groq,Together, OpenRouter pinned Together, Together, Together (direct), Together (via OpenRouter), api.tirtha.ai, cerebras, cerebras direct, deepinfra-plain, groq, openrouter->cerebras, openrouter->cerebras (require_parameters, no fallbacks), together.

Anthropic · Model

sonnet-5

11tests37results22valid#1of 8 on Frontier sweep BFCL (unpinned, OpenRouter)

accuracy95.1%3 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed94.2%4 of 8n=120 · 800 BFCL tasks by task type
typical speed2.84 s7 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.00316 of 7list price · 800 BFCL tasks by task type

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on single calls (97.5%), weakest on parallel calls (92.0%); among the slowest (7 of 8).

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
97.5%
several toolsn=200
93.5%
parallel callsn=200
92.0%
trapsn=120
94.2%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
3
worse95.1% · 3 of 8better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
7
worse2.84 s · 7 of 8better
Cost per right answer800 BFCL tasks by task type (same tasks, same scorer)
6
worse$0.0031 · 6 of 7better
All 37 results13 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Sonnet 5 effort settings on the 800 BFCL + 120 traps (vs tonight's default high) 6 results · 2026-10-17
2026-10-17trap success (all calls scored)97.5%OpenRouter · reasoning.effort=lowvalid
2026-10-17trap success (all calls scored)95.0%OpenRouter · reasoning disabledvalid
2026-10-17cost per task (vendor)$0.0027OpenRouter · reasoning.effort=lowvalid
2026-10-17cost per task (vendor)$0.0028OpenRouter · reasoning disabledvalid
2026-10-17accuracy95.2%OpenRouter · reasoning.effort=lowvalid
2026-10-17accuracy95.2%OpenRouter · reasoning disabledvalid
Offline cost per correct tool call 4 results · 2026-10-10
2026-10-10mean cost per task$0.0029OpenRouter · directdescriptive-only
2026-10-10cost per correct$0.0031OpenRouter · directdescriptive-only
2026-10-10cost per correct$0.0031OpenRouter · QC matched mix (840: 800 #r0 + 40 traps)descriptive-only
2026-10-10correct (outcome==success)874/920 (95.0%)OpenRouter · directdescriptive-only
Contradiction test pilot (note vs inverted note) 3 results · 2026-10-08
2026-10-08runs whose final outcome followed the inverted note1/20 (5.0%)Anthropic (Claude Code subscription) · INVERTEDVOID
2026-10-08runs CORRECT (60 runs = 10 traps x 3 arms x 2)59/60 (98.3%)Anthropic (Claude Code subscription) · all armsVOID
2026-10-08BASELINE_CORRECT traps10/10 (100.0%)Anthropic (Claude Code subscription) · EMPTY (no note)VOID
Contradiction pilot (notes vs model behaviour) 2 results · 2026-10-08
2026-10-08traps correct10/10 (100.0%)Anthropic · no notedescriptive-only
2026-10-08flipped notes overridden19/20 (95.0%)Anthropic · flipped (contradicting) notesdescriptive-only
Frontier tool-calling 6 arms N=800 1 result · 2026-09-25
2026-09-25accuracy95.1%pinned · standard setvalid
800 BFCL tasks by task type (same tasks, same scorer) 9 results · 2026-09-25
2026-09-25trap success (all calls scored)113/120 (94.2%)valid
2026-09-25tokens out per call (mean)163.2 tokensvalid
2026-09-25tokens in per call (mean)676.1 tokensvalid
2026-09-25thinking tokens per call (mean, where reported)14.6 tokensvalid
2026-09-25latency p50 (seconds per call)2.84 sClaude Platform on AWSvalid
2026-09-25cost per right answer$0.0031valid
2026-09-25accuracy, single call97.5%valid
2026-09-25accuracy, several tools to choose from93.5%valid
2026-09-25accuracy, parallel calls92.0%valid
Frontier tool-calling BFCL full n=800 2 results · 2026-09-25
2026-09-25emitted ONE call when several needed3/200 (1.5%) · bfcl-parallelvalid
2026-09-25accuracy95.1% · standard set (run frontier-full-n800)valid
Frontier pilot trap scores, artifact shapes removed 2 results · 2026-09-24
2026-09-24trap score100/100 (100.0%) · trap set run 1, S3+S12 removedvalid
2026-09-24trap score100/100 (100.0%) · trap set run 2, S3+S12 removedvalid
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy98.0%OpenRouter (unpinned) · frontier sweepdescriptive-only
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)95.1%OpenRouter · single run (Tirtha 3-seed)valid
BFCL v4 Sonnet-5 yardstick 1 result · 2026-07-15
2026-07-15accuracy (our harness)95.0%OpenRouter · frontier yardstickvalid
Tool-eval cost run A (BFCL-800) 2 results · 2026-07-15
2026-07-15arg_wrong36/800 (4.5%) · costrun-Adescriptive-only
2026-07-15accuracy95.0% · costrun-Adescriptive-only
BFCL frontier bake-off (Grok 4.5 vs Sonnet 5) 3 results · 2026-07-14
2026-07-14latency p503.58 sOpenRouter · BFCL v4 N=800descriptive-only
2026-07-14cost per successful task$0.0032OpenRouter · BFCL v4 N=800descriptive-only
2026-07-14accuracy94.8%OpenRouter · BFCL v4 N=800descriptive-only
  1. Sample sizes run from n=10 to n=920; 5 of 37 results carry a confidence interval or p-value.
  2. Results dated 2026-07-14 to 2026-10-17; hosts: Anthropic, Anthropic (Claude Code subscription), Claude Platform on AWS, OpenRouter, OpenRouter (unpinned), pinned.
  3. 3 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  5. Also logged as: Sonnet-5, anthropic/claude-sonnet-5, claude-sonnet-5, openrouter:anthropic/claude-sonnet-5, openrouter:anthropic/claude-sonnet-5 (reasoning disabled), openrouter:anthropic/claude-sonnet-5 (reasoning.effort=low), sonnet-5. Hosts: Anthropic, Anthropic (Claude Code subscription), Claude Platform on AWS, OpenRouter, OpenRouter (unpinned), pinned.

Tīrtha (ours) · Tīrtha system

Tīrtha

11tests34results33valid#1of 8 on 800 BFCL tasks by task type (same tasks, sa…

accuracy93.6%4 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed99.2%1 of 8n=120 · 800 BFCL tasks by task type
typical speed0.92 s1 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.0017no shared fieldlist price · Frontier tool-calling 6 arms N=800

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

A Tīrtha system (a gateway policy over several models): best on traps (99.2%), weakest on parallel calls (93.0%); ahead of Sonnet 5 on 3 of 4 task types; one of the fastest (1 of 8).

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
93.5%
several toolsn=200
94.5%
parallel callsn=200
93.0%
trapsn=120
99.2%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
4
worse93.6% · 4 of 8better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
1
worse0.92 s · 1 of 8better
All 34 results16 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier tool-calling 6 arms N=800 6 results · 2026-10-17
2026-10-17wrong answers caught by the checker0/51 (0.0%)api.tirtha.aivalid
2026-10-17latency p500.92 sapi.tirtha.aivalid
2026-10-17escalation rate1.1%api.tirtha.aivalid
2026-10-17cost per correct (our measured cost)$0.00023api.tirtha.aivalid
2026-10-17cost per correct (customer list $2/$8 per 1M)$0.0017api.tirtha.aivalid
2026-10-17accuracy93.6%api.tirtha.aivalid
Real Work track week 0: BFCL v3 multi_turn_base, all 200, official Berkeley checker 1 result · 2026-10-17
2026-10-17accuracy, multi-turn (official checker)52.8%api.tirtha.aivalid
800 BFCL tasks by task type (same tasks, same scorer) 8 results · 2026-10-17
2026-10-17trap success (all calls scored)119/120 (99.2%)api.tirtha.aivalid
2026-10-17tokens out per call (mean)67.5 tokensapi.tirtha.aivalid
2026-10-17tokens in per call (mean)529.4 tokensapi.tirtha.aivalid
2026-10-17latency p50 (seconds per call)0.92 sapi.tirtha.aivalid
2026-10-17cost per right answer$0.0017api.tirtha.aivalid
2026-10-17accuracy, single call93.5%api.tirtha.aivalid
2026-10-17accuracy, several tools to choose from94.5%api.tirtha.aivalid
2026-10-17accuracy, parallel calls93.0%api.tirtha.aivalid
Tool calls through api.tirtha.ai correct 1 result · 2026-10-11
2026-10-11correct162/200 (81.0%)api.tirtha.ai · prodvalid
Tool calls through api.tirtha.ai, per category 5 results · 2026-10-11
2026-10-11correct24/25 (96.0%)api.tirtha.ai · bfcl-simplevalid
2026-10-11correct24/25 (96.0%)api.tirtha.ai · bfcl-multiplevalid
2026-10-11correct24/25 (96.0%)api.tirtha.ai · bfcl-parallelvalid
2026-10-11correct18/25 (72.0%)api.tirtha.ai · bfcl-parallel_multiplevalid
2026-10-11correct72/100 (72.0%)api.tirtha.ai · toolace-singlevalid
Tool calls through api.tirtha.ai 1 result · 2026-10-11
2026-10-11false verify rate100.0%api.tirtha.ai · allvalid
Trap set widened to 12 shapes, live prod 1 result · 2026-10-06
2026-10-06trap success119/120 (99.2%)api.tirtha.ai · trapsvalid
Synthetic standard tasks, live prod 1 result · 2026-10-06
2026-10-06accuracy373/400 (93.2%)api.tirtha.ai · standardvalid
Trap set n=200, live prod 1 result · 2026-07-30
2026-07-30trap success200/200 (100.0%)api.tirtha.ai · prodvalid
BFCL v4 N=400 on DeepInfra 480B tier 1 result · 2026-07-18
2026-07-18accuracy (our harness)94.0%DeepInfra · prodvalid
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)94.0%OpenRouter · single run (Tirtha 3-seed)valid
Reliability trap set competitor comparison 1 result · 2026-07-16
2026-07-16traps served without wrong tool100.0%OpenRouter · trapsvalid
BFCL v4 Tirtha on qwen3-coder tier 2 results · 2026-07-16
2026-07-16accuracy (our harness)94.0%prod · escalate-parallel OFFvalid
2026-07-16accuracy (our harness)95.0%prod · escalate-parallel ONvalid
Tirtha tool-path marginal cost (model) 1 result · 2026-07-16
2026-07-16cost per successful task$0.00008 · what-if modeldescriptive-only
BFCL v4 Tirtha accuracy (V2 schema wall) 2 results · 2026-07-15
2026-07-15accuracy (our harness)93.0%prod :8000 · shipped configvalid
2026-07-15accuracy (our harness)95.0%prod :8000 · parallelvalid
BFCL v4 Tirtha latency/reliability 1 result · 2026-07-15
2026-07-15p50 latency3.48 sprod :8000 · shipped configvalid
  1. Sample sizes run from n=25 to n=800; 17 of 34 results carry a confidence interval or p-value.
  2. Results dated 2026-07-15 to 2026-10-17; hosts: DeepInfra, OpenRouter, api.tirtha.ai, prod, prod :8000.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. A cost labelled 'our cost' is our own spend; it is never ranked against a vendor's price, which is what a customer pays. Tīrtha Frontier is one live night against sealed bars; a replicate is booked before any public claim.
  5. Also logged as: Tirtha, tirtha, tirtha (GLM low-cost tier), tirtha (api.tirtha.ai, gateway 3fca0e99: Base Qwen3-Coder-480B on DeepInfra, checked, escalating to Plus Qwen3-235B), tirtha (live prod 6651b821), tirtha (qwen3-coder low-cost tier), tirtha (qwen3-coder-480B low-cost tier), tirtha (shape-split ladder). Hosts: DeepInfra, OpenRouter, api.tirtha.ai, prod, prod :8000.

Qwen (Alibaba) · Model

qwen3-coder-480b

10tests95results68valid#1of 3 on Drafter candidates: tool accuracy vs genera…

accuracy95.2%2 of 3n=800 · Drafter candidates: tool accuracy vs ge…
traps passed100.0%1 of 3n=40 · Drafter candidates: tool accuracy vs ge…
typical speed1.18 sno shared fieldmedian per call · Ladder M1: is Qwen3-Coder-480B alone sa…
cost per right answer$0.00012no shared fieldlist price · Muse Glimmer 30B vs low-cost rung, tool ca…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 4 shared tests; best place #1 of 3 on Drafter candidates: tool accuracy vs ge….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Escalation ladder rungs on BFCL 2400 + traps#3 of 3 · accuracy bfcl-par…
91.3%
Drafter candidates: tool accuracy vs generation rate#2 of 3 · BFCL overall accu…
95.2%
Drafter candidates: tool accuracy vs generation rate#1 of 3 · BFCL parallel acc…
92.0%
Drafter candidates: tool accuracy vs generation rate#1 of 3 · trap accuracy
100.0%
80%100%
Every scored result over timeEach dot is one accuracy-type result; a tight band means it behaves the same run to run.
90%100%2026-09-23: 92.8% · Pinned escalation ladder re-run, BFCL n=24002026-09-23: 100.0% · Pinned escalation ladder re-run, BFCL n=24002026-09-23: 92.8% · Pinned escalation ladder re-run, BFCL n=24002026-09-23: 91.3% · Escalation ladder rungs on BFCL 2400 + traps2026-09-23: 92.8% · Escalation ladder rungs on BFCL 2400 + traps2026-09-23: 94.2% · Escalation ladder rungs on BFCL 2400 + traps2026-09-23: 100.0% · Escalation ladder rungs on BFCL 2400 + traps2026-08-07: 94.0% · Live ladder BFCL v4 five passes (per pass)2026-08-07: 93.8% · Live ladder BFCL v4 five passes (per pass)2026-08-07: 93.8% · Live ladder BFCL v4 five passes (per pass)2026-08-07: 93.6% · Live ladder BFCL v4 five passes (per pass)2026-08-07: 93.6% · Live ladder BFCL v4 five passes (per pass)2026-08-06: 93.8% · Live ladder BFCL v4 five passes2026-08-06: 100.0% · Live ladder trap set five passes2026-08-06: 93.8% · Five-pass BFCL + traps on the live ladder2026-08-06: 100.0% · Five-pass BFCL + traps on the live ladder2026-07-30: 93.9% · Shape-split ladder BFCL v4, official AST rescore2026-07-30: 94.2% · Shape-split ladder BFCL v4, official AST rescore2026-07-30: 93.5% · Shape-split ladder BFCL v4, official AST rescore2026-07-30: 93.5% · Shape-split ladder BFCL v4, official AST rescore2026-07-29: 93.6% · BFCL shape-split prod: Qwen3-Coder-480B drafter2026-07-29: 93.5% · BFCL shape-split prod: Qwen3-Coder-480B drafter2026-07-29: 95.2% · Drafter candidates: tool accuracy vs generation rate2026-07-29: 92.0% · Drafter candidates: tool accuracy vs generation rate2026-07-29: 100.0% · Drafter candidates: tool accuracy vs generation rateJul 29Sep 23
All 95 results15 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Muse Glimmer 30B vs low-cost rung (Qwen3-Coder 480B), tool calls, n=80 3 results · 2026-10-14
2026-10-14pass80/80 (100.0%)together · incumbentvalid
2026-10-14latency p501.5 stogether · incumbentvalid
2026-10-14cost per pass$0.00012together · incumbentvalid
Glimmer 30B @Together, reasoning effort=minimal, vs low-cost rung (Qwen3-Coder 480B), tool calls 4 results · 2026-10-14
2026-10-14pass12/12 (100.0%)deepinfra · incumbent arm-A frozendescriptive-only
2026-10-14median completion tokens69/12 (575.0%)deepinfra · incumbent arm-A frozendescriptive-only
2026-10-14latency p501687 msdeepinfra · incumbent arm-A frozendescriptive-only
2026-10-14cost per pass$0.00020deepinfra · incumbent arm-A frozendescriptive-only
Ladder M2: which second low-cost model / pair / anchored trio should check Qwen3-Coder-480B on tool calls (GROK-prereg-ladder-m2) 42 results · 2026-10-14
2026-10-14wrong-served rate (served and wrong / all)0.3%DeepInfra/OpenRouter + Bedrock escalation · pair:glm47flashvalid
2026-10-14wrong-served rate (served and wrong / all)0.5%DeepInfra/OpenRouter + Bedrock escalation · pair:gptoss120bvalid
2026-10-14wrong-served rate (served and wrong / all)0.8%DeepInfra/OpenRouter + Bedrock escalation · pair:dsv4flashvalid
2026-10-14wrong-served rate (served and wrong / all)0.3%DeepInfra/OpenRouter + Bedrock escalation · pair:musesparkvalid
2026-10-14wrong-served rate (served and wrong / all)0.5%DeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+gptoss120bvalid
2026-10-14wrong-served rate (served and wrong / all)1.0%DeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+dsv4flashvalid
2026-10-14wrong-served rate (served and wrong / all)0.3%DeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+musesparkvalid
2026-10-14wrong-served rate (served and wrong / all)1.0%DeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+dsv4flashvalid
2026-10-14wrong-served rate (served and wrong / all)0.5%DeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+musesparkvalid
2026-10-14wrong-served rate (served and wrong / all)1.0%DeepInfra/OpenRouter + Bedrock escalation · trio:dsv4flash+musesparkvalid
2026-10-14served-low-cost fraction68.1%DeepInfra/OpenRouter + Bedrock escalation · pair:glm47flashvalid
2026-10-14served-low-cost fraction74.6%DeepInfra/OpenRouter + Bedrock escalation · pair:gptoss120bvalid
2026-10-14served-low-cost fraction72.6%DeepInfra/OpenRouter + Bedrock escalation · pair:dsv4flashvalid
2026-10-14served-low-cost fraction76.9%DeepInfra/OpenRouter + Bedrock escalation · pair:musesparkvalid
2026-10-14served-low-cost fraction80.9%DeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+gptoss120bvalid
2026-10-14served-low-cost fraction77.4%DeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+dsv4flashvalid
2026-10-14served-low-cost fraction84.9%DeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+musesparkvalid
2026-10-14served-low-cost fraction81.4%DeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+dsv4flashvalid
2026-10-14served-low-cost fraction87.7%DeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+musesparkvalid
2026-10-14served-low-cost fraction85.4%DeepInfra/OpenRouter + Bedrock escalation · trio:dsv4flash+musesparkvalid
2026-10-14p50 low-cost-stage latency (max of parallel arms)3.106 sDeepInfra/OpenRouter + Bedrock escalation · pair:glm47flashdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)1.257 sDeepInfra/OpenRouter + Bedrock escalation · pair:gptoss120bdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)11.159 sDeepInfra/OpenRouter + Bedrock escalation · pair:dsv4flashdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)2.334 sDeepInfra/OpenRouter + Bedrock escalation · pair:musesparkdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)3.113 sDeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+gptoss120bdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)11.191 sDeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+dsv4flashdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)3.387 sDeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+musesparkdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)11.159 sDeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+dsv4flashdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)2.356 sDeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+musesparkdescriptive-only
2026-10-14p50 low-cost-stage latency (max of parallel arms)11.191 sDeepInfra/OpenRouter + Bedrock escalation · trio:dsv4flash+musesparkdescriptive-only
2026-10-14PREREG decisionpair:gptoss120b · decisionvalid
2026-10-14M1 wrong-rate after blind 2-family re-adjudication (confirmed wrong only)4.8%DeepInfra direct · qwenvalid
2026-10-14$/correct vs frontier-always0.403496 ratioDeepInfra/OpenRouter + Bedrock escalation · pair:glm47flashvalid
2026-10-14$/correct vs frontier-always0.334871 ratioDeepInfra/OpenRouter + Bedrock escalation · pair:gptoss120bvalid
2026-10-14$/correct vs frontier-always0.354061 ratioDeepInfra/OpenRouter + Bedrock escalation · pair:dsv4flashvalid
2026-10-14$/correct vs frontier-always0.760248 ratioDeepInfra/OpenRouter + Bedrock escalation · pair:musesparkvalid
2026-10-14$/correct vs frontier-always0.290568 ratioDeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+gptoss120bvalid
2026-10-14$/correct vs frontier-always0.321535 ratioDeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+dsv4flashvalid
2026-10-14$/correct vs frontier-always0.686207 ratioDeepInfra/OpenRouter + Bedrock escalation · trio:glm47flash+musesparkvalid
2026-10-14$/correct vs frontier-always0.276627 ratioDeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+dsv4flashvalid
2026-10-14$/correct vs frontier-always0.656812 ratioDeepInfra/OpenRouter + Bedrock escalation · trio:gptoss120b+musesparkvalid
2026-10-14$/correct vs frontier-always0.684263 ratioDeepInfra/OpenRouter + Bedrock escalation · trio:dsv4flash+musesparkvalid
Ladder M1: is Qwen3-Coder-480B alone safe enough that a second low-cost opinion is redundant? (PREREG-m1 + A1) 7 results · 2026-10-14
2026-10-14wrong-rate, trap stratum11.7%DeepInfra direct · qwenvalid
2026-10-14wrong-rate, pooled (300 two-family-checked traps + 100 unused BFCL exec), strict PREREG normalisation10.0%DeepInfra direct · qwenvalid
2026-10-14wrong-rate, bfcl stratum5.0%DeepInfra direct · qwenvalid
2026-10-14latency p501.18 sDeepInfra direct · qwenvalid
2026-10-14VOID rate0.0%DeepInfra direct · qwenvalid
2026-10-14P(frontier wrong | Qwen wrong)40.0%DeepInfra + Bedrock · overlapdescriptive-only
2026-10-14$/item$0.00025DeepInfra direct · qwenvalid
Pinned escalation ladder re-run, BFCL n=2400 8 results · 2026-09-23
2026-09-23trap accuracy40/40 (100.0%)DeepInfra · low-cost mode=lowvalid
2026-09-23latency p500.87 sDeepInfra · low-cost mode=lowvalid
2026-09-23cost per correct answer$0.00024DeepInfra · low-cost mode=lowvalid
2026-09-23blended cost per 1M tokens$0.35DeepInfra · low (low-cost)valid
2026-09-23arm cost (provider-reported)$0.51DeepInfra · low (low-cost)valid
2026-09-23arm cost (harness estimate)$0.57DeepInfra · low (low-cost)VOID
2026-09-23accuracy92.8%DeepInfra · low-cost mode=lowvalid
2026-09-23accuracy92.8%DeepInfra · low (low-cost)valid
Escalation ladder rungs on BFCL 2400 + traps 6 results · 2026-09-23
2026-09-23traps correct40/40 (100.0%)DeepInfra · lowdescriptive-only
2026-09-23latency p902.11 sDeepInfra · lowdescriptive-only
2026-09-23latency p500.87 sDeepInfra · lowdescriptive-only
2026-09-23accuracy bfcl-simple92.8%DeepInfra · lowdescriptive-only
2026-09-23accuracy bfcl-parallel91.3%DeepInfra · low (live LOWCOST_MODEL_TOOLS)descriptive-only
2026-09-23accuracy bfcl-multiple94.2%DeepInfra · lowdescriptive-only
Live ladder BFCL v4 five passes (per pass) 5 results · 2026-08-07
2026-08-07accuracy (our harness)752/800 (94.0%)api.tirtha.ai (DeepInfra) · pass 1valid
2026-08-07accuracy (our harness)750/800 (93.8%)api.tirtha.ai (DeepInfra) · pass 2valid
2026-08-07accuracy (our harness)750/800 (93.8%)api.tirtha.ai (DeepInfra) · pass 3valid
2026-08-07accuracy (our harness)749/800 (93.6%)api.tirtha.ai (DeepInfra) · pass 4valid
2026-08-07accuracy (our harness)749/800 (93.6%)api.tirtha.ai (DeepInfra) · pass 5valid
Live ladder BFCL v4 within-model flip rate 1 result · 2026-08-07
2026-08-07item flip rate2.9%api.tirtha.ai (DeepInfra) · max over 10 pass-pairsvalid
Live ladder BFCL v4 five passes 1 result · 2026-08-06
2026-08-06accuracy (our harness)93.8%api.tirtha.ai (DeepInfra) · 5 passesvalid
Live ladder trap set five passes 1 result · 2026-08-06
2026-08-06trap success200/200 (100.0%)api.tirtha.ai (DeepInfra) · traps x5valid
Five-pass BFCL + traps on the live ladder 2 results · 2026-08-06
2026-08-06traps declined correctly200/200 (100.0%) · trap set, primary pathvalid
2026-08-06accuracy (mean of 5 passes)93.8% · BFCL v4 N=800, five passesvalid
Live ladder rungs: parallel tool-call capability 1 result · 2026-08-06
2026-08-06parallel tool calls emitted6/6 (100.0%)DeepInfra · low-cost-1 toolsdescriptive-only
Shape-split ladder BFCL v4, official AST rescore 4 results · 2026-07-30
2026-07-30accuracy (official AST)93.9%DeepInfra · overallvalid
2026-07-30accuracy (official AST)94.2%DeepInfra · simplevalid
2026-07-30accuracy (official AST)93.5%DeepInfra · multiplevalid
2026-07-30accuracy (official AST)93.5%DeepInfra · parallelvalid
BFCL shape-split prod: Qwen3-Coder-480B drafter 6 results · 2026-07-29
2026-07-29trap wrong-served0/40 (0.0%)DeepInfra · live tools drafter (cascade)valid
2026-07-29success93.6%DeepInfra · live tools drafter (cascade)valid
2026-07-29parallel accuracy93.5%DeepInfra · live tools drafter (cascade)valid
2026-07-29latency p901.36 sDeepInfra · live tools drafter (cascade)valid
2026-07-29latency p500.85 sDeepInfra · live tools drafter (cascade)valid
2026-07-29cost per successful task$0.00024DeepInfra · live tools drafter (cascade)valid
Drafter candidates: tool accuracy vs generation rate 4 results · 2026-07-29
2026-07-29trap accuracy40/40 (100.0%)DeepInfra · solodescriptive-only
2026-07-29generation rate (900 tok)187.5 tok/sDeepInfra · solodescriptive-only
2026-07-29BFCL parallel accuracy92.0%DeepInfra · solodescriptive-only
2026-07-29BFCL overall accuracy95.2%DeepInfra · solodescriptive-only
  1. Sample sizes run from n=1 to n=2488; 23 of 95 results carry a confidence interval or p-value.
  2. Results dated 2026-07-29 to 2026-10-14; hosts: DeepInfra, DeepInfra + Bedrock, DeepInfra direct, DeepInfra/OpenRouter + Bedrock escalation, api.tirtha.ai (DeepInfra), deepinfra and others.
  3. 1 VOID result kept and marked: those runs broke their own rules and are not scores.
  4. Also logged as: Qwen/Qwen3-Coder-480B-A35B-Instruct-Turbo, Qwen/Qwen3-Coder-480B-A35B-Instruct-Turbo (live prod ladder), Qwen3-Coder-480B, Qwen3-Coder-480B (tools tier), Qwen3-Coder-480B + DeepSeek-V4-Flash, Qwen3-Coder-480B + DeepSeek-V4-Flash + Muse Spark 1.3 (standard, reasoning low, tool_choice auto), Qwen3-Coder-480B + GLM-4.7-Flash, Qwen3-Coder-480B + GLM-4.7-Flash + DeepSeek-V4-Flash, Qwen3-Coder-480B + GLM-4.7-Flash + Muse Spark 1.3 (standard, reasoning low, tool_choice auto), Qwen3-Coder-480B + GLM-4.7-Flash + gpt-oss-120b-Turbo (reasoning low), Qwen3-Coder-480B + Muse Spark 1.3 (standard, reasoning low, tool_choice auto), Qwen3-Coder-480B + Sonnet 4.6, Qwen3-Coder-480B + gpt-oss-120b-Turbo (reasoning low), Qwen3-Coder-480B + gpt-oss-120b-Turbo (reasoning low) + DeepSeek-V4-Flash, Qwen3-Coder-480B + gpt-oss-120b-Turbo (reasoning low) + Muse Spark 1.3 (standard, reasoning low, tool_choice auto), Qwen3-Coder-480B-A35B-Instruct-Turbo, Qwen3-Coder-480B-A35B-Instruct-Turbo (tools tier), Qwen3-Coder-480B-Turbo, decision, qwen3-coder-480b. Hosts: DeepInfra, DeepInfra + Bedrock, DeepInfra direct, DeepInfra/OpenRouter + Bedrock escalation, api.tirtha.ai (DeepInfra), deepinfra, together.

xAI · Model

grok-4.5

9tests24results16valid#2of 8 on 800 BFCL tasks by task type (same tasks, sa…

accuracy93.4%5 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed98.3%2 of 8n=120 · 800 BFCL tasks by task type
typical speed1.85 s3 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.00154 of 7list price · 800 BFCL tasks by task type

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on traps (98.3%), weakest on parallel calls (92.0%); ahead of Sonnet 5 on 1 of 4 task types.

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
94.0%
several toolsn=200
93.5%
parallel callsn=200
92.0%
trapsn=120
98.3%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
5
worse93.4% · 5 of 8better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
3
worse1.85 s · 3 of 8better
Cost per right answer800 BFCL tasks by task type (same tasks, same scorer)
4
worse$0.0015 · 4 of 7better
All 24 results9 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Offline cost per correct tool call 3 results · 2026-10-10
2026-10-10mean cost per task$0.0014OpenRouter · directdescriptive-only
2026-10-10cost per correct$0.0015OpenRouter · directdescriptive-only
2026-10-10correct (outcome==success)865/920 (94.0%)OpenRouter · directdescriptive-only
Frontier tool-calling 6 arms N=800 1 result · 2026-09-25
2026-09-25accuracy93.4%pinned · standard setvalid
800 BFCL tasks by task type (same tasks, same scorer) 9 results · 2026-09-25
2026-09-25trap success (all calls scored)118/120 (98.3%)valid
2026-09-25tokens out per call (mean)115.1 tokensvalid
2026-09-25tokens in per call (mean)460.3 tokensvalid
2026-09-25thinking tokens per call (mean, where reported)74.6 tokensvalid
2026-09-25latency p50 (seconds per call)1.85 sxAIvalid
2026-09-25cost per right answer$0.0015valid
2026-09-25accuracy, single call94.0%valid
2026-09-25accuracy, several tools to choose from93.5%valid
2026-09-25accuracy, parallel calls92.0%valid
Frontier tool-calling BFCL full n=800 2 results · 2026-09-25
2026-09-25emitted ONE call when several needed9/200 (4.5%) · bfcl-parallelvalid
2026-09-25accuracy93.4% · standard set (run frontier-full-n800)valid
Frontier pilot trap scores, artifact shapes removed 2 results · 2026-09-24
2026-09-24trap score99/100 (99.0%) · trap set run 1, S3+S12 removedvalid
2026-09-24trap score99/100 (99.0%) · trap set run 2, S3+S12 removedvalid
Frontier pilot: bfcl-simple only + wrongness signals 1 result · 2026-09-24
2026-09-24tok_out ratio (wrong/right)1.41 ratio · pilot bfcl-simple, wrong vs rightdescriptive-only
Frontier tool-call pilot run 2: bootstrap CIs 2 results · 2026-09-24
2026-09-24item flip rate2.1% · run1 -> run2valid
2026-09-24accuracy97.7% · run 2, artifact shapes excludedvalid
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy94.5%OpenRouter (unpinned) · frontier sweepdescriptive-only
BFCL frontier bake-off (Grok 4.5 vs Sonnet 5) 3 results · 2026-07-14
2026-07-14latency p501.66 sOpenRouter · BFCL v4 N=800descriptive-only
2026-07-14cost per successful task$0.0021OpenRouter · BFCL v4 N=800descriptive-only
2026-07-14accuracy92.5%OpenRouter · BFCL v4 N=800descriptive-only
  1. Sample sizes run from n=100 to n=920; 5 of 24 results carry a confidence interval or p-value.
  2. Results dated 2026-07-14 to 2026-10-10; hosts: OpenRouter, OpenRouter (unpinned), pinned, xAI.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: grok-4.5, openrouter:x-ai/grok-4.5, x-ai/grok-4.5. Hosts: OpenRouter, OpenRouter (unpinned), pinned, xAI.

Qwen (Alibaba) · Model

qwen3-235b

9tests72results26valid#1of 3 on Escalation ladder rungs on BFCL 2400 + traps

accuracy95.5%1 of 3n=800 · Drafter candidates: tool accuracy vs ge…
traps passed75.0%5 of 5n=40 · Value rung per task shape: code vs tools
typical speed51.00 sno shared fieldmedian per call · Gateway catch on qwen3-235b
cost per right answer$0.00007no shared fieldlist price · Pinned escalation ladder re-run, BFCL n…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 2 of 6 shared tests; best place #1 of 3 on Escalation ladder rungs on BFCL 2400 +….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Value rung per task shape: code vs tools#5 of 5 · trap accuracy
75.0%
Escalation ladder rungs on BFCL 2400 + traps#1 of 3 · accuracy bfcl-par…
95.7%
Drafter candidates: tool accuracy vs generation rate#1 of 3 · BFCL overall accu…
95.5%
Drafter candidates: tool accuracy vs generation rate#2 of 3 · BFCL parallel acc…
92.0%
Drafter candidates: tool accuracy vs generation rate#2 of 3 · trap accuracy
87.5%
Value rung per task shape: code vs tools#3 of 3 · accuracy
57.1%
50%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyValue rung per task shape: code vs tools
5
worse75.0% · 5 of 5better
All 72 results12 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Correction note on 40 traps, qwen3-235b 16 results · 2026-10-13
2026-10-13usage costNoneDeepInfra · all armsinconclusive
2026-10-13trap-only T success (primary, 30 traps x 3 reps)63/90 (70.0%)DeepInfrainconclusive
2026-10-13trap-only T success (primary, 30 traps x 3 reps)61/90 (67.8%)DeepInfra · CORRECTIONinconclusive
2026-10-13trap-only T success (primary, 30 traps x 3 reps)64/90 (71.1%)DeepInfra · PLACEBOinconclusive
2026-10-13success, 720-call total (T+excluded artifacts+P)651/720 (90.4%)DeepInfrainconclusive
2026-10-13success, 720-call total (T+excluded artifacts+P)656/720 (91.1%)DeepInfra · CORRECTIONinconclusive
2026-10-13success, 720-call total (T+excluded artifacts+P)657/720 (91.2%)DeepInfra · PLACEBOinconclusive
2026-10-13decoys served (all on T; same count in 720 total)27/90 (30.0%)DeepInfrainconclusive
2026-10-13decoys served (all on T; same count in 720 total)29/90 (32.2%)DeepInfra · CORRECTIONinconclusive
2026-10-13decoys served (all on T; same count in 720 total)26/90 (28.9%)DeepInfra · PLACEBOinconclusive
2026-10-13T paired success difference-2.2 ptsDeepInfra · CORRECTION vs NONEinconclusive
2026-10-13T paired success difference-3.3 ptsDeepInfra · CORRECTION vs PLACEBOinconclusive
2026-10-13T paired success difference1.1 ptsDeepInfra · PLACEBO vs NONEinconclusive
2026-10-13P normal-call success558/600 (93.0%)DeepInfravalid
2026-10-13P normal-call success565/600 (94.2%)DeepInfra · CORRECTIONvalid
2026-10-13P normal-call success563/600 (93.8%)DeepInfra · PLACEBOvalid
Gateway catch-and-retry, qwen3-235b 17 results · 2026-10-09
2026-10-09usage cost capturedNoneDeepInfra · all armsvalid
2026-10-09trigger rate on T (as implemented, all T items counted decoy-present)27/300 (9.0%)DeepInfra · catch triggerVOID
2026-10-09trigger fired on T rows with emitted decoy (corrected §6a)27/27 (100.0%)DeepInfra · catch triggervalid
2026-10-09T trap success (success/all)253/300 (84.3%)DeepInfravalid
2026-10-09T trap success (success/all)279/300 (93.0%)DeepInfra · SPECIFICvalid
2026-10-09T trap success (success/all)261/300 (87.0%)DeepInfra · GENERICvalid
2026-10-09T paired success difference8.7 ptsDeepInfra · SPECIFIC vs NONEvalid
2026-10-09T paired success difference6 ptsDeepInfra · SPECIFIC vs GENERICvalid
2026-10-09T paired success difference2.7 ptsDeepInfra · GENERIC vs NONEvalid
2026-10-09T miss rate (incorrect first call, no trigger)20/300 (6.7%)DeepInfra · catch triggervalid
2026-10-09P parallel success (success/all)558/600 (93.0%)DeepInfravalid
2026-10-09P parallel success (success/all)558/600 (93.0%)DeepInfra · SPECIFICvalid
2026-10-09P parallel success (success/all)558/600 (93.0%)DeepInfra · GENERICvalid
2026-10-09P false triggers0/600 (0.0%)DeepInfra · SPECIFIC/GENERICvalid
2026-10-09M BFCL-multiple success (success/all)524/600 (87.3%)DeepInfraVOID
2026-10-09M BFCL-multiple success (success/all)528/600 (88.0%)DeepInfra · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)527/600 (87.8%)DeepInfra · GENERICVOID
Gateway catch level 2 (qwen3-235b, DeepInfra) 5 results · 2026-10-08 to 2026-10-09
2026-10-09emitted decoys triggered27/27 (100.0%)DeepInfra · T set, SPECIFIC vs NONEvalid
2026-10-09decoy first calls emitted27 countDeepInfra · T set, SPECIFIC vs NONEvalid
2026-10-08latency p9063 sDeepInfra · first calldescriptive-only
2026-10-08latency p5051 sDeepInfra · first calldescriptive-only
2026-10-08api_error voids46 countDeepInfra · alldescriptive-only
Gateway catch (level 2) on qwen3-235b 4 results · 2026-10-09
2026-10-09p-value0.00195312 pDeepInfra · SPECIFIC vs NONEvalid
2026-10-09first-call latency p5051 sDeepInfra · first callsdescriptive-only
2026-10-09decoys triggered27/27 (100.0%)DeepInfra · T first callsvalid
2026-10-09decoys emitted27 countDeepInfra · T first callsvalid
Pinned escalation ladder re-run, BFCL n=2400 6 results · 2026-09-23
2026-09-23wall-clock per task11.3 sDeepInfra · value (killed partial run)partial
2026-09-23trap accuracy30/40 (75.0%)DeepInfra · value (frontier-value)valid
2026-09-23cost per correct answer$0.00007DeepInfra · value (frontier-value)valid
2026-09-23blended cost per 1M tokens$0.13DeepInfra · valuevalid
2026-09-23arm cost (measured)$0.0324DeepInfra · value (killed partial)partial
2026-09-23accuracy (of answered)95.6%DeepInfra · value (frontier-value)valid
Escalation ladder rungs on BFCL 2400 + traps 7 results · 2026-09-23
2026-09-23traps correct32/40 (80.0%)DeepInfra · valuedescriptive-only
2026-09-23latency p9011.09 sDeepInfra · valuedescriptive-only
2026-09-23latency p503.56 sDeepInfra · valuedescriptive-only
2026-09-23extra correct answers (all 2400)37/2400 (1.5%)DeepInfra · value vs lowdescriptive-only
2026-09-23accuracy bfcl-simple94.9%DeepInfra · valuedescriptive-only
2026-09-23accuracy bfcl-parallel95.7%DeepInfra · valuedescriptive-only
2026-09-23accuracy bfcl-multiple93.0%DeepInfra · valuedescriptive-only
Cascade vs arms vs oracle on HumanEval+ / BFCL 1 result · 2026-08-16
2026-08-16HumanEval+ plus pass89.6% · single armvalid
Live ladder rungs: parallel tool-call capability 1 result · 2026-08-06
2026-08-06parallel tool calls emitted5/5 (100.0%)DeepInfra · frontier-valuedescriptive-only
Drafter candidates: tool accuracy vs generation rate 4 results · 2026-07-29
2026-07-29trap accuracy35/40 (87.5%)Nebius · solodescriptive-only
2026-07-29generation rate (900 tok)79.1 tok/sNebius · solodescriptive-only
2026-07-29BFCL parallel accuracy92.0%Nebius · solodescriptive-only
2026-07-29BFCL overall accuracy95.5%Nebius · solodescriptive-only
Solo BFCL: gpt-oss-120b vs qwen3-235b 2 results · 2026-07-29
2026-07-29latency p501.18 s · solo, identical harnessdescriptive-only
2026-07-29accuracy95.5% · solo, identical harnessdescriptive-only
Value-rung swap: tool-path re-check and prod latency 3 results · 2026-07-26
2026-07-26trap accuracy30/40 (75.0%) · trapsdescriptive-only
2026-07-26standard success90/100 (90.0%) · synthetic tool benchdescriptive-only
2026-07-26latency p501.79 s · prod after swapdescriptive-only
Value rung per task shape: code vs tools 6 results · 2026-07-26
2026-07-26trap accuracy75.0% · tools trapsdescriptive-only
2026-07-26success92.5% · tools (synthetic, paired)descriptive-only
2026-07-26rescue rate26.9% · escalate COLDdescriptive-only
2026-07-26rescue rate27.9% · escalate REPAIRdescriptive-only
2026-07-26pass@177.9% · code (HumanEval+ + MBPP+)descriptive-only
2026-07-26accuracy24/42 (57.1%) · escalated subset (witness disagrees)descriptive-only
  1. Sample sizes run from n=1 to n=2400; 14 of 72 results carry a confidence interval or p-value.
  2. Results dated 2026-07-26 to 2026-10-13; hosts: DeepInfra, Nebius.
  3. 4 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  5. Also logged as: Qwen/Qwen3-235B-A22B-Instruct-2507, Qwen/Qwen3-235B-A22B-Instruct-2507 (qwen3-235b), Qwen3-235B, Qwen3-235B-A22B, Qwen3-235B-A22B-Instruct-2507, qwen3-235b, qwen3-235b-a22b-2507. Hosts: DeepInfra, Nebius.

Anthropic · Model

sonnet-4.5

9tests33results19valid#1of 8 on BFCL v4 competitor comparison

accuracy95.9%1 of 8n=800 · BFCL v4 competitor comparison
traps passed100.0%1 of 5n=40 · Value rung per task shape: code vs tools
typical speed6.65 sno shared fieldmedian per call · Sonnet 4.5 vs 4.6 frontier-rung head-to…
cost per right answer$0.0049no shared fieldlist price · Pinned escalation ladder re-run, BFCL n…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 5 of 6 shared tests; best place #1 of 8 on BFCL v4 competitor comparison.

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
BFCL v4 competitor comparison#1 of 8 · accuracy (our har…
95.9%
Frontier tool-call pilot run 2: bootstrap CIs#1 of 6 · accuracy
98.6%
Value rung per task shape: code vs tools#1 of 5 · trap accuracy
100.0%
Escalation ladder rungs on BFCL 2400 + traps#2 of 3 · accuracy bfcl-par…
95.5%
Cascade vs arms vs oracle on HumanEval+ / BFCL#1 of 3 · BFCL accuracy
96.0%
Value rung per task shape: code vs tools#1 of 3 · accuracy
73.8%
55%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyBFCL v4 competitor comparison
1
worse95.9% · 1 of 8better
All 33 results12 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Sonnet 4.5 vs 4.6 frontier-rung head-to-head 5 results · 2026-10-09
2026-10-09wrong tool chosen on traps0/120 (0.0%)AWS Bedrock · sonnetvalid
2026-10-09trap set success119/120 (99.2%)AWS Bedrock · sonnetvalid
2026-10-09standard set success385/400 (96.2%)AWS Bedrock · sonnetvalid
2026-10-09latency p506.65 sAWS Bedrock · sonnetvalid
2026-10-09$/task mean$0.0037AWS Bedrock · sonnetvalid
Frontier pilot: bfcl-simple only + wrongness signals 1 result · 2026-09-24
2026-09-24tok_out ratio (wrong/right)1.5 ratio · pilot bfcl-simple, wrong vs rightdescriptive-only
Frontier tool-call pilot run 2: bootstrap CIs 2 results · 2026-09-24
2026-09-24item flip rate2.5% · run1 -> run2valid
2026-09-24accuracy98.6% · run 2, artifact shapes excludedvalid
Escalation low-cost vs frontier paired 1 result · 2026-09-23
2026-09-23accuracy2325/2400 (96.9%)pinned · frontier (mode=max)valid
Pinned escalation ladder re-run, BFCL n=2400 7 results · 2026-09-23
2026-09-23wall-clock per task6 s · max (partial value run)descriptive-only
2026-09-23latency p505.99 sBedrock · max mode=maxvalid
2026-09-23cost per correct answer$0.0049Bedrock · max mode=maxvalid
2026-09-23arm cost (metered at $3/$15)$10.52Bedrock · maxvalid
2026-09-23arm cost (AWS billed)$11.57Bedrock · maxvalid
2026-09-23accuracy96.9%Bedrock · max mode=maxvalid
2026-09-23accuracy96.9%Bedrock · maxvalid
Escalation ladder rungs on BFCL 2400 + traps 1 result · 2026-09-23
2026-09-23accuracy bfcl-parallel95.5%Bedrock · maxdescriptive-only
Cascade vs arms vs oracle on HumanEval+ / BFCL 2 results · 2026-08-16
2026-08-16latency p502.27 sAmazon Bedrock · BFCL addendum, pinnedvalid
2026-08-16BFCL accuracy96.0%Amazon Bedrock · BFCL addendum, pinnedvalid
Value rung per task shape: code vs tools 7 results · 2026-07-26 to 2026-07-27
2026-07-27rescue rate48.1% · escalate PRODSHAPEdescriptive-only
2026-07-26trap accuracy100.0% · tools trapsdescriptive-only
2026-07-26success97.5% · tools (synthetic, paired)descriptive-only
2026-07-26rescue rate33.7% · escalate COLDdescriptive-only
2026-07-26rescue rate51.9% · escalate REPAIRdescriptive-only
2026-07-26pass@182.4% · code (HumanEval+ + MBPP+)descriptive-only
2026-07-26accuracy31/42 (73.8%) · escalated subset (witness disagrees)descriptive-only
openrouter/auto vs our low-cost tier, paired BFCL 1 result · 2026-07-27
2026-07-27trap accuracy100.0% · trap setvalid
Tool-arg convention errors by price tier 4 results · 2026-07-27
2026-07-27overall accuracy96.2%OpenRouter · BFCL-style harnessdescriptive-only
2026-07-27missed (no call)10/440 (2.3%)OpenRouter · arg-extractiondescriptive-only
2026-07-27cost per task$0.0037OpenRouter · BFCL-style harnessdescriptive-only
2026-07-27convention-shaped arg errors0/440 (0.0%)OpenRouter · BFCL-style harnessdescriptive-only
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)95.9%OpenRouter · single run (Tirtha 3-seed)valid
Reliability trap set competitor comparison 1 result · 2026-07-16
2026-07-16traps served without wrong tool100.0%OpenRouter · trapsvalid
  1. Sample sizes run from n=40 to n=2440; 10 of 33 results carry a confidence interval or p-value.
  2. Results dated 2026-07-16 to 2026-10-09; hosts: AWS Bedrock, Amazon Bedrock, Bedrock, OpenRouter, pinned.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: Sonnet, Sonnet-4.5, anthropic.claude-sonnet-4-5-20250929-v1:0, anthropic.claude-sonnet-4-5-20250929-v1:0 (Sonnet 4.5), anthropic/claude-sonnet-4.5, claude-sonnet-4-5, claude-sonnet-4.5, sonnet, sonnet-4-5, sonnet-4.5, us.anthropic.claude-sonnet-4-5, us.anthropic.claude-sonnet-4-5 (Bedrock). Hosts: AWS Bedrock, Amazon Bedrock, Bedrock, OpenRouter, pinned.

Meta · Model

llama-3.3-70b

8tests47results28valid

accuracy93.4%no shared fieldn=800 · Llama 3.3 70B pinned to DeepInfra, 800…
traps passed91.7%no shared fieldn=120 · Llama 3.3 70B pinned to DeepInfra, 800…
typical speed2.66 sno shared fieldmedian per call · Llama 3.3 70B pinned to DeepInfra, 800…

47 results so far, latest: accuracy on Llama 3.3 70B pinned to DeepInfra, 800 BFCL + 120….

Every scored result over timeEach dot is one accuracy-type result; a tight band means it behaves the same run to run.
70%80%90%100%2026-10-17: 93.4% · Llama 3.3 70B pinned to DeepInfra, 800 BFCL + 120 traps (Tirtha Frontier second-opinion candidate)2026-10-17: 91.7% · Llama 3.3 70B pinned to DeepInfra, 800 BFCL + 120 traps (Tirtha Frontier second-opinion candidate)2026-10-09: 70.0% · Gateway catch across families (Together)2026-10-09: 70.0% · Gateway catch across families (Together)2026-10-09: 70.0% · Gateway catch across families (Together)2026-10-09: 94.7% · Gateway catch across families (Together)2026-10-09: 94.7% · Gateway catch across families (Together)2026-10-09: 94.7% · Gateway catch across families (Together)2026-10-09: 71.0% · Gateway catch across families (Together)2026-10-09: 71.0% · Gateway catch across families (Together)2026-10-09: 71.0% · Gateway catch across families (Together)2026-10-08: 79.6% · Per-model mistake atlas, first calls (Together)2026-08-02: 75.0% · Failover slot llama-3.3-70b on Groq: traps2026-08-02: 95.5% · Failover slot llama-3.3-70b on Groq: standard tasks2026-08-02: 95.5% · Failover slot 2 (llama-3.3-70b) on tool set2026-08-02: 75.0% · Failover slot 2 (llama-3.3-70b) on tool set2026-08-02: 78.8% · Failover slot 2 (llama-3.3-70b) on tool set2026-08-02: 95.5% · Provider lottery: llama-3.3-70b pinned vs unpinned2026-08-02: 78.2% · Provider lottery: llama-3.3-70b pinned vs unpinned2026-08-02: 78.2% · Provider lottery: pinned vs unpinned same model2026-08-02: 92.5% · Provider lottery: pinned vs unpinned same model2026-08-02: 95.5% · Provider lottery: pinned vs unpinned same model2026-08-02: 78.2% · Provider lottery: pinned vs unpinned same model2026-07-29: 81.0% · Slot 2 llama-3.3-70b earlier BFCL run2026-07-29: 92.5% · Slot 2 llama-3.3-70b earlier BFCL runJul 29Oct 17
All 47 results11 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Llama 3.3 70B pinned to DeepInfra, 800 BFCL + 120 traps (Tirtha Frontier second-opinion candidate) 4 results · 2026-10-17
2026-10-17trap success (all calls scored)91.7%OpenRouter -> DeepInfravalid
2026-10-17latency p502.66 sOpenRouter -> DeepInfravalid
2026-10-17cost per task (vendor)$0.00005OpenRouter -> DeepInfravalid
2026-10-17accuracy93.4%OpenRouter -> DeepInfravalid
Gateway catch across families (Together) 11 results · 2026-10-09
2026-10-09trigger fired overall2/1500 (0.1%)Together · catch triggervalid
2026-10-09run cost (usage)$0.80Together · all armsvalid
2026-10-09T trap success (success/all)210/300 (70.0%)Togethervalid
2026-10-09T trap success (success/all)210/300 (70.0%)Together · SPECIFICvalid
2026-10-09T trap success (success/all)210/300 (70.0%)Together · GENERICvalid
2026-10-09P parallel success (success/all)426/600 (71.0%)Togethervalid
2026-10-09P parallel success (success/all)426/600 (71.0%)Together · SPECIFICvalid
2026-10-09P parallel success (success/all)426/600 (71.0%)Together · GENERICvalid
2026-10-09M BFCL-multiple success (success/all)568/600 (94.7%)Togethervalid
2026-10-09M BFCL-multiple success (success/all)568/600 (94.7%)Together · SPECIFICvalid
2026-10-09M BFCL-multiple success (success/all)568/600 (94.7%)Together · GENERICvalid
Per-model mistake atlas, first calls (Together) 9 results · 2026-10-08
2026-10-08run cost$0.79Together · NONE (first call)descriptive-only
2026-10-08mistake shape: wrong_tool30/1500 (2.0%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: schema_wrapped_args24/1500 (1.6%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: missed_call137/1500 (9.1%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: false_call60/1500 (4.0%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: decoy_extra_call2/1500 (0.1%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: arg_wrong_VALUE63/1500 (4.2%)Together · NONE (first call)descriptive-only
2026-10-08latency p504.01 sTogether · NONE (first call)descriptive-only
2026-10-08first-call success (T+M+P sets)1194/1500 (79.6%)Together · NONE (first call)descriptive-only
Live ladder rungs: parallel tool-call capability 1 result · 2026-08-06
2026-08-06parallel tool calls emitted5/5 (100.0%)Groq · low-cost-2 bothdescriptive-only
Failover slot llama-3.3-70b on Groq: traps 1 result · 2026-08-02
2026-08-02trap success30/40 (75.0%)Groq (via OpenRouter) · failover slot, Groq-pinnedvalid
Failover slot llama-3.3-70b on Groq: standard tasks 1 result · 2026-08-02
2026-08-02standard task success382/400 (95.5%)Groq (via OpenRouter) · failover slot, Groq-pinnedvalid
Failover slot 2 (llama-3.3-70b) on tool set 4 results · 2026-08-02
2026-08-02trap success75.0%Groq (pinned, = prod) · trapsvalid
2026-08-02success95.5%Groq (pinned, = prod) · standardvalid
2026-08-02success78.8%Groq (pinned, = prod) · no-tool categoryvalid
2026-08-02latency p500.43 sGroq (pinned, = prod) · standardvalid
Provider lottery: llama-3.3-70b pinned vs unpinned 2 results · 2026-08-02
2026-08-02tool-call accuracy95.5%Groq · pinned to Groqvalid
2026-08-02tool-call accuracy78.2%OpenRouter (unpinned) · unpinnedvalid
Provider lottery: pinned vs unpinned same model 5 results · 2026-08-02
2026-08-02trap success92.5%OpenRouter (unpinned) · unpinned trapsvalid
2026-08-02success78.2%OpenRouter (unpinned) · unpinned standardvalid
2026-08-02success95.5%Groq (pinned) · pinnedvalid
2026-08-02success78.2%OpenRouter (unpinned) · unpinnedvalid
2026-08-02latency p501.6 sOpenRouter (unpinned) · unpinned standardvalid
Parallel tool-call capability per provider x model 7 results · 2026-07-30
2026-07-30parallel tool-call probe passed0/6 (0.0%)Together (via OpenRouter) · provider pinned: Togetherdescriptive-only
2026-07-30parallel tool-call probe passed6/6 (100.0%)Groq (via OpenRouter) · provider pinned: Groqdescriptive-only
2026-07-30parallel tool-call probe passed5/6 (83.3%)DeepInfra (via OpenRouter) · provider pinned: DeepInfradescriptive-only
2026-07-30parallel tool-call probe passed0/6 (0.0%)CoreWeave (via OpenRouter) · provider pinned: CoreWeavedescriptive-only
2026-07-30parallel tool-call probe passed0/6 (0.0%)Nebius (via OpenRouter) · provider pinned: Nebiusdescriptive-only
2026-07-30parallel tool-call probe passed0/6 (0.0%)Novita (via OpenRouter) · provider pinned: Novitadescriptive-only
2026-07-30parallel tool-call probe passed0/6 (0.0%)AkashML (via OpenRouter) · provider pinned: AkashMLdescriptive-only
Slot 2 llama-3.3-70b earlier BFCL run 2 results · 2026-07-29
2026-07-29trap success37/40 (92.5%) · trapsdescriptive-only
2026-07-29accuracy81.0% · BFCLdescriptive-only
  1. Sample sizes run from n=5 to n=1500; 6 of 47 results carry a confidence interval or p-value.
  2. Results dated 2026-07-29 to 2026-10-17; hosts: AkashML (via OpenRouter), CoreWeave (via OpenRouter), DeepInfra (via OpenRouter), Groq, Groq (pinned), Groq (pinned, = prod) and others.
  3. Also logged as: llama-3.3-70b, llama-3.3-70b-instruct, meta-llama/Llama-3.3-70B-Instruct-Turbo, meta-llama/llama-3.3-70b-instruct, openrouter:meta-llama/llama-3.3-70b-instruct (pinned DeepInfra, fp8). Hosts: AkashML (via OpenRouter), CoreWeave (via OpenRouter), DeepInfra (via OpenRouter), Groq, Groq (pinned), Groq (pinned, = prod), Groq (via OpenRouter), Nebius (via OpenRouter), Novita (via OpenRouter), OpenRouter (unpinned), OpenRouter -> DeepInfra, Together, Together (via OpenRouter).

DeepSeek · Model

deepseek-v4-flash

7tests51results14valid#1of 5 on Mistake atlas: first-call tool accuracy acr…

accuracy91.0%7 of 8n=800 · BFCL v4 competitor comparison
traps passed100.0%3 of 5n=50 · Trap set per shape
typical speed2.01 s3 of 5median per call · Frontier refresh BFCL v4 latency

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on single calls (32.8%), weakest on parallel calls (8.5%); behind Sonnet 5 on all 4 task types.

Accuracy by task typedeepseek-v41-flash-1017-2026-10-18 (auto-logged run). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
32.8%
several toolsn=200
9.5%
parallel callsn=200
8.5%
trapsn=120
22.5%
0%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyBFCL v4 competitor comparison
7
worse91.0% · 7 of 8better
Speed, typical seconds per callFrontier refresh BFCL v4 latency (from Mac)
3
worse2.01 s · 3 of 5better
All 51 results15 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
deepseek-v41-flash-1017-2026-10-18 (auto-logged run) 9 results · 2026-10-18
2026-10-18trap success (all calls scored)22.5%unknownVOID
2026-10-18tokens out per call (mean)23.8 tokensunknownVOID
2026-10-18latency p9514.56 sunknownVOID
2026-10-18latency p505.26 sunknownVOID
2026-10-18cost per task$0.00001unknownVOID
2026-10-18accuracy, single call32.8%unknownVOID
2026-10-18accuracy, several tools to choose from9.5%unknownVOID
2026-10-18accuracy, parallel calls8.5%unknownVOID
2026-10-18accuracy20.9%unknownVOID
Ladder M2: which second low-cost model / pair / anchored trio should check Qwen3-Coder-480B on tool calls (GROK-prereg-ladder-m2) 3 results · 2026-10-14
2026-10-14rho_err = P(candidate wrong | Qwen wrong), K=19 confirmed21.1%DeepInfra direct · dsv4flashvalid
2026-10-14latency p5011.159 sDeepInfra direct · dsv4flashvalid
2026-10-14$/item$0.00007DeepInfra direct · dsv4flashvalid
Gateway catch across families (Together) 11 results · 2026-10-09
2026-10-09trigger fired overall12/1500 (0.8%)Together · catch triggerVOID
2026-10-09run cost (usage)$0.19Together · all armsVOID
2026-10-09T trap success (success/all)299/300 (99.7%)TogetherVOID
2026-10-09T trap success (success/all)300/300 (100.0%)Together · SPECIFICVOID
2026-10-09T trap success (success/all)300/300 (100.0%)Together · GENERICVOID
2026-10-09P parallel success (success/all)546/600 (91.0%)TogetherVOID
2026-10-09P parallel success (success/all)546/600 (91.0%)Together · SPECIFICVOID
2026-10-09P parallel success (success/all)546/600 (91.0%)Together · GENERICVOID
2026-10-09M BFCL-multiple success (success/all)570/600 (95.0%)TogetherVOID
2026-10-09M BFCL-multiple success (success/all)575/600 (95.8%)Together · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)571/600 (95.2%)Together · GENERICVOID
Gateway catch across families: mistake atlas (Together) 3 results · 2026-10-09
2026-10-09success rate (answered denominator)100.0%Together · ATLAS NONE, T trapsdescriptive-only
2026-10-09success rate (answered denominator)94.7%Together · ATLAS NONE, M bfcl-multipledescriptive-only
2026-10-09success rate (answered denominator)89.7%Together · ATLAS NONE, P bfcl-paralleldescriptive-only
Mistake atlas: first-call tool accuracy across families 3 results · 2026-10-09
2026-10-09success rate (answered denominator)100.0%Together · ATLAS_ONLY T (traps)descriptive-only
2026-10-09success rate (answered denominator)94.7%Together · ATLAS_ONLY M (bfcl-multiple)descriptive-only
2026-10-09success rate (answered denominator)89.7%Together · ATLAS_ONLY P (bfcl-parallel)descriptive-only
Per-model mistake atlas, first calls (Together) 8 results · 2026-10-08
2026-10-08run cost$0.18Together · NONE (first call)descriptive-only
2026-10-08mistake shape: wrong_tool4/1500 (0.3%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: missed_call16/1500 (1.1%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: decoy_extra_call13/1500 (0.9%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: arg_wrong_VALUE71/1500 (4.7%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: answered_in_text5/1500 (0.3%)Together · NONE (first call)descriptive-only
2026-10-08latency p505.51 sTogether · NONE (first call)descriptive-only
2026-10-08first-call success (T+M+P sets)1406/1500 (93.7%)Together · NONE (first call)descriptive-only
Trap rows serving a non-gold extra call 1 result · 2026-08-06
2026-08-06served non-gold extra call scored success3/50 (6.0%) · trap setdescriptive-only
Trap rows that served a non-gold extra call 1 result · 2026-08-06
2026-08-06rows serving a non-gold extra call3/50 (6.0%) · trap setdescriptive-only
Frontier refresh BFCL v4 pinned arms 1 result · 2026-08-06
2026-08-06accuracy (our harness)747/800 (93.4%)DeepInfra · provider-pinnedvalid
Frontier refresh BFCL v4 per category 3 results · 2026-08-06
2026-08-06accuracy (our harness)377/400 (94.2%)DeepInfra · simplevalid
2026-08-06accuracy (our harness)189/200 (94.5%)DeepInfra · multiplevalid
2026-08-06accuracy (our harness)181/200 (90.5%)DeepInfra · parallelvalid
Frontier refresh BFCL v4 latency (from Mac) 1 result · 2026-08-06
2026-08-06p50 latency2.01 sDeepInfra · provider-pinneddescriptive-only
Trap set per shape (frontier refresh arms) 4 results · 2026-08-06
2026-08-06trap success50/50 (100.0%) · S1 forecastvalid
2026-08-06trap success50/50 (100.0%) · S2 destinationvalid
2026-08-06trap success50/50 (100.0%) · S3 poundsvalid
2026-08-06trap success50/50 (100.0%) · S4 no-toolvalid
Frontier refresh benchmark, BFCL-parallel 1 result · 2026-07-30
2026-07-30BFCL-parallel accuracy90.5%valid
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)91.0%OpenRouter · single run (Tirtha 3-seed)valid
Reliability trap set competitor comparison 1 result · 2026-07-16
2026-07-16traps served without wrong tool52.0%OpenRouter · trapsvalid
  1. Sample sizes run from n=19 to n=1500; 7 of 51 results carry a confidence interval or p-value.
  2. Results dated 2026-07-16 to 2026-10-18; hosts: DeepInfra, DeepInfra direct, OpenRouter, Together, unknown.
  3. 11 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  5. Also logged as: DeepSeek-V4-Flash, DeepSeek-V4-Flash-0731, deepseek-ai/DeepSeek-V4-Flash-0731, deepseek-v4-flash, deepseek/deepseek-v4-flash-0731, openrouter:deepseek/deepseek-v4-flash. Hosts: DeepInfra, DeepInfra direct, OpenRouter, Together, unknown.

xAI · Model

grok-4.7

7tests20results16valid#1of 6 on Offline cost per correct tool call

accuracy95.4%2 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed97.5%3 of 8n=120 · 800 BFCL tasks by task type
typical speed1.55 s2 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.00153 of 7list price · 800 BFCL tasks by task type

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on single calls (97.5%), weakest on parallel calls (92.5%); ahead of Sonnet 5 on 3 of 4 task types; one of the fastest (2 of 8).

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
97.5%
several toolsn=200
94.0%
parallel callsn=200
92.5%
trapsn=120
97.5%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
2
worse95.4% · 2 of 8better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
2
worse1.55 s · 2 of 8better
Cost per right answer800 BFCL tasks by task type (same tasks, same scorer)
3
worse$0.0015 · 3 of 7better
All 20 results7 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Offline cost per correct tool call 3 results · 2026-10-10
2026-10-10mean cost per task$0.0014OpenRouter · directdescriptive-only
2026-10-10cost per correct$0.0015OpenRouter · directdescriptive-only
2026-10-10correct (outcome==success)880/920 (95.7%)OpenRouter · directdescriptive-only
Frontier tool-calling 6 arms N=800 1 result · 2026-09-25
2026-09-25accuracy95.4%pinned · standard setvalid
800 BFCL tasks by task type (same tasks, same scorer) 9 results · 2026-09-25
2026-09-25trap success (all calls scored)117/120 (97.5%)valid
2026-09-25tokens out per call (mean)86.4 tokensvalid
2026-09-25tokens in per call (mean)1496.3 tokensvalid
2026-09-25thinking tokens per call (mean, where reported)49.7 tokensvalid
2026-09-25latency p50 (seconds per call)1.55 sxAIvalid
2026-09-25cost per right answer$0.0015valid
2026-09-25accuracy, single call97.5%valid
2026-09-25accuracy, several tools to choose from94.0%valid
2026-09-25accuracy, parallel calls92.5%valid
Frontier tool-calling BFCL full n=800 2 results · 2026-09-25
2026-09-25emitted ONE call when several needed3/200 (1.5%) · bfcl-parallelvalid
2026-09-25accuracy95.4% · standard set (run frontier-full-n800)valid
Frontier pilot trap scores, artifact shapes removed 2 results · 2026-09-24
2026-09-24trap score96/100 (96.0%) · trap set run 1, S3+S12 removedvalid
2026-09-24trap score97/100 (97.0%) · trap set run 2, S3+S12 removedvalid
Frontier pilot: bfcl-simple only + wrongness signals 1 result · 2026-09-24
2026-09-24tok_out ratio (wrong/right)0.8 ratio · pilot bfcl-simple, wrong vs rightdescriptive-only
Frontier tool-call pilot run 2: bootstrap CIs 2 results · 2026-09-24
2026-09-24item flip rate1.2% · run1 -> run2valid
2026-09-24accuracy97.7% · run 2, artifact shapes excludedvalid
  1. Sample sizes run from n=100 to n=920; 5 of 20 results carry a confidence interval or p-value.
  2. Results dated 2026-09-24 to 2026-10-10; hosts: OpenRouter, pinned, xAI.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: grok-4.7, openrouter:x-ai/grok-4.7, x-ai/grok-4.7. Hosts: OpenRouter, pinned, xAI.

Meta · Model

muse-glimmer-30b

6tests25results3valid

accuracy97.0%no shared fieldn=100 · Nemotron 3.5 and Muse-Glimmer run 2 on…
traps passed65.0%no shared fieldn=40 · Muse-Glimmer-30B adversarial traps
typical speed5.61 sno shared fieldmedian per call · Muse Glimmer 30B vs low-cost rung, tool ca…
cost per right answer$0.00059no shared fieldlist price · Muse Glimmer 30B vs low-cost rung, tool ca…

25 results so far, latest: pass on Muse Glimmer 30B vs low-cost rung (Qwen3-Coder 480B)….

Every scored result over timeEach dot is one accuracy-type result; a tight band means it behaves the same run to run.
60%70%80%90%100%2026-08-12: 97.0% · Nemotron 3.5 and Muse-Glimmer run 2 on tool path2026-08-12: 70.0% · Nemotron 3.5 and Muse-Glimmer run 2 on tool path2026-08-11: 96.0% · Muse-Glimmer-30B synthetic tool-calling2026-08-11: 65.0% · Muse-Glimmer-30B adversarial traps2026-08-11: 65.0% · Muse-Glimmer-30B solo on the trap set (first measurement)Aug 11Aug 12
All 25 results7 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Muse Glimmer 30B vs low-cost rung (Qwen3-Coder 480B), tool calls, n=80 3 results · 2026-10-14
2026-10-14pass79/80 (98.8%)together · candidatevalid
2026-10-14latency p505.61 stogether · candidatevalid
2026-10-14cost per pass$0.00059together · candidatevalid
Glimmer 30B @Together, reasoning effort=minimal, vs low-cost rung (Qwen3-Coder 480B), tool calls 11 results · 2026-10-14
2026-10-14pass11/12 (91.7%)together · candidate (effort=minimal)descriptive-only
2026-10-14pass12/12 (100.0%)together · candidate arm-A default reasoningdescriptive-only
2026-10-14median reasoning tokens98/12 (816.7%)together · candidate (effort=minimal)descriptive-only
2026-10-14median reasoning tokens245.5/12 (2045.8%)together · candidate arm-A default reasoningdescriptive-only
2026-10-14median completion tokens204/12 (1700.0%)together · candidate (effort=minimal)descriptive-only
2026-10-14median completion tokens355.5/12 (2962.5%)together · candidate arm-A default reasoningdescriptive-only
2026-10-14latency p506274 mstogether · candidate (effort=minimal)descriptive-only
2026-10-14latency p506300 mstogether · candidate arm-A default reasoningdescriptive-only
2026-10-14decisive verdictNonetogether · candidate (effort=minimal)VOID
2026-10-14cost per pass$0.00048together · candidate (effort=minimal)descriptive-only
2026-10-14cost per pass$0.00068together · candidate arm-A default reasoningdescriptive-only
Nemotron 3.5 and Muse-Glimmer run 2 on tool path 5 results · 2026-08-12
2026-08-12success/answered97.0%Together · standarddescriptive-only
2026-08-12success/answered70.0%Together · trapsdescriptive-only
2026-08-12right tool wrong argument11/40 (27.5%)Together · trapsdescriptive-only
2026-08-12latency p501.58 sTogether · standarddescriptive-only
2026-08-12cost per successful task$0.00040Together · standarddescriptive-only
Listed model ids probed for tool-call capability 1 result · 2026-08-12
2026-08-12tool call emitted1/1 (100.0%)OpenRouter · single probe calldescriptive-only
Muse-Glimmer-30B synthetic tool-calling 1 result · 2026-08-11
2026-08-11accuracy96/100 (96.0%)OpenRouter (unpinned) · standarddescriptive-only
Muse-Glimmer-30B adversarial traps 1 result · 2026-08-11
2026-08-11trap success26/40 (65.0%)OpenRouter (unpinned) · trapsdescriptive-only
Muse-Glimmer-30B solo on the trap set (first measurement) 3 results · 2026-08-11
2026-08-11wrong-tool failures1/40 (2.5%) · solo, trapsdescriptive-only
2026-08-11traps correct26/40 (65.0%) · solo, trapsdescriptive-only
2026-08-11right-tool wrong-argument failures13/40 (32.5%) · solo, trapsdescriptive-only
  1. Sample sizes run from n=1 to n=100; 4 of 25 results carry a confidence interval or p-value.
  2. Results dated 2026-08-11 to 2026-10-14; hosts: OpenRouter, OpenRouter (unpinned), Together, together.
  3. 1 VOID result kept and marked: those runs broke their own rules and are not scores.
  4. Also logged as: meta/muse-glimmer-30b, muse-glimmer-30b. Hosts: OpenRouter, OpenRouter (unpinned), Together, together.

Z.ai (GLM) · Model

glm-5.2

5tests17results2valid#1of 3 on Value rung per task shape: code vs tools

accuracy88.0%8 of 8n=800 · BFCL v4 competitor comparison
traps passed92.5%3 of 5n=40 · Value rung per task shape: code vs tools
typical speed14.10 sno shared fieldmedian per call · Value rung per task shape: code vs tools
cost per right answer$0.0033no shared fieldlist price · Value rung per task shape: code vs tools

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 3 shared tests; best place #1 of 3 on Value rung per task shape: code vs tools.

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
BFCL v4 competitor comparison#8 of 8 · accuracy (our har…
88.0%
Value rung per task shape: code vs tools#3 of 5 · trap accuracy
92.5%
Value rung per task shape: code vs tools#1 of 3 · tool accuracy
97.5%
80%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyBFCL v4 competitor comparison
8
worse88.0% · 8 of 8better
All 17 results7 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 6 results · 2026-07-27
2026-07-27trap accuracy92.5% · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27tool accuracy97.5% · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27latency p503.15 s · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27latency p5014.1 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.0033 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@189.6% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
Tool-arg convention errors by price tier 3 results · 2026-07-27
2026-07-27overall accuracy97.0%OpenRouter · BFCL-style harnessdescriptive-only
2026-07-27cost per task$0.00051OpenRouter · BFCL-style harnessdescriptive-only
2026-07-27convention-shaped arg errors13/440 (3.0%)OpenRouter · BFCL-style harnessdescriptive-only
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)88.0%OpenRouter · single run (Tirtha 3-seed)valid
Reliability trap set competitor comparison 1 result · 2026-07-16
2026-07-16traps served without wrong tool92.5%OpenRouter · trapsvalid
Low-cost ensemble agreement-routing vs Sonnet 2 results · 2026-07-16
2026-07-16trap accuracy37/40 (92.5%)OpenRouter · solo value-rung run trapsdescriptive-only
2026-07-16accuracy88.0%OpenRouter · solo value-rung rundescriptive-only
Tool wall over-escalation diagnosis (BFCL-800) 2 results · 2026-07-14
2026-07-14accuracy88.2%OpenRouter · alonedescriptive-only
2026-07-14accuracy92.8%OpenRouter · alone, on 553 escalated itemsdescriptive-only
Tool wall over-escalation settling run (BFCL-800) 2 results · 2026-07-14
2026-07-14accuracy88.2%OpenRouter · GLM alonedescriptive-only
2026-07-14accuracy92.8%OpenRouter · GLM alone on the 553 escalated itemsdescriptive-only
  1. Sample sizes run from n=40 to n=800; 2 of 17 results carry a confidence interval or p-value.
  2. Results dated 2026-07-14 to 2026-07-27; hosts: OpenRouter.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: GLM-5.2, glm-5.2, z-ai/glm-5.2. Hosts: OpenRouter.

OpenAI · Model

gpt-5.2

5tests8results7valid#2of 6 on Frontier pilot trap scores, artifact shapes…

accuracy90.4%7 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed100.0%2 of 6n=100 · Frontier pilot trap scores, artifact sh…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 4 shared tests; best place #2 of 6 on Frontier pilot trap scores, artifact sh….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier tool-calling 6 arms N=800#7 of 8 · accuracy
90.4%
Frontier tool-calling BFCL full n=800#5 of 6 · accuracy
90.4%
Frontier tool-call pilot run 2: bootstrap CIs#4 of 6 · accuracy
97.3%
Frontier pilot trap scores, artifact shapes removed#2 of 6 · trap score
100.0%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
7
worse90.4% · 7 of 8better
All 8 results5 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier tool-calling 6 arms N=800 1 result · 2026-09-25
2026-09-25accuracy90.4%pinned · standard setvalid
Frontier tool-calling BFCL full n=800 2 results · 2026-09-25
2026-09-25emitted ONE call when several needed3/200 (1.5%) · bfcl-parallelvalid
2026-09-25accuracy90.4% · standard set (run frontier-full-n800)valid
Frontier pilot trap scores, artifact shapes removed 2 results · 2026-09-24
2026-09-24trap score100/100 (100.0%) · trap set run 1, S3+S12 removedvalid
2026-09-24trap score99/100 (99.0%) · trap set run 2, S3+S12 removedvalid
Frontier pilot: bfcl-simple only + wrongness signals 1 result · 2026-09-24
2026-09-24tok_out ratio (wrong/right)1.14 ratio · pilot bfcl-simple, wrong vs rightdescriptive-only
Frontier tool-call pilot run 2: bootstrap CIs 2 results · 2026-09-24
2026-09-24item flip rate5.8% · run1 -> run2valid
2026-09-24accuracy97.3% · run 2, artifact shapes excludedvalid
  1. Sample sizes run from n=100 to n=800; 3 of 8 results carry a confidence interval or p-value.
  2. Results dated 2026-09-24 to 2026-09-25; hosts: pinned.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gpt-5.2. Hosts: pinned.

Anthropic · Model

opus-5.5

5tests19results15valid#1of 8 on 800 BFCL tasks by task type (same tasks, sa…

accuracy95.4%1 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed85.0%8 of 8n=120 · 800 BFCL tasks by task type
typical speed2.07 s4 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.00567 of 7list price · 800 BFCL tasks by task type

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on single calls (97.0%), weakest on traps (85.0%); ahead of Sonnet 5 on 2 of 4 task types; among the priciest per right answer (7 of 7).

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
97.0%
several toolsn=200
94.0%
parallel callsn=200
93.5%
trapsn=120
85.0%
80%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
1
worse95.4% · 1 of 8better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
4
worse2.07 s · 4 of 8better
Cost per right answer800 BFCL tasks by task type (same tasks, same scorer)
7
worse$0.0056 · 7 of 7better
All 19 results5 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Offline cost per correct tool call 3 results · 2026-10-10
2026-10-10mean cost per task$0.0053OpenRouter · directdescriptive-only
2026-10-10cost per correct$0.0056OpenRouter · directdescriptive-only
2026-10-10correct (outcome==success)865/920 (94.0%)OpenRouter · directdescriptive-only
Frontier tool-calling 6 arms N=800 1 result · 2026-09-25
2026-09-25accuracy95.4%pinned · standard setvalid
800 BFCL tasks by task type (same tasks, same scorer) 9 results · 2026-09-25
2026-09-25trap success (all calls scored)102/120 (85.0%)valid
2026-09-25tokens out per call (mean)146.4 tokensvalid
2026-09-25tokens in per call (mean)610.1 tokensvalid
2026-09-25thinking tokens per call (mean, where reported)5.8 tokensvalid
2026-09-25latency p50 (seconds per call)2.07 sAmazon Bedrockvalid
2026-09-25cost per right answer$0.0056valid
2026-09-25accuracy, single call97.0%valid
2026-09-25accuracy, several tools to choose from94.0%valid
2026-09-25accuracy, parallel calls93.5%valid
Frontier tool-calling BFCL full n=800 2 results · 2026-09-25
2026-09-25emitted ONE call when several needed1/200 (0.5%) · bfcl-parallelvalid
2026-09-25accuracy95.4% · standard set (run frontier-full-n800)valid
Frontier pilot trap scores, artifact shapes removed 4 results · 2026-09-24
2026-09-24trap score96/100 (96.0%) · trap set run 1, S3+S12 removedvalid
2026-09-24trap score98/100 (98.0%) · trap set run 2, S3+S12 removedvalid
2026-09-24origin/destination inversions0/18 (0.0%) · S12 get_directions failuresvalid
2026-09-24failures (run 1)20/120 (16.7%) · all 120 trap items, run1/run2descriptive-only
  1. Sample sizes run from n=18 to n=920; 4 of 19 results carry a confidence interval or p-value.
  2. Results dated 2026-09-24 to 2026-10-10; hosts: Amazon Bedrock, OpenRouter, pinned.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: anthropic/claude-opus-5.5, openrouter:anthropic/claude-opus-5.5, opus-5.5. Hosts: Amazon Bedrock, OpenRouter, pinned.

Qwen (Alibaba) · Model

qwen3-coder

5tests32results12valid#1of 5 on Frontier refresh BFCL v4 pinned arms

accuracy93.6%4 of 8n=800 · BFCL v4 competitor comparison
traps passed100.0%1 of 5n=50 · Trap set per shape
typical speed1.44 s2 of 5median per call · Frontier refresh BFCL v4 latency
cost per right answer$0.00014no shared fieldlist price · Value rung per task shape: code vs tools

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 3 of 4 shared tests; best place #1 of 5 on Frontier refresh BFCL v4 pinned arms.

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
BFCL v4 competitor comparison#4 of 8 · accuracy (our har…
93.6%
Frontier refresh BFCL v4 pinned arms#1 of 5 · accuracy (our har…
95.2%
Trap set per shape (frontier refresh arms)#1 of 5 · trap success
100.0%
Frontier refresh benchmark, BFCL-parallel#1 of 3 · BFCL-parallel acc…
92.5%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyBFCL v4 competitor comparison
4
worse93.6% · 4 of 8better
Speed, typical seconds per callFrontier refresh BFCL v4 latency (from Mac)
2
worse1.44 s · 2 of 5better
All 32 results13 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Contradiction test pilot (note vs inverted note) 4 results · 2026-10-08
2026-10-08spend (balance delta, 69 runs incl. smoke + stopped legs)$12.30OpenRouter (Messages bridge) · alldescriptive-only
2026-10-08mean pilot REPAIR30.0%OpenRouter (Messages bridge) · note armsdescriptive-only
2026-10-08mean pilot FOLLOW30.0%OpenRouter (Messages bridge) · INVERTEDdescriptive-only
2026-10-08BASELINE_CORRECT traps5/10 (50.0%)OpenRouter (Messages bridge) · EMPTY (no note)descriptive-only
Contradiction pilot, non-Claude family (qwen3-coder) 8 results · 2026-10-08
2026-10-08cost per run$0.26OpenRouter · all cells (10 runs: 3 smoke + 7 batch)partial
2026-10-08cells banked26/60 (43.3%)OpenRouter · resumed batchpartial
2026-10-08OTHER1/2 (50.0%)OpenRouter · EMPTY (none)partial
2026-10-08OTHER1/2 (50.0%)OpenRouter · INVERTEDpartial
2026-10-08FOLLOWED_INVERTED1/3 (33.3%)OpenRouter · RIGHT (helpful-refuse)partial
2026-10-08FOLLOWED_INVERTED1/2 (50.0%)OpenRouter · INVERTEDpartial
2026-10-08CORRECT1/2 (50.0%)OpenRouter · EMPTY (none)partial
2026-10-08CORRECT2/3 (66.7%)OpenRouter · RIGHT (helpful-refuse)partial
Frontier refresh BFCL v4 pinned arms 1 result · 2026-08-06
2026-08-06accuracy (our harness)762/800 (95.2%)DeepInfra · provider-pinnedvalid
Frontier refresh BFCL v4 per category 3 results · 2026-08-06
2026-08-06accuracy (our harness)388/400 (97.0%)DeepInfra · simplevalid
2026-08-06accuracy (our harness)189/200 (94.5%)DeepInfra · multiplevalid
2026-08-06accuracy (our harness)185/200 (92.5%)DeepInfra · parallelvalid
Frontier refresh BFCL v4 latency (from Mac) 1 result · 2026-08-06
2026-08-06p50 latency1.44 sDeepInfra · provider-pinneddescriptive-only
Frontier refresh incumbent run B (429 storm) 1 result · 2026-08-06
2026-08-06accuracy on answered rows532/565 (94.2%)DeepInfra · run B (235 errors)partial
Frontier refresh same-model noise floor 1 result · 2026-08-06
2026-08-06item flip rate13/565 (2.3%)DeepInfra · B vs B2 repeatvalid
Trap set per shape (frontier refresh arms) 4 results · 2026-08-06
2026-08-06trap success50/50 (100.0%) · S1 forecastvalid
2026-08-06trap success50/50 (100.0%) · S2 destinationvalid
2026-08-06trap success50/50 (100.0%) · S3 poundsvalid
2026-08-06trap success50/50 (100.0%) · S4 no-toolvalid
Frontier refresh benchmark, BFCL-parallel 1 result · 2026-07-30
2026-07-30BFCL-parallel accuracy92.5%valid
Value rung per task shape: code vs tools 3 results · 2026-07-27
2026-07-27latency p501.3 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.00014 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@179.3% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
qwen3-coder BFCL baseline (OpenRouter) 3 results · 2026-07-19
2026-07-19trap success100.0%OpenRouter · trapsdescriptive-only
2026-07-19success96.7%OpenRouter · BFCLdescriptive-only
2026-07-19latency p501.68 sOpenRouter · BFCLdescriptive-only
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)93.6%OpenRouter · single run (Tirtha 3-seed)valid
Reliability trap set competitor comparison 1 result · 2026-07-16
2026-07-16traps served without wrong tool100.0%OpenRouter · trapsvalid
  1. Sample sizes run from n=2 to n=800; 4 of 32 results carry a confidence interval or p-value.
  2. Results dated 2026-07-16 to 2026-10-08; hosts: DeepInfra, OpenRouter, OpenRouter (Messages bridge).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: qwen/qwen3-coder, qwen3-coder, qwen3-coder-solo. Hosts: DeepInfra, OpenRouter, OpenRouter (Messages bridge).

Microsoft · Decision model

decision-1

4tests10results0valid

A decision model checks tool calls; it does not make them. So the numbers below are about checking, not calling, and it is never ranked against the models that make calls. Its scored checks (wrong calls let through, cost per check, speed) sit beside the other checkers on the leaderboard: open its decision-model card →

checks right91.7%no shared fieldn=120 · Decision-1 as tool-call checker on trap…
typical speed0.34 sno shared fieldmedian per call · Decision-1 as tool-call checker on trap…

10 results so far, latest: P(yes) range (min..max) on Decision-1 repeat-request noise.

All 10 results4 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Decision-1 repeat-request noise 1 result · 2026-10-11
2026-10-11P(yes) range (min..max)0.999089..0.99929 · 6 byte-identical requestsdescriptive-only
Decision-1 tool-call verifier prereg (universe v2) 1 result · 2026-10-10
2026-10-10conflicting (task, call) items dropped31/3931 (0.8%) · universe build (burned)VOID
Decision-1 tool-call verifier prereg (first universe) 1 result · 2026-10-10
2026-10-10wrong calls in sample49 count · 11-file sample (burned)VOID
Decision-1 as tool-call checker on trap shapes 7 results · 2026-10-10
2026-10-10wrong actions accepted0/60 (0.0%) · constructed traps, 14 shapesdescriptive-only
2026-10-10spend$0.0019 · constructed traps, 14 shapesdescriptive-only
2026-10-10p50 latency344 ms · constructed traps, 14 shapesdescriptive-only
2026-10-10median wobble0.0001 score · constructed traps, 14 shapesdescriptive-only
2026-10-10median margin0.499 score · constructed traps, 14 shapesdescriptive-only
2026-10-10decision flips between identical calls0/120 (0.0%) · constructed traps, 14 shapesdescriptive-only
2026-10-10accuracy at 0.5 (first call)110/120 (91.7%) · constructed traps, 14 shapesdescriptive-only
  1. Sample sizes run from n=6 to n=3931; 0 of 10 results carry a confidence interval or p-value.
  2. Results dated 2026-10-10 to 2026-10-11.
  3. 2 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Also logged as: Microsoft-Decision-1, decision-1. Hosts: not recorded.

OpenAI · Model

gpt-5.1

4tests6results5valid#3of 6 on Frontier pilot trap scores, artifact shapes…

accuracy80.6%8 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed99.0%3 of 6n=100 · Frontier pilot trap scores, artifact sh…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 3 shared tests; best place #3 of 6 on Frontier pilot trap scores, artifact sh….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier tool-calling 6 arms N=800#8 of 8 · accuracy
80.6%
Frontier tool-call pilot run 2: bootstrap CIs#6 of 6 · accuracy
94.5%
Frontier pilot trap scores, artifact shapes removed#3 of 6 · trap score
99.0%
75%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
8
worse80.6% · 8 of 8better
All 6 results4 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier tool-calling 6 arms N=800 1 result · 2026-09-25
2026-09-25accuracy80.6%pinned · standard setvalid
Frontier pilot trap scores, artifact shapes removed 2 results · 2026-09-24
2026-09-24trap score99/100 (99.0%) · trap set run 1, S3+S12 removedvalid
2026-09-24trap score99/100 (99.0%) · trap set run 2, S3+S12 removedvalid
Frontier pilot: bfcl-simple only + wrongness signals 1 result · 2026-09-24
2026-09-24tok_out ratio (wrong/right)1.59 ratio · pilot bfcl-simple, wrong vs rightdescriptive-only
Frontier tool-call pilot run 2: bootstrap CIs 2 results · 2026-09-24
2026-09-24item flip rate8.3% · run1 -> run2valid
2026-09-24accuracy94.5% · run 2, artifact shapes excludedvalid
  1. Sample sizes run from n=100 to n=800; 3 of 6 results carry a confidence interval or p-value.
  2. Results dated 2026-09-24 to 2026-09-25; hosts: pinned.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gpt-5.1. Hosts: pinned.

OpenAI · Model

gpt-5.1-codex

4tests18results15valid#5of 8 on 800 BFCL tasks by task type (same tasks, sa…

accuracy81.8%6 of 6n=920 · Offline cost per correct tool call
traps passed90.0%5 of 8n=120 · 800 BFCL tasks by task type
typical speed2.40 s5 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.00111 of 7list price · 800 BFCL tasks by task type

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on traps (90.0%), weakest on parallel calls (61.0%); behind Sonnet 5 on all 4 task types; among the lowest-cost per right answer (1 of 7).

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
86.8%
several toolsn=200
88.0%
parallel callsn=200
61.0%
trapsn=120
90.0%
55%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyOffline cost per correct tool call
6
worse81.8% · 6 of 6better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
5
worse2.40 s · 5 of 8better
Cost per right answer800 BFCL tasks by task type (same tasks, same scorer)
1
worse$0.0011 · 1 of 7better
All 18 results4 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Offline cost per correct tool call 3 results · 2026-10-10
2026-10-10mean cost per task$0.00085OpenRouter · directdescriptive-only
2026-10-10cost per correct$0.0010OpenRouter · directdescriptive-only
2026-10-10correct (outcome==success)753/920 (81.8%)OpenRouter · directdescriptive-only
Frontier tool-calling 6 arms N=800 1 result · 2026-09-25
2026-09-25accuracy, parallel category only61.0%pinned · parallelvalid
800 BFCL tasks by task type (same tasks, same scorer) 9 results · 2026-09-25
2026-09-25trap success (all calls scored)108/120 (90.0%)valid
2026-09-25tokens out per call (mean)69.1 tokensvalid
2026-09-25tokens in per call (mean)158.9 tokensvalid
2026-09-25thinking tokens per call (mean, where reported)36.2 tokensvalid
2026-09-25latency p50 (seconds per call)2.4 sAzurevalid
2026-09-25cost per right answer$0.0011valid
2026-09-25accuracy, single call86.8%valid
2026-09-25accuracy, several tools to choose from88.0%valid
2026-09-25accuracy, parallel calls61.0%valid
Frontier tool-calling BFCL full n=800 5 results · 2026-09-25
2026-09-25emitted ONE call when several needed60/200 (30.0%) · bfcl-parallelvalid
2026-09-25accuracy80.6% · standard set (run frontier-full-n800)valid
2026-09-25accuracy61.0% · bfcl-parallelvalid
2026-09-25accuracy86.8% · bfcl-simplevalid
2026-09-25accuracy88.0% · bfcl-multiplevalid
  1. Sample sizes run from n=120 to n=920; 3 of 18 results carry a confidence interval or p-value.
  2. Results dated 2026-09-25 to 2026-10-10; hosts: Azure, OpenRouter, pinned.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gpt-5.1-codex, openai/gpt-5.1-codex, openrouter:openai/gpt-5.1-codex. Hosts: Azure, OpenRouter, pinned.

Meta · Model

muse-spark-open

3tests14results14valid#4of 8 on 800 BFCL tasks by task type (same tasks, sa…

accuracy93.2%6 of 8n=800 · Frontier tool-calling 6 arms N=800
traps passed90.0%6 of 8n=120 · 800 BFCL tasks by task type
typical speed5.51 s8 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.00205 of 7list price · 800 BFCL tasks by task type

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on single calls (95.2%), weakest on parallel calls (89.5%); behind Sonnet 5 on all 4 task types; among the slowest (8 of 8).

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
95.2%
several toolsn=200
93.0%
parallel callsn=200
89.5%
trapsn=120
90.0%
80%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-calling 6 arms N=800
6
worse93.2% · 6 of 8better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
8
worse5.51 s · 8 of 8better
Cost per right answer800 BFCL tasks by task type (same tasks, same scorer)
5
worse$0.0020 · 5 of 7better
All 14 results4 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier tool-calling 6 arms N=800 3 results · 2026-10-17
2026-10-17latency p505.52 sOpenRouter→Togethervalid
2026-10-17cost per correct$0.0020OpenRouter→Togethervalid
2026-10-17accuracy93.2%OpenRouter→Togethervalid
Muse Spark Open, BFCL n=800 by category 1 result · 2026-10-17
2026-10-17accuracy, parallel category only89.5%OpenRouter→Togethervalid
Muse Spark Open trap set 1 result · 2026-10-17
2026-10-17trap success108/120 (90.0%)OpenRouter→Togethervalid
800 BFCL tasks by task type (same tasks, same scorer) 9 results · 2026-10-17
2026-10-17trap success (all calls scored)108/120 (90.0%)valid
2026-10-17tokens out per call (mean)253.5 tokensvalid
2026-10-17tokens in per call (mean)786.7 tokensvalid
2026-10-17thinking tokens per call (mean, where reported)116.7 tokensvalid
2026-10-17latency p50 (seconds per call)5.51 sTogethervalid
2026-10-17cost per right answer$0.0020Togethervalid
2026-10-17accuracy, single call95.2%valid
2026-10-17accuracy, several tools to choose from93.0%valid
2026-10-17accuracy, parallel calls89.5%valid
  1. Sample sizes run from n=120 to n=800; 3 of 14 results carry a confidence interval or p-value.
  2. Results dated 2026-10-17 to 2026-10-17; hosts: OpenRouter→Together, Together.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: meta/muse-spark-open (OpenRouter, pinned Together), muse-spark-open, openrouter:meta/muse-spark-open. Hosts: OpenRouter→Together, Together.

Qwen (Alibaba) · Model

qwen3.5-9b

3tests36results11valid#3of 5 on Mistake atlas: first-call tool accuracy acr…

accuracy91.3%3 of 5Mistake atlas: first-call tool accuracy…
traps passed90.0%no shared fieldn=300 · Gateway catch, Qwen family + gpt-oss
typical speed5.30 sno shared fieldmedian per call · Per-model mistake atlas, first calls

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #3 of 5 on Mistake atlas: first-call tool accuracy….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Mistake atlas: first-call tool accuracy across families#3 of 5 · success rate (ans…
91.3%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyMistake atlas: first-call tool accuracy across families
3
worse91.3% · 3 of 5better
All 36 results5 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Gateway catch across families (Together) 11 results · 2026-10-09
2026-10-09trigger fired overall5/1500 (0.3%)Together · catch triggerVOID
2026-10-09run cost (usage)NoneTogether · all armsVOID
2026-10-09T trap success (success/all)275/300 (91.7%)TogetherVOID
2026-10-09T trap success (success/all)275/300 (91.7%)Together · SPECIFICVOID
2026-10-09T trap success (success/all)275/300 (91.7%)Together · GENERICVOID
2026-10-09P parallel success (success/all)526/600 (87.7%)TogetherVOID
2026-10-09P parallel success (success/all)526/600 (87.7%)Together · SPECIFICVOID
2026-10-09P parallel success (success/all)526/600 (87.7%)Together · GENERICVOID
2026-10-09M BFCL-multiple success (success/all)547/600 (91.2%)TogetherVOID
2026-10-09M BFCL-multiple success (success/all)549/600 (91.5%)Together · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)548/600 (91.3%)Together · GENERICVOID
Gateway catch across families: mistake atlas (Together) 3 results · 2026-10-09
2026-10-09success rate (answered denominator)91.3%Together · ATLAS NONE, T trapsdescriptive-only
2026-10-09success rate (answered denominator)92.8%Together · ATLAS NONE, M bfcl-multipledescriptive-only
2026-10-09success rate (answered denominator)89.0%Together · ATLAS NONE, P bfcl-paralleldescriptive-only
Gateway catch, Qwen family + gpt-oss (DeepInfra) 11 results · 2026-10-09
2026-10-09trigger fired overall6/1500 (0.4%)DeepInfra · catch triggervalid
2026-10-09run cost (usage)NoneDeepInfra · all armsvalid
2026-10-09T trap success (success/all)270/300 (90.0%)DeepInfravalid
2026-10-09T trap success (success/all)270/300 (90.0%)DeepInfra · SPECIFICvalid
2026-10-09T trap success (success/all)270/300 (90.0%)DeepInfra · GENERICvalid
2026-10-09P parallel success (success/all)537/600 (89.5%)DeepInfravalid
2026-10-09P parallel success (success/all)537/600 (89.5%)DeepInfra · SPECIFICvalid
2026-10-09P parallel success (success/all)537/600 (89.5%)DeepInfra · GENERICvalid
2026-10-09M BFCL-multiple success (success/all)558/600 (93.0%)DeepInfravalid
2026-10-09M BFCL-multiple success (success/all)561/600 (93.5%)DeepInfra · SPECIFICvalid
2026-10-09M BFCL-multiple success (success/all)561/600 (93.5%)DeepInfra · GENERICvalid
Mistake atlas: first-call tool accuracy across families 3 results · 2026-10-09
2026-10-09success rate (answered denominator)91.3%Together · ATLAS_ONLY T (traps)descriptive-only
2026-10-09success rate (answered denominator)92.8%Together · ATLAS_ONLY M (bfcl-multiple)descriptive-only
2026-10-09success rate (answered denominator)89.0%Together · ATLAS_ONLY P (bfcl-parallel)descriptive-only
Per-model mistake atlas, first calls (Together) 8 results · 2026-10-08
2026-10-08run cost$0.22Together · NONE (first call)descriptive-only
2026-10-08mistake shape: wrong_tool29/1500 (1.9%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: missed_call30/1500 (2.0%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: decoy_extra_call6/1500 (0.4%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: arg_wrong_VALUE76/1500 (5.1%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: answered_in_text9/1500 (0.6%)Together · NONE (first call)descriptive-only
2026-10-08latency p505.3 sTogether · NONE (first call)descriptive-only
2026-10-08first-call success (T+M+P sets)1365/1500 (91.0%)Together · NONE (first call)descriptive-only
  1. Sample sizes run from n=300 to n=1500; 0 of 36 results carry a confidence interval or p-value.
  2. Results dated 2026-10-08 to 2026-10-09; hosts: DeepInfra, Together.
  3. 11 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  5. Also logged as: Qwen/Qwen3.5-9B, Qwen3.5-9B, qwen3.5-9b. Hosts: DeepInfra, Together.

Z.ai (GLM) · Model

glm-4.7

2tests12results6valid#2of 5 on Value rung per task shape: code vs tools

accuracy93.5%3 of 3n=200 · Value rung per task shape: code vs tools
traps passed100.0%2 of 5n=40 · Value rung per task shape: code vs tools
typical speed0.64 sno shared fieldmedian per call · glm-4.7 on Cerebras as tools drafter
cost per right answer$0.0015no shared fieldlist price · glm-4.7 on Cerebras as tools drafter

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 2 shared tests; best place #2 of 5 on Value rung per task shape: code vs tools.

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Value rung per task shape: code vs tools#2 of 5 · trap accuracy
100.0%
Value rung per task shape: code vs tools#3 of 3 · tool accuracy
93.5%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyValue rung per task shape: code vs tools
2
worse100.0% · 2 of 5better
All 12 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
glm-4.7 on Cerebras as tools drafter (staging) 6 results · 2026-07-29
2026-07-29trap wrong-served1/40 (2.5%)Cerebras · candidate drafter (staging)valid
2026-07-29success94.4%Cerebras · candidate drafter (staging)valid
2026-07-29parallel accuracy95.5%Cerebras · candidate drafter (staging)valid
2026-07-29latency p901.29 sCerebras · candidate drafter (staging)valid
2026-07-29latency p500.64 sCerebras · candidate drafter (staging)valid
2026-07-29cost per successful task$0.0015Cerebras · candidate drafter (staging)valid
Value rung per task shape: code vs tools 6 results · 2026-07-27
2026-07-27trap accuracy100.0% · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27tool accuracy93.5% · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27latency p503.97 s · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27latency p5020.3 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.0033 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@190.2% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
  1. Sample sizes run from n=40 to n=800; 4 of 12 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-29; hosts: Cerebras.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: glm-4.7, z-ai/glm-4.7. Hosts: Cerebras.

Z.ai (GLM) · Model

glm-4.7-flash

2tests5results5valid#6of 8 on BFCL v4 competitor comparison

accuracy92.9%6 of 8n=800 · BFCL v4 competitor comparison
typical speed3.06 sno shared fieldmedian per call · Ladder M2: which second low-cost model / p…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #6 of 8 on BFCL v4 competitor comparison.

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
BFCL v4 competitor comparison#6 of 8 · accuracy (our har…
92.9%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyBFCL v4 competitor comparison
6
worse92.9% · 6 of 8better
All 5 results3 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Ladder M2: which second low-cost model / pair / anchored trio should check Qwen3-Coder-480B on tool calls (GROK-prereg-ladder-m2) 3 results · 2026-10-14
2026-10-14rho_err = P(candidate wrong | Qwen wrong), K=19 confirmed15.8%OpenRouter pinned Venice fp8 · glm47flashvalid
2026-10-14latency p503.057 sOpenRouter pinned Venice fp8 · glm47flashvalid
2026-10-14$/item$0.00010OpenRouter pinned Venice fp8 · glm47flashvalid
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)92.9%OpenRouter · single run (Tirtha 3-seed)valid
Reliability trap set competitor comparison 1 result · 2026-07-16
2026-07-16traps served without wrong tool88.0%OpenRouter · trapsvalid
  1. Sample sizes run from n=19 to n=800; 1 of 5 results carry a confidence interval or p-value.
  2. Results dated 2026-07-16 to 2026-10-14; hosts: OpenRouter, OpenRouter pinned Venice fp8.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: GLM-4.7-Flash, glm-4.7-flash. Hosts: OpenRouter, OpenRouter pinned Venice fp8.

Z.ai (GLM) · Model

glm-5.3-flash

2tests25results0valid#2of 5 on Mistake atlas: first-call tool accuracy acr…

accuracy91.7%2 of 5Mistake atlas: first-call tool accuracy…
typical speed4.32 sno shared fieldmedian per call · Per-model mistake atlas, first calls

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 1 shared tests; best place #2 of 5 on Mistake atlas: first-call tool accuracy….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Mistake atlas: first-call tool accuracy across families#2 of 5 · success rate (ans…
91.7%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyMistake atlas: first-call tool accuracy across families
2
worse91.7% · 2 of 5better
All 25 results4 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Gateway catch across families (Together) 11 results · 2026-10-09
2026-10-09trigger fired overall11/1500 (0.7%)Together · catch triggerVOID
2026-10-09run cost (usage)$0.19Together · all armsVOID
2026-10-09T trap success (success/all)281/300 (93.7%)TogetherVOID
2026-10-09T trap success (success/all)281/300 (93.7%)Together · SPECIFICVOID
2026-10-09T trap success (success/all)281/300 (93.7%)Together · GENERICVOID
2026-10-09P parallel success (success/all)549/600 (91.5%)TogetherVOID
2026-10-09P parallel success (success/all)549/600 (91.5%)Together · SPECIFICVOID
2026-10-09P parallel success (success/all)549/600 (91.5%)Together · GENERICVOID
2026-10-09M BFCL-multiple success (success/all)569/600 (94.8%)TogetherVOID
2026-10-09M BFCL-multiple success (success/all)575/600 (95.8%)Together · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)569/600 (94.8%)Together · GENERICVOID
Gateway catch across families: mistake atlas (Together) 3 results · 2026-10-09
2026-10-09success rate (answered denominator)91.7%Together · ATLAS NONE, T trapsdescriptive-only
2026-10-09success rate (answered denominator)95.0%Together · ATLAS NONE, M bfcl-multipledescriptive-only
2026-10-09success rate (answered denominator)91.5%Together · ATLAS NONE, P bfcl-paralleldescriptive-only
Mistake atlas: first-call tool accuracy across families 3 results · 2026-10-09
2026-10-09success rate (answered denominator)91.7%Together · ATLAS_ONLY T (traps)descriptive-only
2026-10-09success rate (answered denominator)95.0%Together · ATLAS_ONLY M (bfcl-multiple)descriptive-only
2026-10-09success rate (answered denominator)91.5%Together · ATLAS_ONLY P (bfcl-parallel)descriptive-only
Per-model mistake atlas, first calls (Together) 8 results · 2026-10-08
2026-10-08run cost$0.19Together · NONE (first call)descriptive-only
2026-10-08mistake shape: wrong_tool3/1500 (0.2%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: missed_call18/1500 (1.2%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: decoy_extra_call11/1500 (0.7%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: arg_wrong_VALUE83/1500 (5.5%)Together · NONE (first call)descriptive-only
2026-10-08mistake shape: answered_in_text1/1500 (0.1%)Together · NONE (first call)descriptive-only
2026-10-08latency p504.32 sTogether · NONE (first call)descriptive-only
2026-10-08first-call success (T+M+P sets)1394/1500 (92.9%)Together · NONE (first call)descriptive-only
  1. Sample sizes run from n=300 to n=1500; 0 of 25 results carry a confidence interval or p-value.
  2. Results dated 2026-10-08 to 2026-10-09; hosts: Together.
  3. 11 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  5. Also logged as: GLM-5.3-Flash, glm-5.3-flash, zai-org/GLM-5.3-Flash. Hosts: Together.

OpenAI · Model

gpt-5.2-codex

2tests12results9valid#5of 6 on Offline cost per correct tool call

accuracy89.9%5 of 6n=920 · Offline cost per correct tool call
traps passed86.7%7 of 8n=120 · 800 BFCL tasks by task type
typical speed2.46 s6 of 8median per call · 800 BFCL tasks by task type
cost per right answer$0.00132 of 7list price · 800 BFCL tasks by task type

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Best on single calls (92.5%), weakest on traps (86.7%); behind Sonnet 5 on all 4 task types.

Accuracy by task type800 BFCL tasks by task type (same tasks, same scorer). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
92.5%
several toolsn=200
89.0%
parallel callsn=200
87.5%
trapsn=120
86.7%
80%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyOffline cost per correct tool call
5
worse89.9% · 5 of 6better
Speed, typical seconds per call800 BFCL tasks by task type (same tasks, same scorer)
6
worse2.46 s · 6 of 8better
Cost per right answer800 BFCL tasks by task type (same tasks, same scorer)
2
worse$0.0013 · 2 of 7better
All 12 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Offline cost per correct tool call 3 results · 2026-10-10
2026-10-10mean cost per task$0.0011OpenRouter · directdescriptive-only
2026-10-10cost per correct$0.0013OpenRouter · directdescriptive-only
2026-10-10correct (outcome==success)827/920 (89.9%)OpenRouter · directdescriptive-only
800 BFCL tasks by task type (same tasks, same scorer) 9 results · 2026-09-25
2026-09-25trap success (all calls scored)104/120 (86.7%)valid
2026-09-25tokens out per call (mean)64.5 tokensvalid
2026-09-25tokens in per call (mean)158.9 tokensvalid
2026-09-25thinking tokens per call (mean, where reported)12 tokensvalid
2026-09-25latency p50 (seconds per call)2.46 sAzurevalid
2026-09-25cost per right answer$0.0013valid
2026-09-25accuracy, single call92.5%valid
2026-09-25accuracy, several tools to choose from89.0%valid
2026-09-25accuracy, parallel calls87.5%valid
  1. Sample sizes run from n=120 to n=920; 2 of 12 results carry a confidence interval or p-value.
  2. Results dated 2026-09-25 to 2026-10-10; hosts: Azure, OpenRouter.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gpt-5.2-codex, openai/gpt-5.2-codex, openrouter:openai/gpt-5.2-codex. Hosts: Azure, OpenRouter.

NVIDIA · Model

nemotron-3.5-lightning:free

2tests8results0valid

accuracy85.0%no shared fieldn=100 · Nemotron 3.5 and Muse-Glimmer run 2 on…
typical speed1.12 sno shared fieldmedian per call · Nemotron 3.5 and Muse-Glimmer run 2 on…

8 results so far, latest: success/all on Nemotron 3.5 and Muse-Glimmer run 2 on tool path.

All 8 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Nemotron 3.5 and Muse-Glimmer run 2 on tool path 7 results · 2026-08-12
2026-08-12success/answered95.5%Nvidia · standarddescriptive-only
2026-08-12success/answered97.1%Nvidia · trapsdescriptive-only
2026-08-12success/all85.0%Nvidia · standarddescriptive-only
2026-08-12success/all82.5%Nvidia · trapsdescriptive-only
2026-08-12right tool wrong argument0/40 (0.0%)Nvidia · trapsdescriptive-only
2026-08-12retry rate39.0%Nvidia · alldescriptive-only
2026-08-12latency p501.12 sNvidia · standarddescriptive-only
Listed model ids probed for tool-call capability 1 result · 2026-08-12
2026-08-12tool call emitted1/1 (100.0%)OpenRouter · single probe calldescriptive-only
  1. Sample sizes run from n=1 to n=100; 2 of 8 results carry a confidence interval or p-value.
  2. Results dated 2026-08-12 to 2026-08-12; hosts: Nvidia, OpenRouter.
  3. Also logged as: nemotron-3.5-lightning:free, nvidia/nemotron-3.5-lightning:free. Hosts: Nvidia, OpenRouter.

Anthropic · Model

opus

2tests5results2valid#5of 6 on Frontier tool-call pilot run 2: bootstrap C…

accuracy96.8%5 of 6n=220 · Frontier tool-call pilot run 2: bootstr…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #5 of 6 on Frontier tool-call pilot run 2: bootstr….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier tool-call pilot run 2: bootstrap CIs#5 of 6 · accuracy
96.8%
90%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier tool-call pilot run 2: bootstrap CIs
5
worse96.8% · 5 of 6better
All 5 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier pilot: bfcl-simple only + wrongness signals 1 result · 2026-09-24
2026-09-24tok_out ratio (wrong/right)1.46 ratio · pilot bfcl-simple, wrong vs rightdescriptive-only
Frontier tool-call pilot run 2: bootstrap CIs 4 results · 2026-09-24
2026-09-24item flip rate2.9% · run1 -> run2valid
2026-09-24accuracy (default -> high)3/10 -> 2/10 · default vs effort=high, shape S3descriptive-only
2026-09-24accuracy (default -> high)1/10 -> 0/10 · default vs effort=high, shape S12descriptive-only
2026-09-24accuracy96.8% · run 2, artifact shapes excludedvalid
  1. Sample sizes run from n=10 to n=220; 2 of 5 results carry a confidence interval or p-value.
  2. Results dated 2026-09-24 to 2026-09-24.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: opus. Hosts: not recorded.

Qwen (Alibaba) · Model

qwen3-30b

2tests12results11valid

accuracy94.5%no shared fieldn=600 · Gateway catch, Qwen family + gpt-oss
traps passed90.0%no shared fieldn=300 · Gateway catch, Qwen family + gpt-oss

12 results so far, latest: T trap success (success/all) on Gateway catch, Qwen family + gpt-oss (DeepInfra).

All 12 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Gateway catch, Qwen family + gpt-oss (DeepInfra) 11 results · 2026-10-09
2026-10-09trigger fired overall5/1500 (0.3%)DeepInfra · catch triggervalid
2026-10-09run cost (usage)$0.34DeepInfra · all armsvalid
2026-10-09T trap success (success/all)270/300 (90.0%)DeepInfravalid
2026-10-09T trap success (success/all)270/300 (90.0%)DeepInfra · SPECIFICvalid
2026-10-09T trap success (success/all)270/300 (90.0%)DeepInfra · GENERICvalid
2026-10-09P parallel success (success/all)552/600 (92.0%)DeepInfravalid
2026-10-09P parallel success (success/all)552/600 (92.0%)DeepInfra · SPECIFICvalid
2026-10-09P parallel success (success/all)552/600 (92.0%)DeepInfra · GENERICvalid
2026-10-09M BFCL-multiple success (success/all)567/600 (94.5%)DeepInfravalid
2026-10-09M BFCL-multiple success (success/all)567/600 (94.5%)DeepInfra · SPECIFICvalid
2026-10-09M BFCL-multiple success (success/all)567/600 (94.5%)DeepInfra · GENERICvalid
Provider sweep from box: parallel calls + latency 1 result · 2026-07-29
2026-07-29parallel tool calls emitted6/6 (100.0%)Nebius · provider sweepdescriptive-only
  1. Sample sizes run from n=6 to n=1500; 0 of 12 results carry a confidence interval or p-value.
  2. Results dated 2026-07-29 to 2026-10-09; hosts: DeepInfra, Nebius.
  3. Also logged as: Qwen/Qwen3-30B-A3B, qwen3-30b, qwen3-30b-a3b. Hosts: DeepInfra, Nebius.

Qwen (Alibaba) · Model

qwen3.8-max

2tests10results9valid#3of 5 on Frontier refresh BFCL v4 pinned arms

accuracy93.0%3 of 5n=800 · Frontier refresh BFCL v4 pinned arms
traps passed100.0%4 of 5n=50 · Trap set per shape
typical speed3.70 s5 of 5median per call · Frontier refresh BFCL v4 latency

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 2 shared tests; best place #3 of 5 on Frontier refresh BFCL v4 pinned arms; among the slowest (5 of 5).

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier refresh BFCL v4 pinned arms#3 of 5 · accuracy (our har…
93.0%
Trap set per shape (frontier refresh arms)#4 of 5 · trap success
100.0%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier refresh BFCL v4 pinned arms
3
worse93.0% · 3 of 5better
Speed, typical seconds per callFrontier refresh BFCL v4 latency (from Mac)
5
worse3.70 s · 5 of 5better
All 10 results5 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Cascade vs arms vs oracle on HumanEval+ / BFCL 1 result · 2026-08-16
2026-08-16HumanEval+ plus pass89.0% · single armvalid
Frontier refresh BFCL v4 pinned arms 1 result · 2026-08-06
2026-08-06accuracy (our harness)744/800 (93.0%)Alibaba · provider-pinnedvalid
Frontier refresh BFCL v4 per category 3 results · 2026-08-06
2026-08-06accuracy (our harness)379/400 (94.8%)Alibaba · simplevalid
2026-08-06accuracy (our harness)184/200 (92.0%)Alibaba · multiplevalid
2026-08-06accuracy (our harness)181/200 (90.5%)Alibaba · parallelvalid
Frontier refresh BFCL v4 latency (from Mac) 1 result · 2026-08-06
2026-08-06p50 latency3.7 sAlibaba · provider-pinneddescriptive-only
Trap set per shape (frontier refresh arms) 4 results · 2026-08-06
2026-08-06trap success50/50 (100.0%) · S1 forecastvalid
2026-08-06trap success49/50 (98.0%) · S2 destinationvalid
2026-08-06trap success50/50 (100.0%) · S3 poundsvalid
2026-08-06trap success50/50 (100.0%) · S4 no-toolvalid
  1. Sample sizes run from n=50 to n=800; 1 of 10 results carry a confidence interval or p-value.
  2. Results dated 2026-08-06 to 2026-08-16; hosts: Alibaba.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: qwen/qwen3.8-max, qwen3.8-max. Hosts: Alibaba.

Anthropic · Model

sonnet-4.6

2tests7results5valid

typical speed8.99 sno shared fieldmedian per call · Ladder M1: is Qwen3-Coder-480B alone sa…

7 results so far, latest: M1 wrong-rate after blind 2-family re-adjudication (confirmed wrong only) on Ladder M2: which second low-cost model / pair / anch….

All 7 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Ladder M2: which second low-cost model / pair / anchored trio should check Qwen3-Coder-480B on tool calls (GROK-prereg-ladder-m2) 1 result · 2026-10-14
2026-10-14M1 wrong-rate after blind 2-family re-adjudication (confirmed wrong only)5.2%AWS Bedrock us-east-1 · frontierdescriptive-only
Ladder M1: is Qwen3-Coder-480B alone safe enough that a second low-cost opinion is redundant? (PREREG-m1 + A1) 6 results · 2026-10-14
2026-10-14wrong-rate, trap stratum9.7%AWS Bedrock us-east-1 · frontiervalid
2026-10-14wrong-rate, pooled (300 two-family-checked traps + 100 unused BFCL exec), strict PREREG normalisation8.5%AWS Bedrock us-east-1 · frontiervalid
2026-10-14wrong-rate, bfcl stratum5.0%AWS Bedrock us-east-1 · frontiervalid
2026-10-14latency p508.993 sAWS Bedrock us-east-1 · frontierdescriptive-only
2026-10-14VOID rate0.0%AWS Bedrock us-east-1 · frontiervalid
2026-10-14$/item$0.0047AWS Bedrock us-east-1 · frontiervalid
  1. Sample sizes run from n=100 to n=400; 4 of 7 results carry a confidence interval or p-value.
  2. Results dated 2026-10-14 to 2026-10-14; hosts: AWS Bedrock us-east-1.
  3. Also logged as: Claude Sonnet 4.6 (us.anthropic.claude-sonnet-4-6), claude-sonnet-4-6. Hosts: AWS Bedrock us-east-1.

Tīrtha (ours) · Tīrtha system

Tīrtha Frontier

2tests9results8valid

accuracy96.4%no shared fieldn=800 · Llama 3.3 70B pinned to DeepInfra, 800…
traps passed100.0%no shared fieldn=120 · Tirtha Frontier live night: Base + Llam…
typical speed0.86 sno shared fieldmedian per call · Tirtha Frontier live night: Base + Llam…

A Tīrtha system (a gateway policy over several models): best on traps (100.0%), weakest on several tools (95.5%); ahead of Sonnet 5 on 3 of 4 task types.

Accuracy by task typeTirtha Frontier live night: Base + Llama 3.3 70B (Groq) second opinion, disagree -> Sonne… Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
97.5%
several toolsn=200
95.5%
parallel callsn=200
96.0%
trapsn=120
100.0%
85%100%
All 9 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Llama 3.3 70B pinned to DeepInfra, 800 BFCL + 120 traps (Tirtha Frontier second-opinion candidate) 1 result · 2026-10-17
2026-10-17accuracy96.4%OpenRouter -> DeepInfradirectional
Tirtha Frontier live night: Base + Llama 3.3 70B (Groq) second opinion, disagree -> Sonnet 5; 800 BFCL + 120 traps; gold-free agreement 8 results · 2026-10-17
2026-10-17trap success (all calls scored)100.0%api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
2026-10-17latency p50 (policy)0.86 sapi.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
2026-10-17escalation share (sent to Max)22.1%api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
2026-10-17cost per task (our cost)$0.0011api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
2026-10-17accuracy, single call97.5%api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
2026-10-17accuracy, several tools to choose from95.5%api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
2026-10-17accuracy, parallel calls96.0%api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
2026-10-17accuracy96.6%api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5)valid
  1. Sample sizes run from n=120 to n=920; 6 of 9 results carry a confidence interval or p-value.
  2. Results dated 2026-10-17 to 2026-10-17; hosts: OpenRouter -> DeepInfra, api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. A cost labelled 'our cost' is our own spend; it is never ranked against a vendor's price, which is what a customer pays. Tīrtha Frontier is one live night against sealed bars; a replicate is booked before any public claim.
  5. Also logged as: Tirtha Frontier (policy: Tirtha 3fca0e99 + Llama 3.3 70B Groq, disagree -> Sonnet 5), Tirtha Frontier policy, second opinion = Llama on DeepInfra (offline from same-night rows), tirtha-frontier. Hosts: OpenRouter -> DeepInfra, api.tirtha.ai + OpenRouter (Llama 3.3 70B pinned Groq, Sonnet 5).

DeepSeek · Model

DeepSeek-V4-Flash

1test9results0valid

accuracy92.0%no shared fieldn=800 · v4-direct
traps passed73.3%no shared fieldn=120 · v4-direct
typical speed4.17 sno shared fieldmedian per call · v4-direct

Best on single calls (94.2%), weakest on traps (73.3%); ahead of Sonnet 5 on 1 of 4 task types.

Accuracy by task typev4-direct (auto-logged run). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
94.2%
several toolsn=200
94.0%
parallel callsn=200
85.5%
trapsn=120
73.3%
65%100%
All 9 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
v4-direct (auto-logged run) 9 results · 2026-10-18
2026-10-18trap success (all calls scored)73.3%deepinfra-deepseek-v4-flashauto
2026-10-18tokens out per call (mean)110.8 tokensdeepinfra-deepseek-v4-flashauto
2026-10-18latency p9511.06 sdeepinfra-deepseek-v4-flashauto
2026-10-18latency p504.17 sdeepinfra-deepseek-v4-flashauto
2026-10-18cost per taskNonedeepinfra-deepseek-v4-flashauto
2026-10-18accuracy, single call94.2%deepinfra-deepseek-v4-flashauto
2026-10-18accuracy, several tools to choose from94.0%deepinfra-deepseek-v4-flashauto
2026-10-18accuracy, parallel calls85.5%deepinfra-deepseek-v4-flashauto
2026-10-18accuracy92.0%deepinfra-deepseek-v4-flashauto
  1. Sample sizes run from n=120 to n=920; 6 of 9 results carry a confidence interval or p-value.
  2. Results dated 2026-10-18 to 2026-10-18; hosts: deepinfra-deepseek-v4-flash.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: DeepSeek-V4-Flash, deepseek-ai/DeepSeek-V4-Flash. Hosts: deepinfra-deepseek-v4-flash.

Meta · Model

Meta-Llama-3.3-70B-Instruct

1test9results0valid

accuracy69.5%no shared fieldn=800 · llama-sambanova-n800-2026-10-17
traps passed72.5%no shared fieldn=120 · llama-sambanova-n800-2026-10-17
typical speed0.62 sno shared fieldmedian per call · llama-sambanova-n800-2026-10-17

Best on several tools (93.5%), weakest on parallel calls (0.0%); behind Sonnet 5 on all 4 task types.

Accuracy by task typellama-sambanova-n800-2026-10-17 (auto-logged run). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
92.2%
several toolsn=200
93.5%
parallel callsn=200
0.0%
trapsn=120
72.5%
0%100%
All 9 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
llama-sambanova-n800-2026-10-17 (auto-logged run) 9 results · 2026-10-18
2026-10-18trap success (all calls scored)72.5%llama-sambanovaauto
2026-10-18tokens out per call (mean)50.7 tokensllama-sambanovaauto
2026-10-18latency p953.26 sllama-sambanovaauto
2026-10-18latency p500.62 sllama-sambanovaauto
2026-10-18cost per taskNonellama-sambanovaauto
2026-10-18accuracy, single call92.2%llama-sambanovaauto
2026-10-18accuracy, several tools to choose from93.5%llama-sambanovaauto
2026-10-18accuracy, parallel calls0.0%llama-sambanovaauto
2026-10-18accuracy69.5%llama-sambanovaauto
  1. Sample sizes run from n=120 to n=920; 6 of 9 results carry a confidence interval or p-value.
  2. Results dated 2026-10-18 to 2026-10-18; hosts: llama-sambanova.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: Meta-Llama-3.3-70B-Instruct. Hosts: llama-sambanova.

Anthropic · Model

claude-haiku-4.5

1test3results3valid

typical speed1.20 sno shared fieldmedian per call · Muse Spark 1.3 vs value rung, tool call…
cost per right answer$0.0012no shared fieldlist price · Muse Spark 1.3 vs value rung, tool call…

3 results so far, latest: pass on Muse Spark 1.3 vs value rung (Haiku 4.5), tool ca….

All 3 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Muse Spark 1.3 vs value rung (Haiku 4.5), tool calls, n=80 3 results · 2026-10-14
2026-10-14pass80/80 (100.0%)bedrock · incumbentvalid
2026-10-14latency p501.2 sbedrock · incumbentvalid
2026-10-14cost per pass$0.0012bedrock · incumbentvalid
  1. Sample sizes run from n=80 to n=80; 0 of 3 results carry a confidence interval or p-value.
  2. Results dated 2026-10-14 to 2026-10-14; hosts: bedrock.
  3. Also logged as: claude-haiku-4.5, us.anthropic.claude-haiku-4-5-20251001-v1:0. Hosts: bedrock.

DeepSeek · Model

deepseek

1test1result1valid

1 result so far, latest: traps served without wrong tool on Reliability trap set competitor comparison.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Reliability trap set competitor comparison 1 result · 2026-07-16
2026-07-16traps served without wrong tool90.0%OpenRouter · trapsvalid
  1. Results dated 2026-07-16 to 2026-07-16; hosts: OpenRouter.
  2. Also logged as: DeepSeek, deepseek. Hosts: OpenRouter.

DeepSeek · Model

deepseek-chat

1test1result1valid#5of 8 on BFCL v4 competitor comparison

accuracy93.0%5 of 8n=800 · BFCL v4 competitor comparison

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #5 of 8 on BFCL v4 competitor comparison.

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
BFCL v4 competitor comparison#5 of 8 · accuracy (our har…
93.0%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyBFCL v4 competitor comparison
5
worse93.0% · 5 of 8better
All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
BFCL v4 competitor comparison 1 result · 2026-07-16
2026-07-16accuracy (our harness)93.0%OpenRouter · single run (Tirtha 3-seed)valid
  1. Sample sizes run from n=800 to n=800; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-16 to 2026-07-16; hosts: OpenRouter.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: DeepSeek-chat, deepseek-chat. Hosts: OpenRouter.

DeepSeek · Model

deepseek-v3.2

1test1result0valid

1 result so far, latest: pass@1 (full 539) on Value rung per task shape: code vs tools.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 1 result · 2026-07-27
2026-07-27pass@1 (full 539)77.7% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
  1. Sample sizes run from n=539 to n=539; 1 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. Also logged as: deepseek-v3.2. Hosts: not recorded.

DeepSeek · Model

deepseek-v4-pro

1test2results0valid

accuracy90.0%no shared fieldn=100 · Value-rung swap: tool-path re-check and…
traps passed52.5%no shared fieldn=40 · Value-rung swap: tool-path re-check and…

2 results so far, latest: standard success on Value-rung swap: tool-path re-check and prod late….

All 2 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value-rung swap: tool-path re-check and prod latency 2 results · 2026-07-26
2026-07-26trap accuracy21/40 (52.5%) · trapsdescriptive-only
2026-07-26standard success90/100 (90.0%) · synthetic tool benchdescriptive-only
  1. Sample sizes run from n=40 to n=100; 1 of 2 results carry a confidence interval or p-value.
  2. Results dated 2026-07-26 to 2026-07-26.
  3. Also logged as: deepseek-v4-pro. Hosts: not recorded.

DeepSeek · Model

deepseek-v4.1-flash-1017

1test9results0valid

accuracy91.6%no shared fieldn=800 · deepseek-v41-flash-1017-2026-10-18
traps passed88.3%no shared fieldn=120 · deepseek-v41-flash-1017-2026-10-18
typical speed1.43 sno shared fieldmedian per call · deepseek-v41-flash-1017-2026-10-18

Best on single calls (93.0%), weakest on traps (88.3%); behind Sonnet 5 on all 4 task types.

Accuracy by task typedeepseek-v41-flash-1017-2026-10-18 (auto-logged run). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
93.0%
several toolsn=200
91.0%
parallel callsn=200
89.5%
trapsn=120
88.3%
80%100%
All 9 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
deepseek-v41-flash-1017-2026-10-18 (auto-logged run) 9 results · 2026-10-18
2026-10-18trap success (all calls scored)88.3%DeepInfraauto
2026-10-18tokens out per call (mean)141.1 tokensDeepInfraauto
2026-10-18latency p953.89 sDeepInfraauto
2026-10-18latency p501.43 sDeepInfraauto
2026-10-18cost per task$0.00008DeepInfraauto
2026-10-18accuracy, single call93.0%DeepInfraauto
2026-10-18accuracy, several tools to choose from91.0%DeepInfraauto
2026-10-18accuracy, parallel calls89.5%DeepInfraauto
2026-10-18accuracy91.6%DeepInfraauto
  1. Sample sizes run from n=120 to n=920; 5 of 9 results carry a confidence interval or p-value.
  2. Results dated 2026-10-18 to 2026-10-18; hosts: DeepInfra.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: deepseek-v4.1-flash-1017, openrouter:deepseek/deepseek-v4.1-flash-1017. Hosts: DeepInfra.

Google · Model

gemini-3.6-flash

1test1result0valid#3of 8 on Frontier sweep BFCL (unpinned, OpenRouter)

accuracy96.5%3 of 8n=200 · Frontier sweep BFCL

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 1 shared tests; best place #3 of 8 on Frontier sweep BFCL (unpinned, OpenRout….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier sweep BFCL (unpinned, OpenRouter)#3 of 8 · BFCL accuracy
96.5%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier sweep BFCL (unpinned, OpenRouter)
3
worse96.5% · 3 of 8better
All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy96.5%OpenRouter (unpinned) · frontier sweepdescriptive-only
  1. Sample sizes run from n=200 to n=200; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-30 to 2026-07-30; hosts: OpenRouter (unpinned).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gemini-3.6-flash. Hosts: OpenRouter (unpinned).

Google · Model

gemini-3.7-flash

1test2results2valid#2of 3 on Cascade vs arms vs oracle on HumanEval+ / B…

accuracy94.6%2 of 3n=800 · Cascade vs arms vs oracle on HumanEval+…
typical speed1.88 sno shared fieldmedian per call · Cascade vs arms vs oracle on HumanEval+…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #2 of 3 on Cascade vs arms vs oracle on HumanEval+….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Cascade vs arms vs oracle on HumanEval+ / BFCL#2 of 3 · BFCL accuracy
94.6%
85%100%
All 2 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Cascade vs arms vs oracle on HumanEval+ / BFCL 2 results · 2026-08-16
2026-08-16latency p501.88 sGoogle · BFCL addendum, pinnedvalid
2026-08-16BFCL accuracy94.6%Google · BFCL addendum, pinnedvalid
  1. Sample sizes run from n=800 to n=800; 0 of 2 results carry a confidence interval or p-value.
  2. Results dated 2026-08-16 to 2026-08-16; hosts: Google.
  3. Also logged as: gemini-3.7-flash, google/gemini-3.7-flash. Hosts: Google.

Google · Model

gemma-4-26B-A4B-it

1test9results0valid

accuracy91.0%no shared fieldn=800 · gemma-4-26b-a4b-deepinfra-2026-10-18
traps passed94.2%no shared fieldn=120 · gemma-4-26b-a4b-deepinfra-2026-10-18
typical speed0.96 sno shared fieldmedian per call · gemma-4-26b-a4b-deepinfra-2026-10-18

Best on single calls (95.8%), weakest on parallel calls (80.5%); ahead of Sonnet 5 on 1 of 4 task types.

Accuracy by task typegemma-4-26b-a4b-deepinfra-2026-10-18 (auto-logged run). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
95.8%
several toolsn=200
92.0%
parallel callsn=200
80.5%
trapsn=120
94.2%
75%100%
All 9 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
gemma-4-26b-a4b-deepinfra-2026-10-18 (auto-logged run) 9 results · 2026-10-18
2026-10-18trap success (all calls scored)94.2%deepinfra-gemma4-26bauto
2026-10-18tokens out per call (mean)41.5 tokensdeepinfra-gemma4-26bauto
2026-10-18latency p952.73 sdeepinfra-gemma4-26bauto
2026-10-18latency p500.96 sdeepinfra-gemma4-26bauto
2026-10-18cost per taskNonedeepinfra-gemma4-26bauto
2026-10-18accuracy, single call95.8%deepinfra-gemma4-26bauto
2026-10-18accuracy, several tools to choose from92.0%deepinfra-gemma4-26bauto
2026-10-18accuracy, parallel calls80.5%deepinfra-gemma4-26bauto
2026-10-18accuracy91.0%deepinfra-gemma4-26bauto
  1. Sample sizes run from n=120 to n=920; 6 of 9 results carry a confidence interval or p-value.
  2. Results dated 2026-10-18 to 2026-10-18; hosts: deepinfra-gemma4-26b.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gemma-4-26B-A4B-it, google/gemma-4-26B-A4B-it. Hosts: deepinfra-gemma4-26b.

Z.ai (GLM) · Model

glm

1test1result1valid

traps passed90.0%no shared fieldn=40 · openrouter/auto vs our low-cost tier, pair…

1 result so far, latest: trap accuracy on openrouter/auto vs our low-cost tier, paired BFCL.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
openrouter/auto vs our low-cost tier, paired BFCL 1 result · 2026-07-27
2026-07-27trap accuracy90.0% · trap setvalid
  1. Sample sizes run from n=40 to n=40; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. Also logged as: GLM, glm. Hosts: not recorded.

OpenAI · Model

gpt-5.1-codex-mini

1test6results0valid#2of 3 on Value rung per task shape: code vs tools

accuracy95.5%2 of 3n=200 · Value rung per task shape: code vs tools
traps passed82.5%4 of 5n=40 · Value rung per task shape: code vs tools
typical speed8.40 sno shared fieldmedian per call · Value rung per task shape: code vs tools
cost per right answer$0.0026no shared fieldlist price · Value rung per task shape: code vs tools

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 2 shared tests; best place #2 of 3 on Value rung per task shape: code vs tools.

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Value rung per task shape: code vs tools#4 of 5 · trap accuracy
82.5%
Value rung per task shape: code vs tools#2 of 3 · tool accuracy
95.5%
75%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyValue rung per task shape: code vs tools
4
worse82.5% · 4 of 5better
All 6 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 6 results · 2026-07-27
2026-07-27trap accuracy82.5% · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27tool accuracy95.5% · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27latency p502.79 s · tool candidate sweep (paired n=200 + 40 traps)descriptive-only
2026-07-27latency p508.4 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.0026 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@185.4% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
  1. Sample sizes run from n=40 to n=200; 1 of 6 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gpt-5.1-codex-mini. Hosts: not recorded.

OpenAI · Model

gpt-5.6-terra

1test1result0valid#6of 8 on Frontier sweep BFCL (unpinned, OpenRouter)

accuracy88.5%6 of 8n=200 · Frontier sweep BFCL

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #6 of 8 on Frontier sweep BFCL (unpinned, OpenRout….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier sweep BFCL (unpinned, OpenRouter)#6 of 8 · BFCL accuracy
88.5%
80%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier sweep BFCL (unpinned, OpenRouter)
6
worse88.5% · 6 of 8better
All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy88.5%OpenRouter (unpinned) · frontier sweepdescriptive-only
  1. Sample sizes run from n=200 to n=200; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-30 to 2026-07-30; hosts: OpenRouter (unpinned).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gpt-5.6-terra. Hosts: OpenRouter (unpinned).

OpenAI · Model

gpt-6-luna

1test1result1valid

accuracy84.5%no shared fieldn=800 · GPT-6 Luna/Sol on BFCL v4

1 result so far, latest: accuracy (our harness) on GPT-6 Luna/Sol on BFCL v4.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
GPT-6 Luna/Sol on BFCL v4 1 result · 2026-09-27
2026-09-27accuracy (our harness)676/800 (84.5%)OpenAI (pinned) · strictvalid
  1. Sample sizes run from n=800 to n=800; 1 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-09-27 to 2026-09-27; hosts: OpenAI (pinned).
  3. Also logged as: gpt-6-luna. Hosts: OpenAI (pinned).

OpenAI · Model

gpt-6-sol

1test1result1valid

accuracy85.5%no shared fieldn=800 · GPT-6 Luna/Sol on BFCL v4

1 result so far, latest: accuracy (our harness) on GPT-6 Luna/Sol on BFCL v4.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
GPT-6 Luna/Sol on BFCL v4 1 result · 2026-09-27
2026-09-27accuracy (our harness)684/800 (85.5%)OpenAI (pinned) · strictvalid
  1. Sample sizes run from n=800 to n=800; 1 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-09-27 to 2026-09-27; hosts: OpenAI (pinned).
  3. Also logged as: gpt-6-sol. Hosts: OpenAI (pinned).

OpenAI · Model

gpt-oss-120b-Turbo

1test9results0valid

accuracy66.8%no shared fieldn=800 · gpt-oss-turbo-vs-plain-deepinfra-2026-1…
traps passed83.3%no shared fieldn=120 · gpt-oss-turbo-vs-plain-deepinfra-2026-1…
typical speed0.62 sno shared fieldmedian per call · gpt-oss-turbo-vs-plain-deepinfra-2026-1…

Best on single calls (90.2%), weakest on parallel calls (0.0%); behind Sonnet 5 on all 4 task types.

Accuracy by task typegpt-oss-turbo-vs-plain-deepinfra-2026-10-18 (auto-logged run). Solid tick: Sonnet 5 on the 800-task run; dashed tick: the median of that field.
single callsn=400
90.2%
several toolsn=200
86.5%
parallel callsn=200
0.0%
trapsn=120
83.3%
0%100%
All 9 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
gpt-oss-turbo-vs-plain-deepinfra-2026-10-18 (auto-logged run) 9 results · 2026-10-18
2026-10-18trap success (all calls scored)83.3%deepinfra-turboauto
2026-10-18tokens out per call (mean)112.7 tokensdeepinfra-turboauto
2026-10-18latency p951.25 sdeepinfra-turboauto
2026-10-18latency p500.62 sdeepinfra-turboauto
2026-10-18cost per taskNonedeepinfra-turboauto
2026-10-18accuracy, single call90.2%deepinfra-turboauto
2026-10-18accuracy, several tools to choose from86.5%deepinfra-turboauto
2026-10-18accuracy, parallel calls0.0%deepinfra-turboauto
2026-10-18accuracy66.8%deepinfra-turboauto
  1. Sample sizes run from n=120 to n=920; 6 of 9 results carry a confidence interval or p-value.
  2. Results dated 2026-10-18 to 2026-10-18; hosts: deepinfra-turbo.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: gpt-oss-120b-Turbo, openai/gpt-oss-120b-Turbo. Hosts: deepinfra-turbo.

xAI · Model

grok-4.3

1test2results2valid#3of 3 on Cascade vs arms vs oracle on HumanEval+ / B…

accuracy93.4%3 of 3n=800 · Cascade vs arms vs oracle on HumanEval+…
typical speed2.29 sno shared fieldmedian per call · Cascade vs arms vs oracle on HumanEval+…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #3 of 3 on Cascade vs arms vs oracle on HumanEval+….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Cascade vs arms vs oracle on HumanEval+ / BFCL#3 of 3 · BFCL accuracy
93.4%
85%100%
All 2 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Cascade vs arms vs oracle on HumanEval+ / BFCL 2 results · 2026-08-16
2026-08-16latency p502.29 sxAI · BFCL addendum, pinnedvalid
2026-08-16BFCL accuracy93.4%xAI · BFCL addendum, pinnedvalid
  1. Sample sizes run from n=800 to n=800; 0 of 2 results carry a confidence interval or p-value.
  2. Results dated 2026-08-16 to 2026-08-16; hosts: xAI.
  3. Also logged as: grok-4.3, x-ai/grok-4.3. Hosts: xAI.

Maker not recorded · Model

inkling

1test1result0valid#4of 8 on Frontier sweep BFCL (unpinned, OpenRouter)

accuracy95.0%4 of 8n=200 · Frontier sweep BFCL

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 1 shared tests; best place #4 of 8 on Frontier sweep BFCL (unpinned, OpenRout….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier sweep BFCL (unpinned, OpenRouter)#4 of 8 · BFCL accuracy
95.0%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier sweep BFCL (unpinned, OpenRouter)
4
worse95.0% · 4 of 8better
All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy95.0%OpenRouter (unpinned) · frontier sweepdescriptive-only
  1. Sample sizes run from n=200 to n=200; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-30 to 2026-07-30; hosts: OpenRouter (unpinned).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: inkling. Hosts: OpenRouter (unpinned).

Kwaipilot (Kuaishou) · Model

kat-coder-air-v2.5

1test1result0valid

accuracy100.0%no shared fieldn=24 · Value-rung swap: tool-path re-check and…

1 result so far, latest: score on Value-rung swap: tool-path re-check and prod late….

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value-rung swap: tool-path re-check and prod latency 1 result · 2026-07-26
2026-07-26score24/24 (100.0%) · 12 logic puzzlesdescriptive-only
  1. Sample sizes run from n=24 to n=24; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-26 to 2026-07-26.
  3. Also logged as: kat-coder-air-v2.5. Hosts: not recorded.

Kwaipilot (Kuaishou) · Model

kat-coder-pro-v2

1test3results0valid

typical speed2.90 sno shared fieldmedian per call · Value rung per task shape: code vs tools
cost per right answer$0.00025no shared fieldlist price · Value rung per task shape: code vs tools

3 results so far, latest: HumanEval+ pass@1 on Value rung per task shape: code vs tools.

All 3 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 3 results · 2026-07-27
2026-07-27latency p502.9 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.00025 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@184.8% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
  1. Sample sizes run from n=164 to n=164; 1 of 3 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. Also logged as: kat-coder-pro-v2. Hosts: not recorded.

Moonshot (Kimi) · Model

kimi-k2-thinking

1test3results0valid

typical speed45.00 sno shared fieldmedian per call · Value rung per task shape: code vs tools
cost per right answer$0.0064no shared fieldlist price · Value rung per task shape: code vs tools

3 results so far, latest: HumanEval+ pass@1 on Value rung per task shape: code vs tools.

All 3 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 3 results · 2026-07-27
2026-07-27latency p5045 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.0064 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@190.2% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
  1. Sample sizes run from n=164 to n=164; 1 of 3 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. Also logged as: kimi-k2-thinking. Hosts: not recorded.

Moonshot (Kimi) · Model

kimi-k3

1test1result0valid#8of 8 on Frontier sweep BFCL (unpinned, OpenRouter)

accuracy71.0%8 of 8n=200 · Frontier sweep BFCL

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #8 of 8 on Frontier sweep BFCL (unpinned, OpenRout….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier sweep BFCL (unpinned, OpenRouter)#8 of 8 · BFCL accuracy
71.0%
65%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier sweep BFCL (unpinned, OpenRouter)
8
worse71.0% · 8 of 8better
All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy71.0%OpenRouter (unpinned) · frontier sweepdescriptive-only
  1. Sample sizes run from n=200 to n=200; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-30 to 2026-07-30; hosts: OpenRouter (unpinned).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: kimi-k3. Hosts: OpenRouter (unpinned).

Meta · Model

llama-4-scout-17b-16e

1test3results0valid#5of 5 on Mistake atlas: first-call tool accuracy acr…

accuracy70.0%5 of 5Mistake atlas: first-call tool accuracy…

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #5 of 5 on Mistake atlas: first-call tool accuracy….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Mistake atlas: first-call tool accuracy across families#5 of 5 · success rate (ans…
70.0%
65%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyMistake atlas: first-call tool accuracy across families
5
worse70.0% · 5 of 5better
All 3 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Mistake atlas: first-call tool accuracy across families 3 results · 2026-10-09
2026-10-09success rate (answered denominator)70.0%Together · ATLAS_ONLY T (traps)descriptive-only
2026-10-09success rate (answered denominator)94.5%Together · ATLAS_ONLY M (bfcl-multiple)descriptive-only
2026-10-09success rate (answered denominator)69.5%Together · ATLAS_ONLY P (bfcl-parallel)descriptive-only
  1. Results dated 2026-10-09 to 2026-10-09; hosts: Together.
  2. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  3. Also logged as: Llama-4-Scout-17B-16E-Instruct, llama-4-scout-17b-16e. Hosts: Together.

MiniMax · Model

minimax-m2.5

1test3results0valid

typical speed12.90 sno shared fieldmedian per call · Value rung per task shape: code vs tools
cost per right answer$0.0015no shared fieldlist price · Value rung per task shape: code vs tools

3 results so far, latest: HumanEval+ pass@1 on Value rung per task shape: code vs tools.

All 3 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 3 results · 2026-07-27
2026-07-27latency p5012.9 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.0015 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@187.8% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
  1. Sample sizes run from n=164 to n=164; 1 of 3 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. Also logged as: minimax-m2.5. Hosts: not recorded.

Mistral · Model

mistral-7b

1test1result0valid

1 result so far, latest: calls completed on mistral-7b drafting fresh BFCL calls.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
mistral-7b drafting fresh BFCL calls 1 result · 2026-10-11
2026-10-11calls completed0 countOpenRouter · drafterVOID
  1. Results dated 2026-10-11 to 2026-10-11; hosts: OpenRouter.
  2. 1 VOID result kept and marked: those runs broke their own rules and are not scores.
  3. Also logged as: mistral-7b, mistral-7b-instruct. Hosts: OpenRouter.

NVIDIA · Model

nemotron-3-nano-30b

1test5results0valid

5 results so far, latest: effective output price multiple of sticker on Nemotron nano reasoning tokens vs sticker price.

All 5 results2 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Nemotron nano reasoning tokens vs sticker price 1 result · 2026-07-19
2026-07-19effective output price multiple of sticker14.7 xOpenRouter · single tool calldescriptive-only
Nemotron-3-nano reasoning-token overhead 4 results · 2026-07-19
2026-07-19tool call emitted0/1 (0.0%)OpenRouter · max_tokens=100descriptive-only
2026-07-19reasoning tokens of completion123/1 (12300.0%)OpenRouter · one tool calldescriptive-only
2026-07-19effective output price per 1M$2.93OpenRouter · one tool calldescriptive-only
2026-07-19cost of call$0.00004OpenRouter · one tool calldescriptive-only
  1. Sample sizes run from n=1 to n=1; 0 of 5 results carry a confidence interval or p-value.
  2. Results dated 2026-07-19 to 2026-07-19; hosts: OpenRouter.
  3. Also logged as: nemotron-3-nano-30b, nvidia/nemotron-3-nano-30b-a3b. Hosts: OpenRouter.

NVIDIA · Model

nemotron-3.5-lightning

1test1result0valid

1 result so far, latest: tool call emitted on Listed model ids probed for tool-call capability.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Listed model ids probed for tool-call capability 1 result · 2026-08-12
2026-08-12tool call emitted0/1 (0.0%)OpenRouter · single probe calldescriptive-only
  1. Sample sizes run from n=1 to n=1; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-08-12 to 2026-08-12; hosts: OpenRouter.
  3. Also logged as: nemotron-3.5-lightning, nvidia/nemotron-3.5-lightning. Hosts: OpenRouter.

Anthropic · Model

opus-5

1test1result0valid#2of 8 on Frontier sweep BFCL (unpinned, OpenRouter)

accuracy96.5%2 of 8n=200 · Frontier sweep BFCL

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 1 of 1 shared tests; best place #2 of 8 on Frontier sweep BFCL (unpinned, OpenRout….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier sweep BFCL (unpinned, OpenRouter)#2 of 8 · BFCL accuracy
96.5%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier sweep BFCL (unpinned, OpenRouter)
2
worse96.5% · 2 of 8better
All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy96.5%OpenRouter (unpinned) · frontier sweepdescriptive-only
  1. Sample sizes run from n=200 to n=200; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-30 to 2026-07-30; hosts: OpenRouter (unpinned).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: opus-5. Hosts: OpenRouter (unpinned).

Pair of makers · Model

pair-qwen3coder-gptoss120b

1test1result0valid

1 result so far, latest: ship gate (BFCL stratum, wrong-served <=0.5% and Wilson UCL <=1%) on Ladder M2-ext: ship GATE for pair Qwen3-Coder + g….

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Ladder M2-ext: ship GATE for pair Qwen3-Coder + gpt-oss-120b (GROK-prereg-ladder-m2-ext) 1 result · 2026-10-14
2026-10-14ship gate (BFCL stratum, wrong-served <=0.5% and Wilson UCL <=1%)NoneDeepInfra direct · pair:gptoss120bstopped-preflight
  1. Sample sizes run from n=0 to n=0; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-10-14 to 2026-10-14; hosts: DeepInfra direct.
  3. Also logged as: Qwen3-Coder-480B + gpt-oss-120b-Turbo (pair), pair-qwen3coder-gptoss120b. Hosts: DeepInfra direct.

Qwen (Alibaba) · Model

qwen-2.5-7b

1test2results0valid

accuracy76.2%no shared fieldn=898 · qwen-2.5-7b drafting fresh BFCL calls

2 results so far, latest: harness success on qwen-2.5-7b drafting fresh BFCL calls.

All 2 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
qwen-2.5-7b drafting fresh BFCL calls 2 results · 2026-10-11
2026-10-11harness success684/898 (76.2%)OpenRouter (Phala) · drafter (data, 898 tasks)descriptive-only
2026-10-11harness success491/653 (75.2%)OpenRouter (Phala) · drafter (data2, 653 tasks)descriptive-only
  1. Sample sizes run from n=653 to n=898; 0 of 2 results carry a confidence interval or p-value.
  2. Results dated 2026-10-11 to 2026-10-11; hosts: OpenRouter (Phala).
  3. Also logged as: qwen-2.5-7b, qwen-2.5-7b-instruct. Hosts: OpenRouter (Phala).

Qwen (Alibaba) · Model

qwen3-32b

1test11results0valid

11 results so far, latest: T trap success (success/all) on Gateway catch, Qwen family + gpt-oss (DeepInfra).

All 11 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Gateway catch, Qwen family + gpt-oss (DeepInfra) 11 results · 2026-10-09
2026-10-09trigger fired overall4/1500 (0.3%)DeepInfra · catch triggerVOID
2026-10-09run cost (usage)NoneDeepInfra · all armsVOID
2026-10-09T trap success (success/all)295/300 (98.3%)DeepInfraVOID
2026-10-09T trap success (success/all)295/300 (98.3%)DeepInfra · SPECIFICVOID
2026-10-09T trap success (success/all)295/300 (98.3%)DeepInfra · GENERICVOID
2026-10-09P parallel success (success/all)550/600 (91.7%)DeepInfraVOID
2026-10-09P parallel success (success/all)550/600 (91.7%)DeepInfra · SPECIFICVOID
2026-10-09P parallel success (success/all)550/600 (91.7%)DeepInfra · GENERICVOID
2026-10-09M BFCL-multiple success (success/all)571/600 (95.2%)DeepInfraVOID
2026-10-09M BFCL-multiple success (success/all)571/600 (95.2%)DeepInfra · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)571/600 (95.2%)DeepInfra · GENERICVOID
  1. Sample sizes run from n=300 to n=1500; 0 of 11 results carry a confidence interval or p-value.
  2. Results dated 2026-10-09 to 2026-10-09; hosts: DeepInfra.
  3. 11 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Also logged as: Qwen/Qwen3-32B, qwen3-32b. Hosts: DeepInfra.

Qwen (Alibaba) · Model

qwen3-coder-next

1test3results0valid

typical speed1.70 sno shared fieldmedian per call · Value rung per task shape: code vs tools
cost per right answer$0.00014no shared fieldlist price · Value rung per task shape: code vs tools

3 results so far, latest: HumanEval+ pass@1 on Value rung per task shape: code vs tools.

All 3 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 3 results · 2026-07-27
2026-07-27latency p501.7 s · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27cost per solved$0.00014 · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
2026-07-27HumanEval+ pass@176.8% · code bake-off HumanEval+ (paired vs low-cost 87.2%)descriptive-only
  1. Sample sizes run from n=164 to n=164; 1 of 3 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. Also logged as: qwen3-coder-next. Hosts: not recorded.

Qwen (Alibaba) · Model

qwen3-max

1test1result0valid

1 result so far, latest: HumanEval+ pass@1 on Value rung per task shape: code vs tools.

All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Value rung per task shape: code vs tools 1 result · 2026-07-27
2026-07-27HumanEval+ pass@167.1% · code bake-off HumanEval+ (paired vs low-cost 87.2%)VOID
  1. Sample sizes run from n=164 to n=164; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-27 to 2026-07-27.
  3. 1 VOID result kept and marked: those runs broke their own rules and are not scores.
  4. Also logged as: qwen3-max. Hosts: not recorded.

Qwen (Alibaba) · Model

qwen3.5-397b

1test11results0valid

11 results so far, latest: T trap success (success/all) on Gateway catch, Qwen family + gpt-oss (DeepInfra).

All 11 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Gateway catch, Qwen family + gpt-oss (DeepInfra) 11 results · 2026-10-09
2026-10-09trigger fired overall3/1500 (0.2%)DeepInfra · catch triggerVOID
2026-10-09run cost (usage)$1.42DeepInfra · all armsVOID
2026-10-09T trap success (success/all)292/300 (97.3%)DeepInfraVOID
2026-10-09T trap success (success/all)292/300 (97.3%)DeepInfra · SPECIFICVOID
2026-10-09T trap success (success/all)292/300 (97.3%)DeepInfra · GENERICVOID
2026-10-09P parallel success (success/all)567/600 (94.5%)DeepInfraVOID
2026-10-09P parallel success (success/all)567/600 (94.5%)DeepInfra · SPECIFICVOID
2026-10-09P parallel success (success/all)567/600 (94.5%)DeepInfra · GENERICVOID
2026-10-09M BFCL-multiple success (success/all)561/600 (93.5%)DeepInfraVOID
2026-10-09M BFCL-multiple success (success/all)561/600 (93.5%)DeepInfra · SPECIFICVOID
2026-10-09M BFCL-multiple success (success/all)561/600 (93.5%)DeepInfra · GENERICVOID
  1. Sample sizes run from n=300 to n=1500; 0 of 11 results carry a confidence interval or p-value.
  2. Results dated 2026-10-09 to 2026-10-09; hosts: DeepInfra.
  3. 11 VOID results kept and marked: those runs broke their own rules and are not scores.
  4. Also logged as: Qwen/Qwen3.5-397B-A17B, qwen3.5-397b. Hosts: DeepInfra.

Qwen (Alibaba) · Model

qwen3.7-flash

1test9results8valid#2of 5 on Trap set per shape (frontier refresh arms)

accuracy92.6%4 of 5n=800 · Frontier refresh BFCL v4 pinned arms
traps passed100.0%2 of 5n=50 · Trap set per shape
typical speed3.69 s4 of 5median per call · Frontier refresh BFCL v4 latency

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 2 shared tests; best place #2 of 5 on Trap set per shape (frontier refresh ar….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier refresh BFCL v4 pinned arms#4 of 5 · accuracy (our har…
92.6%
Trap set per shape (frontier refresh arms)#2 of 5 · trap success
100.0%
85%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier refresh BFCL v4 pinned arms
4
worse92.6% · 4 of 5better
Speed, typical seconds per callFrontier refresh BFCL v4 latency (from Mac)
4
worse3.69 s · 4 of 5better
All 9 results4 tests, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier refresh BFCL v4 pinned arms 1 result · 2026-08-06
2026-08-06accuracy (our harness)741/800 (92.6%)Alibaba · provider-pinnedvalid
Frontier refresh BFCL v4 per category 3 results · 2026-08-06
2026-08-06accuracy (our harness)374/400 (93.5%)Alibaba · simplevalid
2026-08-06accuracy (our harness)187/200 (93.5%)Alibaba · multiplevalid
2026-08-06accuracy (our harness)180/200 (90.0%)Alibaba · parallelvalid
Frontier refresh BFCL v4 latency (from Mac) 1 result · 2026-08-06
2026-08-06p50 latency3.69 sAlibaba · provider-pinneddescriptive-only
Trap set per shape (frontier refresh arms) 4 results · 2026-08-06
2026-08-06trap success50/50 (100.0%) · S1 forecastvalid
2026-08-06trap success50/50 (100.0%) · S2 destinationvalid
2026-08-06trap success50/50 (100.0%) · S3 poundsvalid
2026-08-06trap success50/50 (100.0%) · S4 no-toolvalid
  1. Sample sizes run from n=50 to n=800; 1 of 9 results carry a confidence interval or p-value.
  2. Results dated 2026-08-06 to 2026-08-06; hosts: Alibaba.
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: qwen/qwen3.7-flash, qwen3.7-flash. Hosts: Alibaba.

Maker not recorded · Model

sol

1test1result0valid#7of 8 on Frontier sweep BFCL (unpinned, OpenRouter)

accuracy86.0%7 of 8n=200 · Frontier sweep BFCL

Each bar is the field that ran the same test, worse on the left, better on the right. Tick: the field median. Dot: this model.

Above the field median on 0 of 1 shared tests; best place #7 of 8 on Frontier sweep BFCL (unpinned, OpenRout….

Against the field, test by testIts score on each test other models also ran (largest fields first). Dashed tick: the median of that test.
Frontier sweep BFCL (unpinned, OpenRouter)#7 of 8 · BFCL accuracy
86.0%
80%100%
Where it sits in the fieldEach strip is one test: every model that ran it is a dot, this one is outlined, its rank is in the circle. Right is better.
AccuracyFrontier sweep BFCL (unpinned, OpenRouter)
7
worse86.0% · 7 of 8better
All 1 result1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Frontier sweep BFCL (unpinned, OpenRouter) 1 result · 2026-07-30
2026-07-30BFCL accuracy86.0%OpenRouter (unpinned) · frontier sweepdescriptive-only
  1. Sample sizes run from n=200 to n=200; 0 of 1 results carry a confidence interval or p-value.
  2. Results dated 2026-07-30 to 2026-07-30; hosts: OpenRouter (unpinned).
  3. Ranks and ticks only compare models that ran the same test on the same metric. Sonnet 5 ticks come from the 800-task run (2026-09-25).
  4. Also logged as: sol. Hosts: OpenRouter (unpinned).

Anthropic · Model

us.anthropic.claude-sonnet-4-6

1test7results7valid

accuracy93.8%no shared fieldn=400 · Sonnet 4.5 vs 4.6 frontier-rung head-to…
traps passed85.0%no shared fieldn=120 · Sonnet 4.5 vs 4.6 frontier-rung head-to…
typical speed6.45 sno shared fieldmedian per call · Sonnet 4.5 vs 4.6 frontier-rung head-to…

7 results so far, latest: standard set success on Sonnet 4.5 vs 4.6 frontier-rung head-to-head.

All 7 results1 test, grouped, newest first · VOID runs kept and marked
datemetricresulthost / armstatus
Sonnet 4.5 vs 4.6 frontier-rung head-to-head 7 results · 2026-10-09
2026-10-09wrong tool chosen on traps7/120 (5.8%)AWS Bedrock · sonnet46valid
2026-10-09trap shape S2 success0/10 (0.0%)AWS Bedrock · sonnet46valid
2026-10-09trap set success102/120 (85.0%)AWS Bedrock · sonnet46valid
2026-10-09standard set success375/400 (93.8%)AWS Bedrock · sonnet46valid
2026-10-09standard arg_wrong rate13/400 (3.2%)AWS Bedrock · sonnet46valid
2026-10-09latency p506.45 sAWS Bedrock · sonnet46valid
2026-10-09$/task mean$0.0038AWS Bedrock · sonnet46valid
  1. Sample sizes run from n=10 to n=400; 3 of 7 results carry a confidence interval or p-value.
  2. Results dated 2026-10-09 to 2026-10-09; hosts: AWS Bedrock.
  3. Also logged as: us.anthropic.claude-sonnet-4-6, us.anthropic.claude-sonnet-4-6 (Sonnet 4.6). Hosts: AWS Bedrock.

Source: Tirtha results ledger (ALL-RESULTS-2026-10-14.json). Best rank is the model's best place on any shared-test card of the leaderboard. Names merged where one model was logged under several labels.