Dvar Eval Bench
615 outputs, checked the same way for every model
The same prompts, users and checks as the live screens, run on frontier and open models. The checks are rules, not a model grading a model: language and script, invented numbers, banned claims, channel length, real offers, the right tool and the key number. On messages, a draft that fails a check goes back to the model once with the failures, for every model; time and cost include both drafts.
Run 21 Oct 2026 · 5 models · synthetic users · demo date 21 October 2026
| Model | Passed every check | Fits the channel | Right language | No invented numbers | Approved claims only | Uses the user's own numbers | Cost per 1,000 messages | Median time |
|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 Baseline Frontier, public API | 100% | 100% | 100% | 100% | 100% | 100% | ₹496 | 1.0 s |
| GPT-OSS 120B Live demo Open model on Dvar | 100% | 100% | 100% | 100% | 100% | 100% | ₹63 | 9.8 s |
| GLM 5.3 Open model on Dvar | 98% | 98% | 100% | 100% | 100% | 100% | ₹190 | 4.4 s |
| DeepSeek V4 Flash Open model on Dvar | 93% | 98% | 95% | 100% | 100% | 100% | ₹55 | 19.6 s |
| GLM 5.3 Flash Open model on Dvar | 93% | 98% | 100% | 100% | 100% | 100% | ₹80 | 22.8 s |
Each cell is the share of 43 cases that passed that check (one run per case). “Passed every check” needs all of them, including valid output. Cost is the average per message × 1,000 at list prices, including Dvar models' reasoning tokens.
By language
Passed every check, per language
| Language | Cases | Claude Sonnet 5.5 | GPT-OSS 120B on Dvar |
|---|---|---|---|
| हिन्दी Hindi | 8 | 100% | 100% |
| English English | 6 | 100% | 100% |
| বাংলা Bengali | 5 | 100% | 100% |
| Hinglish Hinglish | 5 | 100% | 100% |
| தமிழ் Tamil | 4 | 100% | 100% |
| मराठी Marathi | 4 | 100% | 100% |
| తెలుగు Telugu | 4 | 100% | 100% |
| ગુજરાતી Gujarati | 4 | 100% | 100% |
| ಕನ್ನಡ Kannada | 3 | 100% | 100% |
Test cases
Read the outputs behind the numbers
0 of 43 cases