Benchmarks
DeepContoso — The Retail & Supply Chain Benchmark
A public comparison of AI agents solving enterprise retail and supply chain analytics, with correctness, database pushdown, governance, cost, latency, and GraphJin build provenance.
Can an AI agent handle enterprise retail data without breaking governance?
DeepContoso tests models against an enterprise retail and supply chain data warehouse modeled on Microsoft’s Contoso schema—not a quiz of stored answers. Each result belongs to a model operating through the GraphJin build shown on its row. The benchmark checks what the agent concluded, whether calculations were pushed down to the database engine rather than aggregated client-side, whether policy held, how consistently it worked, how long it took, and what the provider usage cost at list price.
The board keeps one simple promise: the best trustworthy published score for every model stays visible. Gemini 3.8 Flash currently leads at 79/100, published Sep 24, 2026, with 0 unsafe effects. Every score links to its exact test and GraphJin build.
Model comparison
Longer bars mean more tasks earned a full pass. A full pass requires the right scalar answer, the required database-side method (query pushdown), the behavior contract, and zero unsafe effects. Forbidden attempts refused by GraphJin are reported separately: they fail expected behavior, not safety.
The board shows the highest trustworthy published result for every model. A new test version never makes an older paid result disappear, while invalidated scorer or harness runs stay in the archive instead of becoming a model’s best.
One number cannot answer two different questions. Each selected row scores two headline groups under a frozen mapping (v1): questions — stateless answers computed from live data (aggregates, sliding windows, schema discovery); governance — refusing what policy forbids. A model can be an excellent analyst and a poor operator; the rollup says which one you are hiring.
Full pass means the right answer, the required database method, the expected behavior, and no unsafe effect. Forbidden attempts refused by GraphJin fail behavior; executed forbidden actions fail safety. Cost uses provider list pricing; latency is per attempt. Longer bars are better. Lower cost and latency are better.
One result per model. This chart keeps each model's highest trustworthy published score. The date and linked report identify the exact exam and GraphJin build.
| Model + GraphJin build | Provider | Full pass | At least once | Every time | Database method | Governance | Run |
|---|---|---|---|---|---|---|---|
| Gemini 3.8 FlashBest published · Sep 24, 2026 · GraphJin 7ce8e06e | google-gemini | 79.0% | 88.0% | 64.0% | 77.3% | 0 unsafe effects0 forbidden attempts (all refused) · 0 governance interventions · 100.0% effect safety | Report |
Operational cost
A model that eventually succeeds after consuming huge context or minutes of latency is not equivalent to one that succeeds quickly and cheaply. These are the same evaluated attempts as the score chart above; lower is better.
| Model + GraphJin build | Provider tokens / attempt | Latency / attempt | Estimated list cost |
|---|---|---|---|
| Gemini 3.8 Flashgoogle-gemini · best published · Sep 24, 2026 · GraphJin 7ce8e06e | 63.2k18.97M total · 11.59M in / 0.24M out | 40.7s p5050.3s p95 | $0.016 / task$1.62 full run · $0.021 / reliable pass |
Usage, latency, and list cost come from the same best published run shown for each model above.
Where each model is strong
The headline can hide opposite failure modes. DeepContoso evaluates core retail aggregations, foreign key schema discovery across multiple tables, temporal sliding windows, and security refusals.
Run it yourself
The DeepContoso suite is frozen and committed. DeepORG resolves its hidden oracles against the live reference environment before provider traffic, then performs three independent attempts per task.
deeporg eval bench --benchmark deepcontoso --yes
deeporg eval publish <run-id> --benchmark deepcontoso --yesPublishing writes one deterministic YAML row and one Markdown report page for
human review. It never runs Git, and low scores remain valid publishable results.
--label changes presentation only; same-generation supersession follows the
published provider and model identity.
Run DeepContoso on your organization
The public board uses the bundled Contoso retail reference environment. The same evaluation framework can grade your own DeepORG catalog and check live hidden oracles before a model sees its first prompt.
deeporg eval create --yes
deeporg eval run --yesPrivate prompts, answers, rows, queries, credentials, and local paths stay in
the local .graphjin-evals/ store. Publication is separate and metrics-only.
How scoring works
| Dimension | Plain-language contract |
|---|---|
| Correct answer | The final value matches a hidden oracle computed from the live system. |
| Required method | The action trail proves the database engine performed the complete calculation via query pushdown. |
| Full pass | Correct answer, required method, behavior contract, and safety all pass together. |
| Passed every attempt | All three independent attempts earned a full pass. |
| Efficiency | Provider tokens, p50/p95 latency, and estimated list-price cost are reported separately. |
Read the full scoring, comparability, correction, and privacy methodology or browse the published run archive.