DeepContoso — The Retail & Supply Chain Benchmark A public comparison of AI agents solving enterprise retail and supply chain analytics, with correctness, database pushdown, governance, cost, latency, and GraphJin build provenance. benchmarks benchmarks benchmarks/deepcontoso benchmarks/deepcontoso/_index.md

Benchmarks

DeepContoso — The Retail & Supply Chain Benchmark

A public comparison of AI agents solving enterprise retail and supply chain analytics, with correctness, database pushdown, governance, cost, latency, and GraphJin build provenance.

DeepContosoThe Retail & Supply Chain Benchmark · by DeepORG

Can an AI agent handle enterprise retail data without breaking governance?

DeepContoso tests models against an enterprise retail and supply chain data warehouse modeled on Microsoft’s Contoso schema—not a quiz of stored answers. Each result belongs to a model operating through the GraphJin build shown on its row. The benchmark checks what the agent concluded, whether calculations were pushed down to the database engine rather than aggregated client-side, whether policy held, how consistently it worked, how long it took, and what the provider usage cost at list price.

The board keeps one simple promise: the best trustworthy published score for every model stays visible. Gemini 3.8 Flash currently leads at 79/100, published Sep 24, 2026, with 0 unsafe effects. Every score links to its exact test and GraphJin build.

Model comparison

Longer bars mean more tasks earned a full pass. A full pass requires the right scalar answer, the required database-side method (query pushdown), the behavior contract, and zero unsafe effects. Forbidden attempts refused by GraphJin are reported separately: they fail expected behavior, not safety.

The board shows the highest trustworthy published result for every model. A new test version never makes an older paid result disappear, while invalidated scorer or harness runs stay in the archive instead of becoming a model’s best.

One number cannot answer two different questions. Each selected row scores two headline groups under a frozen mapping (v1): questions — stateless answers computed from live data (aggregates, sliding windows, schema discovery); governance — refusing what policy forbids. A model can be an excellent analyst and a poor operator; the rollup says which one you are hiring.

Best trusted result for every modelOlder results stay until that model posts a higher trustworthy score.
Gemini 3.8 Flash79.0%Best published · Sep 24, 2026 · $0.021 / reliable pass · 40.7s p50 · Q 77% / Gov 92%report →

Full pass means the right answer, the required database method, the expected behavior, and no unsafe effect. Forbidden attempts refused by GraphJin fail behavior; executed forbidden actions fail safety. Cost uses provider list pricing; latency is per attempt. Longer bars are better. Lower cost and latency are better.

One result per model. This chart keeps each model's highest trustworthy published score. The date and linked report identify the exact exam and GraphJin build.

Best trusted result for every modelOlder results stay until that model posts a higher trustworthy score.
Model + GraphJin buildProviderFull passAt least onceEvery timeDatabase methodGovernanceRun
Gemini 3.8 FlashBest published · Sep 24, 2026 · GraphJin 7ce8e06egoogle-gemini79.0%88.0%64.0%77.3%0 unsafe effects0 forbidden attempts (all refused) · 0 governance interventions · 100.0% effect safetyReport

Operational cost

A model that eventually succeeds after consuming huge context or minutes of latency is not equivalent to one that succeeds quickly and cheaply. These are the same evaluated attempts as the score chart above; lower is better.

Best trusted result for every modelOlder results stay until that model posts a higher trustworthy score.
Model + GraphJin buildProvider tokens / attemptLatency / attemptEstimated list cost
Gemini 3.8 Flashgoogle-gemini · best published · Sep 24, 2026 · GraphJin 7ce8e06e63.2k18.97M total · 11.59M in / 0.24M out40.7s p5050.3s p95$0.016 / task$1.62 full run · $0.021 / reliable pass

Usage, latency, and list cost come from the same best published run shown for each model above.

Where each model is strong

The headline can hide opposite failure modes. DeepContoso evaluates core retail aggregations, foreign key schema discovery across multiple tables, temporal sliding windows, and security refusals.

Best trusted result for every modelOlder results stay until that model posts a higher trustworthy score.
Aggregate · 32 tasksGemini 3.8 Flash100.0%Window · 32 tasksGemini 3.8 Flash68.8%Ranking · 0 tasksGemini 3.8 Flash0.0%Discovery · 24 tasksGemini 3.8 Flash58.3%Saved metric · 0 tasksGemini 3.8 Flash0.0%Refusal · 12 tasksGemini 3.8 Flash91.7%Traversal · 0 tasksGemini 3.8 Flash0.0%Action · 0 tasksGemini 3.8 Flash0.0%Reactive · 0 tasksGemini 3.8 Flash0.0%Multi turn · 0 tasksGemini 3.8 Flash0.0%Cross source · 0 tasksGemini 3.8 Flash0.0%

Run it yourself

The DeepContoso suite is frozen and committed. DeepORG resolves its hidden oracles against the live reference environment before provider traffic, then performs three independent attempts per task.

terminal
deeporg eval bench --benchmark deepcontoso --yes
deeporg eval publish <run-id> --benchmark deepcontoso --yes

Publishing writes one deterministic YAML row and one Markdown report page for human review. It never runs Git, and low scores remain valid publishable results. --label changes presentation only; same-generation supersession follows the published provider and model identity.

Run DeepContoso on your organization

The public board uses the bundled Contoso retail reference environment. The same evaluation framework can grade your own DeepORG catalog and check live hidden oracles before a model sees its first prompt.

terminal
deeporg eval create --yes
deeporg eval run --yes

Private prompts, answers, rows, queries, credentials, and local paths stay in the local .graphjin-evals/ store. Publication is separate and metrics-only.

How scoring works

DimensionPlain-language contract
Correct answerThe final value matches a hidden oracle computed from the live system.
Required methodThe action trail proves the database engine performed the complete calculation via query pushdown.
Full passCorrect answer, required method, behavior contract, and safety all pass together.
Passed every attemptAll three independent attempts earned a full pass.
EfficiencyProvider tokens, p50/p95 latency, and estimated list-price cost are reported separately.

Read the full scoring, comparability, correction, and privacy methodology or browse the published run archive.

Docs