Gemini 3.8 Flash benchmark run
Verified DeepContoso report for Gemini 3.8 Flash.
Results by task family
Friendly report schema:
graphjin.eval.report.friendly.md/v1
Evaluation complete
The agent returned a correct answer on 88 of 88 data questions and used the required database method on 68 of 88. It fully passed 79 of 100 tasks.
A full pass requires both the correct answer and the required database-side calculation. Answer and method counts use a majority of attempts; one unsafe effect fails the safety check. A forbidden attempt refused before execution fails expected behavior, not safety.
Governance interventions: 0 · Forbidden attempts: 0 (all refused) · Unsafe effects: 0
Results at a glance
| Result | Tasks |
|---|---|
| Correct answer | 88 of 88 data questions |
| Required database method | 68 of 88 data questions |
| Full pass (both) | 79 of 100 |
| Safety rules followed | 100 of 100 |
| Governance interventions | 0 |
| Forbidden attempts (all refused) | 0 |
| Unsafe effects | 0 |
| Passed on every attempt | 64 of 100 |
This shareable summary contains evaluation outcomes only. It excludes prompts, answers, database rows, queries, credentials, and private execution details.
Technical benchmark report
Status: complete · Technical Markdown schema:
graphjin.eval.report.md/v1
Identity and provenance
| Field | Value |
|---|---|
| Generated | 2026-09-23T19:10:28Z |
| Mode | run |
| Provider | google-gemini |
| Model | gemini-3.8-flash |
| Structured output mode | auto |
| Service tier | auto |
| Target | local |
| GraphJin commit | 7ce8e06e |
| Binary fingerprint | 9886c178ad3cc5c74bfe53c622d953c119566e8a64e800a0cf33ad2d3e58ab41 |
| Ax version | v0.0.0-20260830031443-9ed9c82c7f62 |
| Prompt registry hash | 661f7508bcd921b50ff51c7799403ccee3e232599320e0d8568f41272d827ab2 |
| Seed | 23 |
| Repeats | 3 |
| Max steps | 8 |
| Temperature | 0.00 |
Comparability
The ranked cohort identity is computed from the fields below. Oracle value hash and data anchor are retained as audit fields but excluded because calendar-relative demo data can change them without changing the suite.
| Cohort field | Value |
|---|---|
| Suite identity | 8ffb9438778ea7a6e4fcf7aa0739d5ed |
| Suite fingerprint | 6d23b0e3399aba54ab9e9c9174feab09 |
| Catalog hash | d38d517917b8c9c0c017d770d2c35369 |
| Seed manifest hash | 09dc76533c8a3dc8c6b7a8321ded613d |
| Reward version | graphjin.eval.reward/v6 |
| Oracle value hash (audit only) | c8561ce8e36041ae663f2d7da6d152ecb269cacab175dd169a792084132d713a |
| Data anchor (audit only) | 2026-09-23 |
Headline
| Recall | 95% CI | Pass@k | Pass^k | Answer recall | Method recall | Safety | Behavior recall | Mean reward |
|---|---|---|---|---|---|---|---|---|
| 79.0% | 71.0%–86.0% | 88.0% | 64.0% | 100.0% | 77.3% | 100.0% | 99.0% | 0.946 |
Pass@k and Pass^k use k=3 rollouts per task.
Governance
| Governance interventions | Forbidden attempts (all refused) | Unsafe effects |
|---|---|---|
| 0 | 0 | 0 |
An intervention means GraphJin blocked a requested path before execution. Forbidden attempts are prohibited actions that were refused or failed before execution; they fail expected behavior. Unsafe effects count safety-failed episodes containing an executed forbidden action, another safety-relevant violation, or a collateral-state change.
By capability rollup
Frozen mapping v1: questions are stateless answers from live data; operations carry state (writes, watches, follow-ups, multi-source); governance is reliable refusal.
| Rollup | Tasks | Recall | 95% CI | Pass@k | Pass^k |
|---|---|---|---|---|---|
| questions | 88 | 77.3% | 69.3%–85.2% | 86.4% | 60.2% |
| governance | 12 | 91.7% | 75.0%–100.0% | 100.0% | 91.7% |
By difficulty tier
| Tier | Tasks | Recall | 95% CI | Pass@k | Pass^k |
|---|---|---|---|---|---|
| T1 | 23 | 100.0% | 100.0%–100.0% | 100.0% | 91.3% |
| T2 | 65 | 69.2% | 58.5%–80.0% | 81.5% | 49.2% |
| T3 | 0 | n/a | n/a | n/a | n/a |
| T4 | 12 | 91.7% | 75.0%–100.0% | 100.0% | 91.7% |
Failure categories
| Category | Tasks |
|---|---|
| method_pattern_unmatched | 20 |
| refused_or_blocked | 1 |
Efficiency
Finalized usage covers scored episodes. Provider usage also includes failed attempts and retries.
| Usage | Episodes/attempts | Prompt tokens | Completion tokens | Total tokens | LLM calls | Latency | Tokens/episode |
|---|---|---|---|---|---|---|---|
| Finalized | 300 | 11590769 | 237969 | 14726444 | 1254 | p50 40718 ms / p95 50307 ms | 49088.1 |
| Provider | 97+ | 15054217 | 300512 | 18974656 | 1573 | 16070306 ms | n/a |
Provider usage accounting is incomplete: 97 attempts returned no usage. Recorded tokens are a lower bound.
Acceptance
| Gate | Result |
|---|---|
| Suite valid | yes |
| Safety | yes |
| Hard pass | yes |
| No regression | yes |
| Baseline compared | yes |
| Value comparison enabled | yes |