Gemini 3.8 Flash benchmark run Verified DeepContoso report for Gemini 3.8 Flash. benchmarks benchmarks benchmarks/deepcontoso/runs/20260923t135926-3603ef14 benchmarks/deepcontoso/runs/20260923t135926-3603ef14.md

Gemini 3.8 Flash benchmark run

Verified DeepContoso report for Gemini 3.8 Flash.

Reliable (recall)79.0%
Questions (stateless answers)77.3%
Refusal (rollup)91.7%
At least once (pass@3)88.0%
Every time (pass³)64.0%
Complete DB method (method recall)77.3%
Governance0 unsafe effects0 forbidden attempts (all refused) · 0 governance interventions · 100.0% effect safety
CohortRanked
Generation scope2026.1 — governed retail Q&A (aggregations, time windows, schema discovery, governance)2026.1 — governed retail Q&A (aggregations, time windows, schema discovery, governance)
Provider tokens / attempt63.2k18.97M total
Latency / attempt40.7s p5050.3s p95
Estimated list cost$0.016 / task$1.62 run · $0.021 / reliable pass

Results by task family

Aggregate100.0%Window68.8%Discovery58.3%Refusal91.7%

Friendly report schema: graphjin.eval.report.friendly.md/v1

Evaluation complete

The agent returned a correct answer on 88 of 88 data questions and used the required database method on 68 of 88. It fully passed 79 of 100 tasks.

A full pass requires both the correct answer and the required database-side calculation. Answer and method counts use a majority of attempts; one unsafe effect fails the safety check. A forbidden attempt refused before execution fails expected behavior, not safety.

Governance interventions: 0 · Forbidden attempts: 0 (all refused) · Unsafe effects: 0

Results at a glance

ResultTasks
Correct answer88 of 88 data questions
Required database method68 of 88 data questions
Full pass (both)79 of 100
Safety rules followed100 of 100
Governance interventions0
Forbidden attempts (all refused)0
Unsafe effects0
Passed on every attempt64 of 100

This shareable summary contains evaluation outcomes only. It excludes prompts, answers, database rows, queries, credentials, and private execution details.


Technical benchmark report

Status: complete · Technical Markdown schema: graphjin.eval.report.md/v1

Identity and provenance

FieldValue
Generated2026-09-23T19:10:28Z
Moderun
Providergoogle-gemini
Modelgemini-3.8-flash
Structured output modeauto
Service tierauto
Targetlocal
GraphJin commit7ce8e06e
Binary fingerprint9886c178ad3cc5c74bfe53c622d953c119566e8a64e800a0cf33ad2d3e58ab41
Ax versionv0.0.0-20260830031443-9ed9c82c7f62
Prompt registry hash661f7508bcd921b50ff51c7799403ccee3e232599320e0d8568f41272d827ab2
Seed23
Repeats3
Max steps8
Temperature0.00

Comparability

The ranked cohort identity is computed from the fields below. Oracle value hash and data anchor are retained as audit fields but excluded because calendar-relative demo data can change them without changing the suite.

Cohort fieldValue
Suite identity8ffb9438778ea7a6e4fcf7aa0739d5ed
Suite fingerprint6d23b0e3399aba54ab9e9c9174feab09
Catalog hashd38d517917b8c9c0c017d770d2c35369
Seed manifest hash09dc76533c8a3dc8c6b7a8321ded613d
Reward versiongraphjin.eval.reward/v6
Oracle value hash (audit only)c8561ce8e36041ae663f2d7da6d152ecb269cacab175dd169a792084132d713a
Data anchor (audit only)2026-09-23

Headline

Recall95% CIPass@kPass^kAnswer recallMethod recallSafetyBehavior recallMean reward
79.0%71.0%–86.0%88.0%64.0%100.0%77.3%100.0%99.0%0.946

Pass@k and Pass^k use k=3 rollouts per task.

Governance

Governance interventionsForbidden attempts (all refused)Unsafe effects
000

An intervention means GraphJin blocked a requested path before execution. Forbidden attempts are prohibited actions that were refused or failed before execution; they fail expected behavior. Unsafe effects count safety-failed episodes containing an executed forbidden action, another safety-relevant violation, or a collateral-state change.

By capability rollup

Frozen mapping v1: questions are stateless answers from live data; operations carry state (writes, watches, follow-ups, multi-source); governance is reliable refusal.

RollupTasksRecall95% CIPass@kPass^k
questions8877.3%69.3%–85.2%86.4%60.2%
governance1291.7%75.0%–100.0%100.0%91.7%

By difficulty tier

TierTasksRecall95% CIPass@kPass^k
T123100.0%100.0%–100.0%100.0%91.3%
T26569.2%58.5%–80.0%81.5%49.2%
T30n/an/an/an/a
T41291.7%75.0%–100.0%100.0%91.7%

Failure categories

CategoryTasks
method_pattern_unmatched20
refused_or_blocked1

Efficiency

Finalized usage covers scored episodes. Provider usage also includes failed attempts and retries.

UsageEpisodes/attemptsPrompt tokensCompletion tokensTotal tokensLLM callsLatencyTokens/episode
Finalized30011590769237969147264441254p50 40718 ms / p95 50307 ms49088.1
Provider97+1505421730051218974656157316070306 msn/a

Provider usage accounting is incomplete: 97 attempts returned no usage. Recorded tokens are a lower bound.

Acceptance

GateResult
Suite validyes
Safetyyes
Hard passyes
No regressionyes
Baseline comparedyes
Value comparison enabledyes
Docs