Maxwell Zhang
← All writing
WORKING DRAFT
Research notes / 001

Enterprise Agentic Evaluation

Why enterprise evaluation needs domain-specific tasks, and how we build them with DI-Bench.

3 min read

01 / Public benchmarks are only the starting point

Enterprise tasks differ from public benchmark tasks in their data, business definitions, and required combinations of skills. A strong general benchmark score does not tell us whether an agent can retrieve a company's rules and apply them correctly to its data.

Enterprise complexity lives in the connections.

Enterprise databases span many related tables; definitions, policies, and procedures live in unstructured documents. Real tasks require connecting both. Evaluation must test that integration: finding the knowledge, joining the data, and applying the right rules to the calculation.

Consider “revenue from valid orders.” The agent must discover what valid means, resolve the refund policy, and make those definitions change its calculation. In DI-Bench, performance drops as retrieval, analytics, and rule modification are combined.

OPUS 4.6 / DI-BENCHAccuracy drops as tasks get harder

Within DI-Bench · Cumulative task tiers · Figure 2 ↗

02 / Expert-level gold data is expensive

A trustworthy evaluation item takes more than a plausible question. An expert must identify the data and rules, derive and verify the answer, and check that the task is meaningful. At 35 minutes per task, manual construction costs approximately $117 per task in expert labor at an assumed $200/hour. Changing definitions means paying for this work again. Time estimate: Appendix G.

03 / Synthetic generation + expert judgment

A synthetic pipeline handles construction; experts verify the results. At 1.8 minutes per task, verification costs about $6 per task at the same hourly rate—approximately 95% less human effort.

Expert labor per task≈95% less human effort

Labor only · Assumed $200/hour · Excludes compute and setup.
Calculation: minutes ÷ 60 × $200. Time estimates: Appendix G.

04 / Our solution: DI-Bench

DI-Bench turns a database and business knowledge into verifiable evaluation tasks. It links the artifacts, selects task skeletons, executes queries for reference answers, then generates and validates questions. Experts remain in the loop.

From graph to benchmark.

DI-BENCH / THE PIPELINE
01 / 06

Connect the artifacts.

AUTO LOOP
Tables, metrics, rules, and dimensions. One linked graph.
Animation playing.

Animated example · toy data · no live LLM calls. Read the method ↗

4.1 / Artifact linkage graph

Link tables, metrics, dimensions, and rule documents. A rule → metric edge records which business definition changes a calculation. This makes retrieval part of the task’s dependency structure. §3.1 ↗

4.2 / Subgraphs → task skeletons

Sample connected artifacts to define a task skeleton. Quality scoring and MMR balance relevance with diversity; a coverage pass adds missing metrics and rules. Appendix D ↗

4.3 / Execution → gold answers

Instantiate the skeleton and execute rule-aware SQL and analytical tools. The resulting output anchors the reference answer before the question is written. §3.3 ↗

4.4 / LLM generation + validation

Generate a natural-language question from the skeleton and gold answer. Programmatic checks, LLM assessment, and expert review screen for errors and business relevance. §3.4–3.5 ↗

Read the research

Jiangyun Zhang et al. DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents. EMNLP 2026.