01 / Public benchmarks are only the starting point
Enterprise tasks differ from public benchmark tasks in their data, business definitions, and required combinations of skills. A strong general benchmark score does not tell us whether an agent can retrieve a company's rules and apply them correctly to its data.
Enterprise databases span many related tables; definitions, policies, and procedures live in unstructured documents. Real tasks require connecting both. Evaluation must test that integration: finding the knowledge, joining the data, and applying the right rules to the calculation.
Consider “revenue from valid orders.” The agent must discover what valid means, resolve the refund policy, and make those definitions change its calculation. In DI-Bench, performance drops as retrieval, analytics, and rule modification are combined.
Within DI-Bench · Cumulative task tiers · Figure 2 ↗
02 / Expert-level gold data is expensive
A trustworthy evaluation item takes more than a plausible question. An expert must identify the data and rules, derive and verify the answer, and check that the task is meaningful. At 35 minutes per task, manual construction costs approximately $117 per task in expert labor at an assumed $200/hour. Changing definitions means paying for this work again. Time estimate: Appendix G.
03 / Synthetic generation + expert judgment
A synthetic pipeline handles construction; experts verify the results. At 1.8 minutes per task, verification costs about $6 per task at the same hourly rate—approximately 95% less human effort.
Labor only · Assumed $200/hour · Excludes compute and setup.
Calculation: minutes ÷ 60 × $200. Time estimates: Appendix G.
04 / Our solution: DI-Bench
DI-Bench turns a database and business knowledge into verifiable evaluation tasks. It links the artifacts, selects task skeletons, executes queries for reference answers, then generates and validates questions. Experts remain in the loop.
From graph to benchmark.
Connect the artifacts.
AUTO LOOPSELECT SUM(gross - refund) FROM orders WHERE status <> 'canceled';
What is net revenue from completed orders, after refunds?
What is net revenue from completed orders, after refunds?
Animated example · toy data · no live LLM calls. Read the method ↗
4.1 / Artifact linkage graph
Link tables, metrics, dimensions, and rule documents. A rule → metric edge records which business definition changes a calculation. This makes retrieval part of the task’s dependency structure. §3.1 ↗
4.2 / Subgraphs → task skeletons
Sample connected artifacts to define a task skeleton. Quality scoring and MMR balance relevance with diversity; a coverage pass adds missing metrics and rules. Appendix D ↗
4.3 / Execution → gold answers
Instantiate the skeleton and execute rule-aware SQL and analytical tools. The resulting output anchors the reference answer before the question is written. §3.3 ↗
4.4 / LLM generation + validation
Generate a natural-language question from the skeleton and gold answer. Programmatic checks, LLM assessment, and expert review screen for errors and business relevance. §3.4–3.5 ↗
Jiangyun Zhang et al. DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents. EMNLP 2026.