Skip to content
Julius Research

DeepFinanceBench: Can AI diligence a $100MM deal?

Contents · 5 sections

DeepFinanceBench is an ultra long-horizon benchmark for measuring models on real world Private Equity and Investment Banking workflows that would take a human professional between 20 to 30 hours to complete. DFB was composed and reviewed by Private Equity (PE) and Investment Banking (IB) professionals from firms like JP Morgan, UBS, Evercore and institutions like Harvard Business School, Wharton, Columbia Business School among others.

  • Task count: 21
  • Average human time: 13 hours
  • Average rubric criteria: 84 criteria per task
  • Average Environment size: 19 files per data room
  • Average prompt length: 240 words, ~29% shorter than GDPval prompts

Philosophy

There is a fundamental gap in how AI models are evaluated on real-world high finance work. We believe this gap stems from a deeply flawed understanding of how analysts in high finance perform real-world tasks.

Existing evals measure the models in two ways: their ability to retrieve answers from a set of financial documents and their ability to produce numerically correct final outputs. However, real world private equity and investment banking workflows are meaningfully more complicated.

In software benchmarks (like SWE-bench and DeepSWE), models are graded on their output's ability to pass tests. If tests pass and the fix works, the path (or trajectory) the agent took matters less. Finance has no similar test suite. A high finance valuation is like a pull request that has to survive line-by-line code review, where every number cites its source, and every assumption needs to hold up in front of the MD, the IC, and a client.

In PE & IB, while two analysts could produce identical numbers for the same task, one output could rest on completely incorrect assumptions. Change a single number upstream, and the entire workflow no longer makes sense. The trajectory of an ultra long-horizon high-finance task is full of assumptions and judgement calls. Without being able to audit the steps and judgement of the analyst, a professional in high-finance cannot blindly trust the work that is presented to them.

Failure modes of current evals

Existing high-finance benchmarks have two main limitations in how they currently evaluate models in high finance tasks:

  1. Over emphasizing question-answer ability from financial documents
  2. Grading that is limited to numerical correctness of the outputs

Existing benchmarks often fail to consider the real-world working environment that high-finance professionals face daily. A set of financial documents and revisions in varied formats, long email chains, sparse meeting notes, and other data room objects almost invariably contains conflicting information that must be reconciled. The model's ability to judge and choose from overlapping sources is rarely measured.

  • FinanceBench: Questions are simple and too easy; this is not realistic or representative of the deal work analysts are hired for.
  • Finance Agent Benchmark: Each question is a short research task on public SEC filings (avg human time = 16.8 minutes; DFB up to 20 hours).
  • BizBench: Checks one final number against an answer key; DFB grades end to end workflows against up to 140 criteria.
  • FinanceQA: Spoon-feeds the model relevant 10-K sections with each question, leaving no room to evaluate model trajectory.
  • BigFinanceBench: Uses solid trajectory reasoning but limits the scope of questions and stops well short of end-to-end workflows.

Benchmarks like OfficeQA by Databricks are great at properly tracking a company taxonomy, but rarely step beyond grading pure numerical correctness. For high-finance ultra long-horizon tasks, trajectory is crucial to understand and audit the underlying method and assumptions that were put into the work.

ELO-based preference ranking used by benchmarks like GDPVal miss the nuance in the complex long horizon workflows by only comparing binary final outputs, and don't grade on the atomic trajectory.

There are benchmarks, like Halluminate's Westworld Finance Diligence Bench, that have attempted to measure semi-long horizon work. These benchmarks only study the agent's trajectory and leave it out of the score, since every point on their leaderboard is from checks on the final output. While this might be an effective way to check final answers, this fundamentally is not how finance work is judged in the real world. Correct numbers without a source or auditability will be scrutinized by an investment committee. DFB grades whether every number is sourced and every judgement call is auditable and traceable.

These highlighted approaches do not represent real world uses. DFB was built to demonstrate that the path of the task itself carries the assumptions and therefore grading must account for the trajectory when evaluating model performance. Auditability is paramount to internal and external stakeholders like lenders, clients, and brokers. Understanding how a model (or an analyst) came to an assumption is the key to understanding what could go right, wrong, or sideways in a multi-million dollar deal.

Methodology

Task construction

DeepFinanceBench covers four primary pillars of high-finance work:

  • Valuation: estimate what a business is worth and justify the assumptions.
  • Due diligence: investigate the records, identify important risks and explain their effect on the deal.
  • Underwriting: assess the deal's risks and economics, then determine acceptable financing or investment terms.
  • Financial modeling: build a model whose assumptions, calculations and outputs remain consistent.

Each ultra long-horizon task pairs a request with a data room: the collection of records needed to complete the assignment. Tasks use real data that has been scrubbed and anonymized. The request states the objective, constraints and expected deliverables; from there the agent must work out how to complete the analysis. Every DFB problem statement has on average 240 words; we find this to be realistic in crafting a real world end to end workflow. This compares to other evals that have prompts that are very short, but only handle simple arithmetic / quantitative tasks.

Each DFB task comes with a data room of on average 22 files, comprising the same material analysts work with from day to day. File types include spreadsheets (.xlsx), Word documents (.docx), emails (.eml), PDFs and raw CSV extracts. Inside are emails, chats and deal-team notes; models and projections; CIMs, SPAs, term sheets, commitment letters, and transaction data; diligence reports; IC minutes; and market data and comps.

Each ultra long-horizon task is representative of end-to-end work done at top PE & IB firms like Evercore, JP Morgan, UBS, and every task utilizes real data that has been anonymized and scrubbed. The work simulates analyst work on real deals, not just simple Q&A on sets of documents, or its ability to find one off answers. There is ambiguity, and there are conflicting sources.

Sample task: a football field valuation for the Board

Prompt

Noor‚ we have been asked to prepare a football field valuation of Larchmont's cloud infrastructure for the Board as of Sep 30‚ 2026 to share with the Board on Tuesday‚ Oct 6․ Today is Thursday‚ Oct 1․ See section 5 of the valuation guidelines for instructions on preparing both low and high share price valuations using eight different methods‚ selecting a share price range and premium‚ passing through the four gates‚ and making a recommendation․ You’ll need to make Larchmont_Valuation․xlsx and Larchmont_Board_Memo․md which need to be saved in the /app/output folder․ Don’t round numbers in the model, keep it formula driven, only round numbers in final outputs․ This project needs to be completed ASAP․

Context given

23 files in the model’s working folder. Every file is synthetic, written by the task’s author for this task.

Policies, standards, minutes, filings and the tax lines, as Word documents.

  • 01_Larchmont_Strategic_Valuation_Standards.docx38 KB
  • 02_Larchmont_Non_GAAP_and_Rule_of_40_Policy.docx37 KB
  • 03_Larchmont_Valuation_Methodology.docx37 KB
  • 04_Larchmont_Capitalization_and_Security_Treatment.docx37 KB
  • 09_Larchmont_Board_Minutes_Excerpt_2026-09-25.docx37 KB
  • 10_Larchmont_Press_Release_2026-09-28.docx36 KB
  • 11_Sector_Wrap_Cloud_Infrastructure_2026-09-25.docx36 KB
  • 20_Larchmont_Q2_2026_Form_10-Q_Excerpt.docx37 KB
  • 23_Larchmont_Tax_Lines_for_Linden.docx36 KB

Which file would you trust?

Two files in the data room disagree about a $76.0m accrual, and the equity bridge either deducts it or it doesn’t. This is the call the rubric’s headline Judgement criterion checks. Make it before you read on.

Scoring

Graders in DFB evaluate the finished deliverable together with evidence of the trajectory taken. Sources, tool results, calculations and revisions can show whether the agent completed the required work, and agents are graded on labeled rubrics with label categories:

  • Output: Are the calculations, conclusions and requested deliverables correct?
  • Workflow: Did the agent find the relevant evidence and use it in the analysis?
  • Judgment: Does the evidence support the agent's assumptions and decisions?

Each rubric criterion was written and verified by experts from the world's top PE and IB firms.

By tracking the steps taken with respect to these three categories, both experts and reviewers are able to audit both workflow and judgement, irrespective of final output. When it comes to ultra long-horizon tasks that can take 20-30+ hours for an analyst to complete, trajectory is invaluable.

DFB tasks test whether an agent can carry decisions through a long assignment and abide by "best practices" along the way. A source chosen early may affect a forecast, the amount a company can borrow, its valuation and the final recommendation. When an input changes, every calculation that depends on it needs to change too. But that doesn't mean that an agent should only be penalized for upstream decisions that go wrong; we should also evaluate the quality of steps and judgements made along the way, irrespective of output. This is how we capture trajectory.

DFB graders give partial credit when the models take the right steps on the trajectory.

Quality assurance

Every task in DFB was reviewed by multiple layers of agentic verifiers and human expert peers with backgrounds from top firms like Evercore and institutions like Harvard Business School and Wharton. And each task on average has 90 rubric criteria.

Expert review and verifiers established four parameters:

  1. The task is clear and the source files contain enough information to complete it.
  2. The reference solution is supported by the evidence.
  3. The rubric checks the requested work and accepts valid solutions.
  4. A failed criterion reflects the agent's work, rather than a faulty reference, grader, or environment issue.

Reviewers traced disputed grades and expert commentary back to the source files and agent traces. This distinguishes model mistakes from evaluation mistakes, ensuring that errors in the evaluation process do not influence our conclusions about model capability.

Harness

Contrary to many benchmarks, DeepFinanceBench evaluates models in the harness they are most likely to be used in. This means that our environment supports a multi-harness setup where:

  • Claude models run in claude-code v.2.1.318
  • GPT models run in codex v.0.166.0
  • Gemini, Muse, Deepseek, GLM, minimax, Kimi models run in opencode-ai v.1.19.3
  • Qwen models run in qwen-code v.0.26.6
  • Grok models run in x.ai v.1.0.70
  • Mistral models run in mistral-vibe v.3.0.4
  • Node runs on v.22.23.4

Each harness has nearly every native tool enabled, including web search. The one exception is the ask-user tool, which we turn off so every run can finish on its own. Many current evals cut off the agent's access to the internet so agents cannot look up the answer keys. DFB doesn't need to. Every run gets its own sandbox, and the answer key and rubric sit in separate, isolated grading environments.

The goal of DFB is to test agents in environments that are representative of those that PE and IB professionals use in their day-to-day work. This approach provides a more realistic evaluation of model performance rather than using a single universal harness that imposes arbitrary or artificial constraints on the models.

Results

Quantitative

DeepFinanceBench leaderboard: score with 95% confidence interval and worst-to-best attempt spread, agent cost and output tokens per attempt, and pass rate by criterion type. Column headers sort the table.
#Model
=1GPT‑6 AstraOpenAI · Extra high · Codex56.5 ±8.2A 38 · T 89 · J 82$4.32
=1Claude Sonnet 5.5Anthropic · Max · Claude Code · 188/190 attempts55.5 ±9.8A 40 · T 87 · J 75$6.64
=1GPT‑6.1 SolgraderOpenAI · Extra high · Codex55.0 ±8.0A 37 · T 88 · J 80$0.77
=1Claude Opus 5.5selectionAnthropic · Max · Claude Code51.6 ±9.3A 34 · T 79 · J 67$7.86
=1Claude Fable 5.1Anthropic · Max · Claude Code · 187/190 attempts46.4 ±5.5A 24 · T 83 · J 73$11.15
=1GPT‑6.1 TerraOpenAI · Extra high · Codex44.7 ±7.2A 24 · T 82 · J 73$0.36
2Gemini 4 ArgonGoogle · Default · OpenCode · 187/190 attempts41.1 ±6.5A 19 · T 71 · J 61$2.56
6Grok 4.7SpaceXAI · Extra high · Grok Build · 188/190 attempts33.0 ±5.2A 15 · T 66 · J 58$4.16
7Muse Spark 1.3Meta · Default · OpenCode32.6 ±4.7A 14 · T 65 · J 59$0.67
=8DeepSeek V4.1 FlashDeepSeek · Default · OpenCode · 181/190 attempts30.6 ±3.5A 14 · T 68 · J 57$0.12
=8Qwen 3.8 MaxAlibaba · Default · Qwen Code · 168/190 attempts30.4 ±3.8A 14 · T 69 · J 55$4.12
=8GPT‑6 LunaselectionOpenAI · Extra high · Codex25.7 ±4.9A 11 · T 61 · J 53$0.08
12Mistral Large 4Mistral · Default · Mistral Vibe · 66/190 attempts20.7 ±5.8A 7 · T 43 · J 28$1.70
ModelAccuracyTrajectoryJudgement
GPT-6 Astra Extra high
Claude Sonnet 5.5 Max
GPT-6.1 Sol Extra high
Claude Opus 5.5 Max
Claude Fable 5.1 Max
GPT-6.1 Terra Extra high
Gemini 4 Argon Default
Grok 4.7 Extra high
Muse Spark 1.3 Default
DeepSeek V4.1 Flash Default
Qwen 3.8 Max Default
GPT-6 Luna Extra high
Mistral Large 4 Default
Share of each type of criterion passed across all counted attempts, with its 95% interval clustered by task. All three columns share one 0–100 scale. Gray rows are configurations whose attempts mostly left no file.

171 of 1,544 criteria were never passed by any of the 13 ranked configurations, in any attempt; 31 were passed every time.

Accuracy · 991 criteria

Are the numbers right?

157 criteria (16%) never passed

Trajectory · 330 criteria

Was the work done and checked?

2 criteria (1%) never passed

Judgement · 223 criteria

Did it trust the right source?

12 criteria (5%) never passed

Criteria per pass-rate bin, by criterion type, for All ranked configurations
Pass rateAccuracyTrajectoryJudgement
never passed157212
passed by more than 0% and at most 10% of attempts338915
passed by more than 10% and at most 20% of attempts99145
passed by more than 20% and at most 30% of attempts1631818
passed by more than 30% and at most 40% of attempts381715
passed by more than 40% and at most 50% of attempts22168
passed by more than 50% and at most 60% of attempts30129
passed by more than 60% and at most 70% of attempts272010
passed by more than 70% and at most 80% of attempts172516
passed by more than 80% and at most 90% of attempts192831
passed by more than 90% and less than 100% of attempts7814679
passed by every attempt3235
Each criterion is placed by the share of graded attempts that passed it, pooled over every ranked configuration. The first and last bars are criteria passed by no attempt and by every attempt; the bars between are ten-point bins. Bars share one scale across the three panels. Configurations whose attempts mostly left no file are out of the pool. Rubric text stays private.

Qualitative analysis

  1. GPT-6.1 outpaces Claude Opus 5.5 on ultra long-horizon finance tasks

    These tasks require agents to reconcile extensive source materials, make financial decisions and carry their consequences through interconnected calculations. The results show that the GPT-6 class of models in the Codex harness hold a sustained advantage when handling common PE and IB workflows across end-to-end task performance.

  2. Even frontier models are weak at turning evidence into proper judgement

    In most DFB tasks, agents are able to find the relevant information without recognizing its implications. They also have difficulty challenging a supplied assumption or evaluating better opportunities unless strictly told to, even in situations where a human analyst would instinctively do so. This means that when a CFO says one thing and an analyst says another, there is little reconciliation effort between balancing the importance of both perspectives.

  3. Upstream mistakes cause cascading downstream failures

    Incorrect earnings, source selection or completion timing can affect debt capacity, fees, cash flows, valuations and investor returns. Atomic output checks reveal those consequences, but the number of failed checks should not be interpreted as the number of independent reasoning failures.

  4. Models often execute calculations correctly after selecting the wrong inputs

    In a lending task, agents correctly calculated debt capacity using an incorrect earnings subtotal. In a pharmaceutical royalty task, they correctly applied payment caps and milestone rules to a forecast that omitted an entire sales channel. The agents knew how to perform the calculations but failed to verify the numbers feeding them.

  5. Agents find decision-relevant evidence without acting on it

    In one acquisition task, moving the closing date changed shareholder payouts and lowered the required offer by $1.25 million. All attempts by frontier models missed this saving, even though several had identified the buyer's right to choose the date.

The path towards AGI in dealmaking

For AGI to be useful in dealmaking, its work needs to hold up when someone checks it. Across high-finance assignments that take professionals 20-30 hours, agents must choose sound assumptions, use them consistently and explain the decisions that matter.

DeepFinanceBench evaluates both the result and the work behind it. Auditability gives professionals a way to check the analysis and gives researchers a way to find the mistakes that need fixing. That brings us closer to measuring whether agents can take an assignment from a data room to a decision people can rely on.