DAYJOB: Tax Benchmark Evaluates AI Agent Performance on Professional Tax Assignments

Evaluating Professional Judgment in Tax AI

AI agents currently struggle to replicate the professional judgment required for tax advisory, often missing critical deadlines or failing to verify foundational assumptions. While models can perform individual calculations or retrieve data, they frequently fail to identify "consequential issues"—problems that exist outside the initial scope of a client's request but are vital to the financial outcome.

This gap in performance is the focus of DAYJOB: Tax, a new benchmark from Surge AI designed to evaluate AI agents on 50 professional tax environments. Unlike previous benchmarks that focus on isolated questions or simple calculations, DAYJOB: Tax requires agents to process messy, real-world records and determine what actions are necessary to protect the client's interests.

The DAYJOB: Tax Benchmark

DAYJOB: Tax assesses performance across eight areas of practice, including individual, small-business, partnership, estate, and international taxation. The benchmark includes 1,789 workspace files, ranging from standard tax forms and K-1s to unstructured data like handwritten receipts, field dispatch logs, and email correspondence.

To pass a task, an agent must meet a 95% rubric threshold, which evaluates:

  • Professional Judgment: Recognizing issues the client failed to mention.
  • Evidence Evaluation: Resolving conflicting information and identifying missing documentation.
  • Instruction Following: Delivering accurate, compliant workpapers or advice.

Performance Results

Current frontier models show limited success in these complex environments. Claude Sonnet 5.5 leads the leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 (9.6%) and Claude Opus 5.5 (9.2%). These results highlight that while models are improving, they remain far from a human-level baseline for end-to-end tax advisory.

Case Study: The Hidden Deadline

In one scenario, an agent was asked to evaluate the tax implications of selling a condo. While the agent correctly identified that the sale date satisfied the residence requirement for a $250,000 tax break, it failed to warn the client about a looming deadline that would cause them to lose eligibility if the sale were delayed. This illustrates the core challenge: the agent answered the specific question asked but failed to act as a professional advisor who anticipates risks the client does not know to ask about.

Community Perspectives

Discussion surrounding the benchmark and the broader application of AI in tax preparation highlights several key tensions:

  • The Need for Scaffolding: Some observers suggest that performance could be improved by using task-specific harnesses or "scaffolding" that forces the model to compile requirements before generating advice. As one commenter noted, "These types of detail-oriented professional tasks seem like they would perform better with task-specific harnesses."
  • Math and Logic Limitations: Users point out that LLMs often struggle with basic arithmetic and physical-world reasoning, which are critical for tax work. One contributor noted, "For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a support service."
  • Privacy and Trust: A significant concern remains the security of sensitive financial data. Users expressed hesitation about feeding personal financial information into LLMs, noting that while human accountants also make mistakes, the lack of accountability for AI-generated errors remains a major hurdle.
  • Practical Utility: Many users report that AI is currently a useful tool for auditing their own draft returns or categorizing transactions, but it is not yet reliable enough to serve as a standalone tax preparer. As one user summarized: "It is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check."

Sources