Mercor Benchmark Map: 90% of US Labor Remains Uncovered by AI Benchmarks

The Gap Between AI Benchmarks and Economic Reality

Recent analysis from Mercor indicates that 90% of American working hours are spent on tasks with little or no coverage from current AI benchmarks. While AI capability research has proliferated, existing benchmarks disproportionately focus on a narrow set of tasks, leaving the vast majority of the US economy unmeasured and poorly understood in the context of AI automation.

Mercor’s "Benchmark Map" analyzed 923 O*NET occupations and 18,797 associated tasks against 29 selected benchmarks. The findings reveal that only 208 occupations have tasks covered by at least five benchmark prompts, and 6 of 22 job families have no such coverage at all. This suggests that current evaluation methods are not yet equipped to assess how AI models perform on the actual work that drives the US economy.

Why Benchmarks Focus on a Narrow Slice

Benchmark coverage is not distributed randomly; it is heavily skewed toward high-wage, AI-exposed roles. The data shows that O*NET tasks rated as "exposed" to AI are approximately six times more likely to be represented in benchmarks than unexposed tasks. Furthermore, even when controlling for AI exposure, tasks associated with occupations that have larger total wage bills are significantly overrepresented.

This concentration reflects a rational, albeit limited, allocation of research resources: developers prioritize measuring AI progress on the most economically significant and technically feasible tasks. However, this creates a "blind spot" for the long tail of middle-income knowledge work and physical-world tasks, which remain largely absent from current evaluation suites.

Challenges in Measuring Real-World Performance

Attempting to map AI performance to specific jobs faces significant methodological hurdles:

• Statistical Power: Due to the breadth of the US economy, any single O*NET task often has very few corresponding benchmark items, making performance estimates volatile.
• Task Fidelity: Benchmarks vary in difficulty and quality. Some are saturated, while others are poorly defined or susceptible to reward hacking, making it difficult to create a standardized "readiness" score for an occupation.
• Automation vs. Augmentation: Most current benchmarks measure end-to-end task automation. In practice, AI is often used to augment human productivity. Measuring this human-in-the-loop capability remains a difficult, non-scalable challenge.
• Physical Constraints: Approximately 63% of US jobs involve physical-world components that are difficult to capture in digital-first benchmarks, though this may change as robotics capabilities advance.

Community Perspectives on AI and Labor

Discussion surrounding these findings highlights a divide between the theoretical potential of AI and the reality of daily work. Many observers note that "knowledge work" often involves significant time spent in meetings, stakeholder management, and error correction—tasks that are rarely captured in standard benchmarks.

Commenters on the research raised several critical points:

"I think 'covered well' is being used here as a synonym for 'covered at all.' It’s self-evident that the spectrum of work carried out by a social worker... cannot be well represented by thirteen prompts."

"AI both can't and doesn't do your thinking for you and worse yet, it robs you of the environment where you do your best thinking..."

"It's true that we've got a lot of published benchmarks, but they tend to cover the lowest value add tasks. It's really easy to fall into the McNamara fallacy when rolling out these kinds of solutions."

Others pointed out that the focus on automating high-wage jobs may be an intentional effort to reduce corporate costs, rather than a reflection of where AI provides the most societal value. The consensus among critics is that while benchmarks provide a useful snapshot of specific model capabilities, they currently serve as a poor proxy for the overall impact of AI on the broader labor market.