Hugging Face Benchmarking Open Models on Agentic Tooling

Hugging Face has developed a benchmarking harness to measure not just whether a coding agent reaches a correct answer, but the effort—measured in tokens, time, and turns—required to get there. Using the transformers library as a case study, the research demonstrates that optimizing software for agents (e.g., adding a CLI or curated documentation) can significantly reduce latency for large models while potentially introducing ambiguity and failures for smaller models.

Measuring Agentic Efficiency Beyond Final Answers

Traditional benchmarks often focus solely on the final output, which masks the operational cost and reliability of the process. Hugging Face argues that two agents reaching the same result can have vastly different profiles in terms of cost, latency, and token usage. For example, one agent might write a complex 40-line Python script to perform sentiment classification, while another might use a single CLI command.

To capture these nuances, the benchmarking harness evaluates agents across three distinct tiers of environment access:

• Bare: The agent only has the library installed via pip install transformers.
• Clone: The agent has the full source code of the library checked out in its working directory.
• Skill: The agent is provided with a packaged "Skill," consisting of curated CLI documentation and task-specific examples loaded into its context.

Impact of Model Size on Tool Adoption

The research reveals a divergent impact based on the size and capability of the open models driving the agents:

Large Open Models

For highly capable models, task completion often saturates near 100%, making "match %" a less useful metric. Instead, the focus shifts to the effort required. The study found that introducing a dedicated CLI and Skill reduced the median time spent on tasks for large models, as they shifted from debugging Python scripts to using streamlined CLI commands.

However, this efficiency comes with a token trade-off. In the clone variant, agents often read the new CLI implementation and example scripts to learn the interface, increasing the median input from approximately 4k to 6.4k tokens. This discovery cost is typically a one-time expense that would be amortized over multiple tasks in a real-world session.

Small Open Models

For smaller models, the "match %" remains a critical metric. The findings indicate that while a good tool surface is essential, adding new affordances can be counterproductive. For instance, the Qwen3-4B model saw a massive jump in token consumption (from ~2.4k to ~23k) in the clone tier after the CLI was added, as it read the source code in bulk without any gain in accuracy.

More critically, some smaller models experienced a collapse in correctness. The Qwen3-14B model's match rate for sentiment classification dropped from 100% (in the clone variant) to 0% when using the Skill variant. Traces revealed that the model mistook the Skill documentation for a tool it could call directly (e.g., transformers(command="classify", ...)), rather than a command to be executed via a bash shell, leading it to declare the task impossible.

Technical Implementation and "Markers"

To analyze agent behavior beyond raw metrics, the harness utilizes "markers"—named patterns that match specific behaviors within a run. For the transformers study, two primary markers were used:

• cli: Triggered when the agent invokes the transformers command-line tool.
• pipeline: Triggered when the agent uses the high-level pipeline(...) Python API.

Data showed that larger models were more likely to leverage new context (the CLI) via the Skill tier, with an adoption rate of 55.3%, whereas smaller models relied more on memorized API patterns from their training data.

Key Takeaways for Library Maintainers

The primary conclusion for software developers is that agent-facing APIs must be evaluated across a spectrum of model sizes. A feature that streamlines the workflow for a strong model can introduce ambiguity or trigger failure modes in smaller models. To mitigate this, Hugging Face suggests using tools like Upskill, which generates and validates Skills using strong models only when they measurably improve the performance of smaller models.

Tooling and Availability

The benchmarking harness is implemented as a CLI called agent-eval. It is designed to be profile-based and adaptable to any library that can be operated from the command line. The system utilizes Hugging Face Jobs for parallel execution on identical hardware and stores results and traces in Hugging Face Buckets for analysis via the Hub's agent-traces viewer.