Hugging Face Evaluation on the Hub

Hugging Face has launched Evaluation on the Hub, a no-code tool powered by AutoTrain that enables users to evaluate any model on any dataset using any metric directly within the Hugging Face ecosystem. This tool addresses the "evaluation crisis" in modern AI by removing the technical hurdles of manual evaluation and improving the reproducibility of reported results.

Standardizing Model Evaluation on the Hub

Evaluation on the Hub simplifies the process of measuring model performance by integrating evaluation directly into the Hub's infrastructure. Instead of relying on self-reported results, which can be prone to implementation differences or bugs, the tool allows for verified performance metrics to be encoded directly into model card metadata.

Key capabilities include:

• Finding Optimal Models: Users can access dataset-specific leaderboards to identify top-performing models or run evaluations on new models not yet listed.
• Baseline Establishment: Users can upload new datasets to the Hub and run baselines across multiple models without writing code.
• Cross-Dataset Validation: Model developers can evaluate a single model across numerous related datasets to test generalization.

Technical Integration and Workflow

Evaluation on the Hub is integrated into the existing Hugging Face ecosystem through a combination of Spaces and AutoTrain. The workflow typically begins on a dataset page, leading to the model-evaluator Space.

Configuring Evaluation Jobs

While many datasets include metadata that allows for one-click evaluation, users can also manually configure jobs via Advanced configuration, which includes:

• Task and Split Selection: Defining the specific machine learning task, dataset, and data split to be used.
• Dataset Mapping: Mapping dataset columns to a standard format (e.g., specifying which columns contain images and which contain labels).
• Metric Selection: Choosing from default metrics associated with a task (such as accuracy or F1 score) or selecting specialized metrics like the Matthew's correlation coefficient.

Execution and Results

Once a user selects the models to be evaluated and submits the job, the backend (powered by AutoTrain) automatically executes the evaluation. The results are then submitted back to the Hub via a Pull Request (PR) to the model's model card, ensuring that performance data is transparent and verified. Users can also copy evaluation metadata into the dataset card to streamline future evaluations for the community.

Addressing the AI Evaluation Crisis

Hugging Face positions Evaluation on the Hub as a response to systemic issues in AI benchmarking. The company notes that while hardware and algorithms have advanced rapidly, evaluation methods have remained stagnant, leading to several critical problems:

• Benchmark Saturation: Models are outperforming humans on certain test sets faster than new benchmarks can be created.
• Brittleness and Bias: High leaderboard scores often mask underlying malicious biases or brittleness in AI systems.
• Lack of Reproducibility: The lack of openness and standardized implementation makes it difficult to reproduce reported results.
• Deployment Overlooked: Practical concerns such as efficiency and fairness are often ignored in favor of raw accuracy metrics.

Future Roadmap

Evaluation on the Hub builds upon the Hugging Face evaluate library and existing benchmarks like RAFT and GEM. Hugging Face intends to expand the tool to support more tasks and integrate a new and improved data measurements tool to move toward a more holistic, trustable, and reproducible evaluation paradigm.