Hugging Face Open LLM Leaderboard: CO₂ Emissions and Model Performance Analysis

Hugging Face has integrated carbon emission estimates into the Open LLM Leaderboard to provide transparency regarding the environmental impact of model inference. The analysis reveals that community fine-tuned models often achieve higher carbon efficiency than their official counterparts, likely due to more concise outputs and better instruction following.

Methodology for Calculating CO₂ Cost

To estimate the environmental impact of model evaluations, Hugging Face uses a heuristic based on the specific hardware and electricity used during inference. The calculation factors in:

  • Evaluation time: The duration of the inference process.
  • Energy usage: The power consumption of the cluster's hardware.
  • Carbon intensity: The specific carbon footprint of the electricity source powering the hardware.

This data represents the emissions generated during the specific inference setup used for the Open LLM Leaderboard evaluations and is not a universal constant for the models themselves.

General Trends in Model Performance and Emissions

Analysis of over 2,700 models shows a clear correlation between model size and CO₂ cost, though performance gains do not always scale proportionally with emissions.

Official Provider Models

Official models from major research groups and corporations (e.g., Google, Meta, Alibaba) show the following patterns:

  • Diminishing Returns: As model size increases, leaderboard scores do not always increase proportionally to the rise in CO₂ cost.
  • MoE Efficiency: Mixture-of-Experts (MoE) models can exhibit a poor leaderboard score-to-emission ratio because extremely long inference times can offset the computational advantages of activating only a subset of parameters.
  • Instruction Tuning Verbosity: While instruction-tuned models generally outperform base models, some are exceedingly verbose. This verbosity inflates inference time and energy consumption during generative evaluations like MATH and IFEval.

Community Releases

Community-driven models show distinct trends compared to official releases:

  • High Efficiency in Small Models: Models below 10B parameters from the community can achieve average scores of up to 35 with less than 5kg of CO₂.
  • Superior Efficiency in Fine-tunes: Community fine-tunes and merges tend to be more CO₂ efficient than the official models they are based on.

Detailed Insights: High-Parameter vs. Compact Models

The relationship between fine-tuning and emissions varies depending on the model scale.

High-Parameter Models (e.g., 70B)

For large-scale models like Qwen2.5 and Llama3.1, official fine-tuned versions can consume twice as much energy as their base models. In contrast, for Qwen2, the base model was found to be significantly more energy-intensive than its fine-tuned versions.

Compact Models (7B+)

There is no consistent trend for models in the 7B+ range. For Llama3.1 and Qwen2.5, base models consume twice as much energy as fine-tuned versions. However, for Qwen2 and Mistral v0.3, community fine-tunes showed higher consumption.

Case Study: Verbosity and Efficiency in Qwen2 and Llama Models

Analysis of specific model behaviors suggests that verbosity and the ability to follow instructions are primary drivers of inference-related emissions.

The Impact of Instruction Following

Base models often struggle with instruction following, frequently attempting to "continue" a prompt rather than answering it. This leads to repetitive patterns or excessively long, incoherent outputs. Because generative tasks require inferring more tokens, this verbosity significantly increases CO₂ emissions.

  • Qwen2-72B Analysis: The base model Qwen2-72B produced much higher emissions than the community fine-tune calme-2.1-qwen2-72b. The community fine-tune provided more concise, direct responses, whereas the base model produced verbose and sometimes repetitive text.
  • Llama-3.1-8B Analysis: The base model produced extremely long responses (e.g., 5,475 characters for a single task) due to its tendency to repeat constraints. While the official instruct model improved coherence and reduced verbosity, the community fine-tune Llama-TI-8B occasionally produced meta-commentary and long responses, similar to the base model.

Conclusion and Open Questions

Fine-tuning improves output coherence and conciseness, which reduces computational load and potentially lowers CO₂ emissions. However, the exact mechanism—whether it is due to shorter token counts or improved task termination—remains a subject for further research.

Open research questions include:

  • Whether dataset contamination in benchmarks like MATH or IFEval allows models to terminate inference earlier, artificially improving efficiency.
  • How token parsing and verbosity in chat models specifically influence energy consumption.
  • The factors driving unexpectedly high emissions in certain MoE architectures.

Sources