Artificial Analysis LLM Performance Leaderboard on Hugging Face

Hugging Face has integrated the Artificial Analysis LLM Performance Leaderboard to provide AI engineers with a comprehensive tool for comparing the quality, price, and speed of over 100 serverless LLM API endpoints. This integration allows developers to optimize for responsiveness and cost, which are often as critical as model accuracy for consumer applications and agentic systems.

Comprehensive Performance Metrics

The LLM Performance Leaderboard evaluates both open and proprietary models across three primary dimensions to facilitate informed decision-making for AI-enabled applications:

  • Quality: A simplified index derived from MMLU, MT-Bench, HumanEval scores (as reported by model authors), and Chatbot Arena rankings.
  • Price: Reported as input/output per-token pricing. To allow for direct comparison between hosting providers, a "blended" price is calculated using a 3:1 input-to-output token ratio.
  • Speed: Measured via two key metrics:
    • Throughput: The speed of token output during inference, measured in tokens per second (TPS). The leaderboard reports median, P5, P25, P75, and P95 values over a 14-day window.
    • Latency: The Time to First Token (TTFT), measured in seconds. Like throughput, this is reported across multiple percentiles (median, P5, P25, P75, P95) over 14 days.

Additionally, the leaderboard tracks the Context Window, representing the maximum number of tokens a model can process simultaneously.

Testing Methodology and Workloads

To ensure data accuracy and reliability, Artificial Analysis employs a rigorous testing schedule:

  • Frequency: Every API endpoint is tested eight times per day.
  • Data Window: Figures represent the median measurement of the last 14 days.
  • Workload Variations: The leaderboard allows users to explore performance across six different combinations of prompt lengths (~100, ~1k, and ~10k tokens) and parallel query volumes (1 query vs. 10 parallel queries).

Market Insights and Model Segmentation (May 2024)

As of May 2024, the LLM market shows significant variance in performance and cost. There is a 300x pricing spread between high-end models like Claude 3 Opus and smaller models like Llama 3 8B.

Models are categorized into three quality segments:

  • High Quality (Higher Price/Slower): GPT-4 Turbo and Claude 3 Opus.
  • Moderate Quality (Balanced Price/Speed): Llama 3 70B, Mixtral 8x22B, Command R+, Gemini 1.5 Pro, and DBRX.
  • Lower Quality (Faster/Cheaper): Llama 3 8B, Claude 3 Haiku, and Mixtral 8x7B.

Strategic Application: Balancing Speed, Price, and Quality

Optimizing for speed and price can lead to higher overall system quality. In complex design patterns, using a combination of smaller, faster models for initial processing and larger models for final synthesis can be more effective than relying on a single large model.

For example, a web-browsing chatbot could use Llama 3 8B to extract highlights from dozens of web pages in parallel—reducing latency and cost—and then use GPT-4 Turbo to summarize the most relevant results. This approach can be more cost-effective and result in higher quality outcomes even when processing ten times more content.

Sources