The Open Medical-LLM Leaderboard: Benchmarking Large Language Models in Healthcare

Hugging Face has launched the Open Medical-LLM Leaderboard, a standardized platform designed to evaluate and compare the performance of Large Language Models (LLMs) across a diverse range of medical tasks. This initiative addresses the critical need for domain-specific benchmarking in healthcare, where inaccuracies in model outputs can lead to severe consequences for patient care and treatment outcomes.

Evaluation Framework and Datasets

The Open Medical-LLM Leaderboard uses accuracy as its primary evaluation metric to measure the percentage of correct answers provided by models across several specialized medical QA datasets:

MedQA

MedQA consists of multiple-choice questions from the United States Medical Licensing Examination (USMLE), assessing the medical knowledge and reasoning skills required for U.S. medical licensure. It includes 11,450 questions in the development set and 1,273 in the test set.

MedMCQA

Derived from Indian medical entrance examinations (AIIMS/NEET), MedMCQA is a large-scale dataset covering 2.4k healthcare topics and 21 medical subjects. It contains over 187,000 questions in the development set and 6,100 in the test set.

PubMedQA

PubMedQA is a closed-domain QA dataset consisting of 1,000 expert-labeled pairs. Models must provide a yes/no/maybe answer based on an associated PubMed abstract, testing the ability to comprehend and reason over scientific biomedical literature.

MMLU Subsets

The leaderboard utilizes specific subsets of the Measuring Massive Multitask Language Understanding (MMLU) benchmark to evaluate specialized domains:

  • Clinical Knowledge: 265 questions on clinical decision-making.
  • Medical Genetics: 100 questions.
  • Anatomy: 135 questions.
  • Professional Medicine: 272 questions.
  • College Biology: 144 questions.
  • College Medicine: 173 questions.

Key Insights and Model Analysis

Evaluation of various models reveals a performance gap between commercial and open-source options, though some smaller models remain competitive:

  • Commercial Dominance: GPT-4-base and Med-PaLM-2 consistently achieve the highest accuracy scores across various medical datasets.
  • Open-Source Competitiveness: Models with approximately 7 billion parameters, including Starling-LM-7B, gemma-7b, Mistral-7B-v0.1, and Hermes-2-Pro-Mistral-7B, show competitive performance on specific tasks.
  • Shared Strengths: Both commercial and open-source models perform well in applying clinical knowledge (MMLU Clinical Knowledge) and reasoning over biomedical literature (PubMedQA).
  • Gemini Pro Performance: While Gemini Pro excels in data-intensive and procedural tasks such as Biostatistics, Cell Biology, and Obstetrics & Gynecology, it shows moderate to low performance in Anatomy, Cardiology, and Dermatology.

Model Submission Requirements

To be evaluated on the leaderboard, researchers must ensure their models meet the following technical criteria:

  1. Safetensors Format: Model weights must be converted to safetensors for safer and faster loading.
  2. AutoClasses Compatibility: Models and tokenizers must be loadable using the AutoConfig, AutoModel, and AutoTokenizer classes from the Transformers library.
  3. Public Access: Models must be publicly accessible; private models cannot be evaluated.

Currently, the leaderboard does not support models requiring use_remote_code=True, though this feature is in development.

Future Roadmap

The Open Medical-LLM Leaderboard intends to expand its scope to include:

  • Diverse Datasets: Incorporating data from radiology, pathology, and genomics through industry and research collaborations.
  • Advanced Metrics: Moving beyond simple accuracy to explore Pointwise scores and other domain-specific metrics that better capture the unique requirements of medical applications.

Sources