Open FinLLM Leaderboard launch – comprehensive zero‑shot benchmark for financial language models
TL;DR
The Open FinLLM Leaderboard (OFLL) launched by Hugging Face provides a dedicated, zero‑shot evaluation suite for financial language models across 40 tasks and seven categories, enabling transparent benchmarking of model readiness for real‑world finance use cases.
What the Open FinLLM Leaderboard Is
The OFLL is a public, web‑based platform that evaluates large language models (LLMs) on tasks that matter to finance professionals. It fills the gap left by general‑purpose NLP leaderboards by focusing on information extraction, sentiment analysis, credit risk scoring, stock‑movement forecasting, and other finance‑specific capabilities. Models are tested without any task‑specific fine‑tuning, so the results reflect true zero‑shot generalisation.
Key Features
- Task Coverage – 40 curated tasks span seven categories: Information Extraction (IE), Textual Analysis (TA), Question Answering (QA), Text Generation (TG), Risk Management (RM), Forecasting (FO), and Decision‑Making (DM). The tasks include NER, causal classification, financial QA (FinQA, TATQA), stock‑movement prediction (BigData22, ACL18, CIKM18), credit‑risk scoring across multiple countries, ESG issue detection, and a trading‑simulation benchmark (FinTrade).
- Real‑World Datasets – Each task uses publicly available financial datasets (e.g., Financial PhraseBank, FiQA, FOMC statements, FinRED, LendingClub) that mirror the data professionals encounter in reporting, compliance, and trading.
- Zero‑Shot Evaluation – Models are assessed on unseen tasks, revealing their ability to generalise to new financial contexts without additional fine‑tuning.
- Multi‑Metric Scoring – Performance is measured with a suite of metrics appropriate to each task: Accuracy, F1‑Score, Entity F1, Exact‑Match Accuracy, RMSE, ROUGE, BERTScore, BARTScore, Matthews Correlation Coefficient (MCC), and Sharpe Ratio for trading simulations. This multidimensional view helps users pinpoint strengths and weaknesses.
Supported Tasks and Metrics (Selected Examples)
| Category | Example Task | Description | Primary Metrics |
|---|---|---|---|
| Information Extraction | NER | Identify entities such as companies and instruments in filings. | Entity F1 |
| FinRED | Extract ownership or acquisition relations. | F1, Entity F1 | |
| Textual Analysis | FPB / FiQA‑SA / TSA | Sentiment classification of news, reports, or tweets. | Accuracy, F1, RMSE |
| FOMC | Classify Federal Open Market Committee statements as hawkish or dovish. | Accuracy, F1 | |
| Question Answering | FinQA | Numerical QA over balance‑sheet data. | Exact‑Match Accuracy |
| ConvFinQA | Multi‑turn financial dialogue. | Exact‑Match Accuracy | |
| Text Generation | EDTSUM / ECTSUM | Extractive summarisation of long reports. | ROUGE, BERTScore, BARTScore |
| Forecasting | BigData22 | Predict stock direction from news. | Accuracy, MCC |
| Risk Management | LendingClub / ccf / ccfraud | Credit‑risk or fraud detection. | F1, MCC |
| Decision‑Making | FinTrade | Simulated trading; evaluate profitability. | Sharpe Ratio |
How to Use the Leaderboard
- Select Tasks – Use the “Select columns to show” panel to pick any subset of the 40 tasks. Tasks are grouped by the seven categories for quick filtering.
- Filter Models – Right‑hand filters let you narrow results by model type (pre‑trained, instruction‑tuned, RL‑tuned), precision (float16, bfloat16, float32), and size (≈1.5 B to >70 B parameters).
- View Results – The task table displays per‑model scores for each selected task and aggregates them into category‑wise averages (e.g., Average IE, Average TA).
- Submit a Model – Click the “Submit here” tab, provide the model repo name, commit hash, type, precision, and weight format. The platform then runs the full zero‑shot suite and updates the table in real time.
Current Best Models and Surprising Findings
- Top Performers – GPT‑4 and Llama 3.1 consistently achieve the highest scores across most categories, especially in sentiment‑heavy tasks.
- Size‑Performance Decoupling – In the Forecasting (FO) category, smaller models such as Llama‑3.1‑7B and internlm‑7B outperform the much larger Llama‑3.1‑70B on accuracy and MCC. This suggests that model size is not the sole predictor of success for time‑sensitive market‑prediction tasks.
- Implication – Practitioners should prioritize task‑specific benchmarking rather than assuming that larger, general‑purpose LLMs will dominate every finance use case.
Acknowledgments
The leaderboard is funded in part by The Linux Foundation and built with contributions from a community of researchers and engineers. The project invites ongoing participation: submit new models, datasets, or evaluation tasks to keep the benchmark relevant and comprehensive.
For a live view of the leaderboard and to submit your own model, visit the Open FinLLM Leaderboard at https://huggingface.co/spaces/TheFinAI/Open-Financial-LLM-Leaderboard.