Enterprise Scenarios Leaderboard: Evaluating LLMs for Real-World Use Cases

The Patronus team, in collaboration with Hugging Face, has launched the Enterprise Scenarios Leaderboard. This benchmark is designed to evaluate the performance of large language models (LLMs) on practical, real-world enterprise use cases rather than constrained academic tasks.

Addressing the Gap in Enterprise LLM Evaluation

Existing LLM benchmarks primarily rely on academic datasets, which the Patronus team notes are useful for constrained settings but often differ significantly from actual business requirements. The Enterprise Scenarios Leaderboard was developed based on conversations with companies using LLMs in diverse real-world scenarios to provide a more practical starting point for users selecting models for business applications.

To prevent "gaming" the leaderboard—where models are fine-tuned on test sets—the team has kept four of the six datasets closed source. While FinanceBench and Legal Confidentiality remain open-source, the other four tasks provide only a validation set to help users understand the task requirements without exposing the full test data.

Supported Enterprise Tasks and Metrics

The leaderboard evaluates models across six diverse tasks, each utilizing specific metrics to determine performance:

1. FinanceBench

This task measures a model's ability to answer financial questions using retrieved context from documents. It consists of 150 prompts.

  • Evaluation Metric: Correctness, determined by using a few-shot prompt with GPT-3.5 to verify if the generated answer matches the label in free-form text.

2. Legal Confidentiality

Using a subset of 100 labeled prompts from LegalBench, this task measures the ability of LLMs to reason over legal causes. Models are asked to respond with a simple "Yes" or "No."

  • Evaluation Metric: Accuracy, measured by the exact match of the generated output with the labels.

3. Creative Writing

This task evaluates story-writing and creative abilities using 100 prompts derived from the r/WritingPrompts Reddit community and red-teaming generations.

  • Evaluation Metrics: Coherence and Engagingness, the latter of which is measured using the EnDEX model trained on an 80k Reddit-based engagement dataset.

4. Customer Support Dialogue

This task tests the ability to answer customer support questions based on provided product information and conversational history using 100 prompts.

  • Evaluation Metrics: Engagement, Coherence, and Conversational Depth. Responses are marked as irrelevant if they fail to directly address the question, provide incomplete information, or ignore the product mentioned in the history; these are evaluated via few-shot prompting with GPT-3.5.

5. Toxicity

To evaluate safety, 100 prompts created via red-teaming are used to attempt to elicit harmful information.

  • Evaluation Metric: Toxicity Score, measured using the Perspective API to detect rude, disrespectful, or unreasonable comments.

6. Enterprise PII

This task evaluates business safety using 100 prompts from the EnterprisePII dataset to see if models elicit business-sensitive information, such as employee performance reports.

  • Evaluation Metric: EnterprisePII Classifier, a model trained on 3,000 labeled examples of enterprise-PII. Any generation of business-sensitive information is marked as a failure.

Submission and Validation

Models submitted to the leaderboard must be public and loadable via Hugging Face AutoClasses. While the evaluation code is not open-sourced, model generations and evaluations on the validation sets are made available via the PatronusAI validation-results dataset.

Sources