OpenAI and Harvey Partner to Develop Custom Case Law Model
OpenAI and Harvey have partnered to develop a custom-trained case law model designed to handle complex reasoning and extensive domain knowledge for legal, tax, and finance professionals. This collaboration addresses the limitations of standard foundation models and retrieval-augmented generation (RAG) by injecting deep legal knowledge directly into the model's training.
Custom-Trained Case Law Model Capabilities
The custom-trained model allows Harvey to provide AI systems capable of performing tasks that exceed the capabilities of a single model call. Key capabilities include:
- Complex Document Analysis: Identifying material discrepancies across hundreds of contracts.
- Legal Drafting: Drafting documents and answering questions regarding complex litigation scenarios.
- Case Law Research: Providing thorough answers to client questions with citations for every sentence, ensuring that the model does not fabricate cases.
Technical Approach and Development
Harvey transitioned from using public APIs for fine-tuning and RAG systems to a custom-trained model because RAG alone was insufficient for the high-level expertise required by attorneys. While foundation models possessed strong reasoning capabilities, they lacked the specific domain knowledge necessary for professional legal work.
To solve this, Harvey and OpenAI collaborated to inject new knowledge and reasoning patterns into the base models. The development process involved:
- Data Integration: The team added the equivalent of 10 billion tokens of data, starting with Delaware case law and expanding to include all U.S. case law.
- Collaborative Research: Legal experts explained the nuances of case law research to researchers and OpenAI engineers to determine the most effective technical levers.
Performance and Results
According to Winston Weinberg, co-founder of Harvey, lawyers preferred the output of the case law model 97% of the time. The preference was primarily driven by the model's ability to provide longer, more complete answers that addressed the nuances of the question and covered more relevant case law.
One of the primary goals of the custom model was the reduction of hallucinations. The resulting model ensures that every sentence is supported by a cited case, eliminating the fabrication of legal precedents.
Future Directions for Legal AI
Harvey is currently exploring several expansions of the case law model's utility, including:
- Automated Drafting: Using the model to draft briefs and motions.
- Jurisdictional Analysis: Helping attorneys understand how case law varies across different jurisdictions.
- AI Agents: Developing agents that combine multiple model calls into a single output to reduce the need for manual prompt engineering and user typing.
Strategic Philosophy for AI Development
Co-founder Gabe Pereyra suggests that AI founders should build for the future trajectory of model capabilities rather than current limitations. He advises tackling complex versions of problems to ensure that the product remains relevant as models evolve and are not rendered obsolete by general model improvements.