Stanford Law Study: AI Outperforms Law Professors in Legal Tutoring

AI-Generated Answers Preferred Over Human Peer Responses

A study led by Professor Julian Nyarko of Stanford Law School reveals that law professors overwhelmingly prefer AI-generated answers to student questions over responses written by fellow instructors. In a blind evaluation of nearly 3,000 anonymized comparisons, AI won 75% of head-to-head matchups, suggesting that large language models (LLMs) can effectively serve as tutors for complex subjects like contract law.

This finding is significant because legal reasoning requires nuanced judgment and the ability to navigate ambiguity, rather than simple factual recall. According to the study, professors flagged AI responses as "pedagogically harmful" only 3.5% of the time, compared to 12% for peer-written answers.

Methodology and Scope of the Study

The research team, which included colleagues from Yale, NYU, and the University of Chicago, focused on the potential for AI to act as a tutor for law students. The study utilized 16 law professors who created 40 representative contract law questions typical of student inquiries during office hours. These professors then evaluated responses without knowing whether they were generated by AI or a human peer.

To ensure validity, the researchers calibrated AI responses to match the length and structure of human answers. The study tested various models, including Google's NotebookLM and Gemini 2.5 Pro, finding that NotebookLM—which utilized added legal resources—was considered slightly better by evaluators.

Implications for Legal Education

While the results suggest AI can meet the professional standards legal educators use to evaluate arguments, the researchers emphasize that this does not advocate for the wholesale replacement of human instructors. Professor Nyarko cautioned that the conversation should shift from whether AI can provide high-quality responses to how these tools can be deployed responsibly to benefit students.

Alejandro Salinas, first author of the study, noted that AI tutors can offer high-quality, on-demand support that complements classroom instruction and potentially broadens access to expert guidance in judgment-rich fields.

Critical Analysis and Counterpoints

Despite the positive findings, the study faced criticism from the technical and legal communities regarding its methodology and conclusions:

Statistical and Methodological Concerns

Critics argued that the sample size of 16 professors is too small to provide meaningful statistical power. Some observers noted that the results may be skewed by specific factors:

  • Answer Length Bias: Some analysts suggest that answer length was a strong predictor of win rate, and that professors, instructed to be concise, may have produced shorter, less comprehensive answers than the AI.
  • Model Selection: There are concerns that the results primarily featured Google models, potentially introducing bias.
  • Training Data Overlap: Some suggest that the AI may have been trained on the very textbooks used to generate the questions, making the task one of recall rather than reasoning.

Practical and Ethical Risks

Discussion among practitioners highlighted the difference between academic tutoring and professional legal practice:

  • Accountability: A recurring concern is the issue of liability. If an AI provides incorrect legal advice, there is no clear framework for who is held responsible.
  • Reasoning vs. Hallucination: Critics pointed out that LLMs cannot explain the underlying grounds for their statements when cross-examined and may hallucinate sources or rely on obsolete regulations.
  • Academic vs. Practice: Some practitioners argued that law professors' preferences reflect academic and theoretical reasoning, which differs significantly from the demands of private legal practice.

"The legal academy is supposed to have outlying opinions on things and present novel philosophical answers to questions... it doesn't reveal much new information."

Potential for Accessibility

Conversely, some view these developments as a way to democratize legal knowledge. By lowering the cost of legal training and providing a "first-pass" tutoring tool, AI could make the rule of law more accessible to those who cannot afford expensive legal counsel.

Sources