Phosphor AI Tutor: Achieving 0.71-1.30 SD Effect Size in Dartmouth Statistics Course

Integrating LLM-powered formative assessment directly into course readings can significantly improve student outcomes while maintaining high voluntary engagement. In a pilot deployment at Dartmouth College, the digital learning platform Phosphor was associated with an increase in final exam performance between 0.71 and 1.30 standard deviations (SD), depending on whether the analysis adjusted for prior exam scores.

The Efficacy of Constructed-Response Questions

Active generation of answers is a primary driver of learning gains. The study found that lesson-level dosage (the number of lessons completed) tracked exam performance only when quizzes included constructed-response questions (CRQ)—which are graded by Claude Sonnet 4.6 against instructor rubrics—rather than multiple-choice questions (MCQ).

When the platform transitioned to MCQ-only quizzes for Module 2, the positive correlation between lesson completion and exam performance vanished. This suggests that the "doer effect"—where completing practice questions integrated into readings yields higher learning impact than reading alone—is specifically amplified when students must actively construct their answers.

Impact on Exam Performance and Engagement

Phosphor was presented as an optional, ungraded alternative to traditional readings, yet it achieved adoption rates far exceeding typical reading compliance.

Learning Outcomes

• Full Engagement Gap: The gap between students with full engagement (24 lessons and 3 reviews) and zero engagement was 1.30 SD on the final exam.
• Adjusted Effect: When controlling for midterm performance to account for student motivation and prior ability, the gap remained significant at 0.71 SD.
• Module Reviews: Passing all three cumulative Module Reviews was the strongest single predictor of success, associated with a 7.1-point increase on the final exam (d = 0.66).

Engagement Metrics

• Adoption: 90.2% of enrolled students engaged with the platform at least once.
• Reading Compliance: Total reading compliance was estimated between 48% and 76%, a massive increase over the reported baseline of 10–15% for the course.
• Student Sentiment: 94% of surveyed students found the platform more engaging, and 97% reported better retention.

Platform Architecture and Design

Phosphor is designed to move AI from a "crutch" (external tools used to bypass work) to a "learning aid" by embedding it into the content delivery system. The platform consists of three core components:

1. Lesson Quizzes: A mix of MCQ and CRQ. CRQs are graded by an LLM using a prompt that includes the question, a model answer, and explicit rubric criteria.
2. Module Reviews: Cumulative, interleaved quizzes covering multiple lessons with a 90% pass threshold. Data shows students often used these for spaced repetition, with 55% of retries occurring a day or more after the first attempt.
3. RAG-based Chat Assistant: A retrieval-augmented generation sidebar for course-specific queries. Interestingly, this feature was underutilized, with only 72 total queries, as students found general-purpose LLMs faster or the reference content sufficient.

Critical Analysis and Limitations

While the results are promising, the study is observational and lacks randomized controls, introducing several caveats:

• Selection Bias: Students who engage more with the platform may be more motivated or higher-performing regardless of the tool. The author acknowledges that the 0.71 SD figure is a conservative lower bound because midterm controls may absorb learning already produced by Phosphor.
• Novelty Effect: Some critics suggest the high initial engagement may be attributed to the Hawthorne effect or the novelty of a new digital tool.
• Content Overlap: There is a question of whether the exam questions overlapped directly with the Phosphor materials, which would measure reading compliance rather than general learning efficacy.

Community Perspectives

Discussion among technical observers highlights a tension between the tool's design and its labeling. Some argue that Phosphor is less an "AI tutor" and more a "practice quiz platform with an AI autograder," noting that the most effective part of the system was the assessment, not the conversational AI.

"The conclusion is essentially that people who do practice quizzes will do better on exams." — @wxw

Others view this as a stepping stone toward "Aristocratic Tutoring," where AI provides the 1-on-1 mastery learning previously only available to the elite, potentially bridging the gap described in Bloom's Two Sigma Problem.