New Scholarship: We’re Obsolete!
A madman ran into the marketplace shouting that law teaching was dead and that we had killed it. But how did we do this? How could we drink up the sea? Who gave us the sponge to wipe away the entire horizon? Apparently, we did it because we invented large language models. Or that’s one possible implication of this scholarship on SSRN:
Here’s the abstract:
Large language models (LLMs) are increasingly promoted as educational tutors, yet most evaluations focus on domains with a single ground truth. Many disciplines, however, hinge on judgment: reasoning, weighing ambiguity, and reaching defensible conclusions. Law provides a sharp test. We conducted a blinded evaluation of short-answer tutoring in contracts courses with sixteen U.S. law professors. Participants created 40 representative questions, wrote answers, and judged 2,918 anonymized comparisons between human and LLM responses. Professors rated LLMs far higher than their peers (average win rate = 75.33%), with models performing similarly to the best instructor. LLM responses were also rarely flagged as harmful (3.53% vs 12.06% for professors). Preferences for LLM answers were consistent across evaluators and reflected shared professional standards. Our evaluation can be reliably extended to additional models by employing a separate LLM as a judge, rendering expert agreement an effective, scalable method to evaluate AI tutors in judgment-rich domains.
Take heart. This tremendous event has not yet reached the ears of the professoriate. The light of the stars requires time; deeds, though done, still require time to be seen and heard. This deed is still more distant from us than most distant stars.
But beware. The madman is seeking entry into our lecture halls and casebooks, insinuating his ideas into our thoughts through the Social Science Research Network. He seeks entry to our classrooms. “What are these lecture halls now,” he implores, “if they are not the tombs and sepulchers of legal education?”