Missouri In House Counsel Legal news, updates and analysis
Artificial intelligence has been a part of the law for years, infiltrating practice areas like a rising tide. The next phase may well be a tsunami.
AI has now reached a point where it can, in a matter of seconds, produce text that is often indistinguishable from (or better than) what a human might write. It can answer questions so well that it’s hard to remember there isn’t a person on the other end. It can pass the bar exam.
In short, language — the core of the lawyer’s craft, the thing that practices as divergent as contracts and litigation and appellate law share — is now in the hands of the machines.
This spells disruption to the field of law, on a scale that will rival if not exceed the introduction of the internet. That legal research you once assigned to clerks or associates to be completed in a matter of days? Available almost instantaneously at the cost of a written sentence. That outline you painstakingly sketched out? A few seconds of work. That brief or letter or motion for which you once billed handsomely? A few well-formed queries, and there it is — perfectly serviceable and possibly better than what you might have authored.
Lawyering isn’t over. The technology, as amazing as it is, needs a hand to guide it. Calculators don’t know what numbers they are supposed to multiply; spreadsheets don’t know which column is most important. AI isn’t a substitute for a competent attorney.
But if you plan to enter a world with calculators armed with long division on a pad of paper, get ready for a rough time. And if the competition seems tough now, wait six months.
Joe Regalia
“It’s still very much a tool,” said Joe Regalia, a law professor at the University of Nevada, Las Vegas who focuses on legal writing and technology and innovation. “It’s just the most effective freaking cool tool I’ve ever seen in my life.”
AI, of course, has already been incorporated into the legal profession in areas such as document review and e-discovery. The AI programs making headlines these days are known as large language models. What are they? Let’s ask ChatGPT:
“A large language model is a computer program that has been trained on a vast amount of text data to understand the rules and patterns of language,” the program wrote when asked to explain the concept in plain English. “This enables it to perform various language-related tasks, such as answering questions, generating text, and translating languages.”
The best known such programs are from the company OpenAI, which has produced a series of Generative Pre-trained Transformer models, or GPT. Last November, OpenAI released a publicly accessible chat interface known as ChatGPT, giving everyday users a chance to try out the technology. Even as that program’s considerable capabilities were sinking in, the company released a more powerful version, GPT-4, on March 14.
Perhaps no AI accomplishment has caught attorneys’ attention more than GPT-4’s ability to pass the Uniform Bar Exam that is administered to prospective lawyers in most states, including Missouri. In a paper released shortly after GPT-4’s release, a team of researchers showed that the program had taken both the multiple-choice question and the essay portions of the test and gotten a composite score of 297.
That puts it in the 90th percentile of test-takers and would count as a passing score in every jurisdiction that uses the UBE. In comparison, the average passing grade in Missouri’s July 2022 exam was 280. The minimum passing threshold in Missouri is 260.
Of course, no one is planning to issue a law license to a computer, and no potential lawyer could use GPT to cheat his or her way to a passing score on the tightly controlled exam. But it does mean that AI technology is progressing at an astounding rate. Passage of the bar exam, which had seemed an insurmountable obstacle just a short time ago, has now been achieved with flying colors.
Daniel Martin Katz
“If I went back a year and said we’re going to have a computer pass the bar exam, what world do you think you’re living in?” Daniel Martin Katz, a law professor at Illinois Tech’s Chicago-Kent College of Law and one of the authors of the bar-exam paper, said in an interview. “I’d say a different world is the answer.”
GPT-4’s performance on the bar exam exceeded that of all prior versions of the program. Katz said versions of the program introduced before 2020 couldn’t navigate even the multiple-choice component of the test, or else scored no better than chance. According to a paper he helped write late last year, ChatGPT got nearly half of the questions right but wasn’t able to navigate the exam’s essays.
In other words, AI is progressing more rapidly than most busy lawyers have time to process.
“All of the gains have happened in the last 24 months, mostly in the last 12 months,” Katz said. “So, if people have a view of what these technologies are capable of that they locked into two or three years ago, it would have been totally correct then. And it’s not correct now, in my view.”
So, what will this mean for lawyers? No one can say exactly where AI is headed or what kinds of disruptions it will bring. But a program that can produce in a matter of seconds what a human lawyer would need several hours’ or days’ worth of billable hours to match can’t help but shake up the practice of law. And, as Katz noted, the programs’ current capabilities are a floor not a ceiling; they will only improve over time.
Jonathan Sternberg
Jonathan Sternberg, an appellate lawyer in Kansas City, said he hasn’t used AI for any of his actual cases, but he has played with several existing programs by asking questions to which he already knew the answers to see how well it did. He also assigned them the kinds of legal tasks he might give to a law clerk, such as finding examples of certain kinds of Missouri cases.
Sternberg said the results were generally disappointing, often missing cases he knew existed. But given the advancements in recent months, the day may come when he no longer needs to hire law clerks to do legal research.
“Which is unfortunate, because my clerks are 2Ls and 3Ls at UMKC and KU, and they learn from this,” he said. “But why would I need them?”
Sternberg added that those concerns extend to his own standing as a full-time appellate lawyer.
“I don’t think I’m to the point where I would write briefs with it, but what began to scare me was, if this really had slightly more intuition and full access to all the legal research that I do, why would you need me?”
Those concerns, while real, probably aren’t immediate — particularly in a conservative industry like the law that resists rapid change. And while GPT might be able to craft a brief or a motion, it’s not going to be able to interview a witness or try a case. Like many time-saving tools throughout history, AI could render some legal tasks obsolete but free lawyers to concentrate on more specialized tasks.
There is also the problem of accuracy. The public version of ChatGPT is known to “hallucinate” occasionally when asked questions, sometimes offering summaries that sound convincing but are dead wrong.
Jayne Woods, an associate teaching professor of law at University of Missouri School of Law, experimented with ChatGPT to create outlines of legal arguments and to generate potential questions that a court might ask during an appellate hearing. In a guest post on the Appellate Advocacy Blog, she detailed the results, which overall she found to be quite good and useful.
Jayne Woods
However, she also noted that one of the crisp case summaries ChatGPT wrote was incorrect: The program insisted that the plaintiff in a First Amendment case had prevailed when in fact it lost because the court said it failed to establish viewpoint discrimination.
Woods noted that the case was an outlier in First Amendment law, which may explain why the program struggled with it. In an interview, she said that even such mistakes can be useful for students.
“They should know these cases before they ask it to summarize them, so they should be able to recognize when it gives them an incorrect summary, and they should be checking them, which I think it good practice for any lawyer preparing for oral argument,” she said.
Woods said crafting effective prompts for AI programs is likely to become an important skill for lawyers to develop.
“I think we’re also going to have to focus a lot more on the critical thinking and revision aspect of writing,” she said. “If you can have AI draft a motion to dismiss, it can draft the motion, but then your job as the lawyer is going to be reviewing that motion with a critical eye. We’re almost going to elevate everybody kind of to the level of partner now, reviewing a lower associate’s work.”
“It’s still very much a tool. It’s just the most effective freaking cool tool I’ve ever seen in my life.”
— Joe Regalia, a law professor at the University of Nevada, Las Vegas
Regalia said the reason for such hallucinations is usually a lack of boundaries for the program to operate within. For example, he said, if you ask ChatGPT who wins a game of golf, the program might confidently, though falsely, report that it’s the player who gets the most points, which is how most other games are won. But if you first ask the program to recite the rules of golf and then ask it who wins, it will get the correct answer.
Regalia said AI programs are like “the smartest toddler in the world.”
“Just by giving it some context and some guidance about how you want it to think about this task, you can improve its accuracy drastically and a lot of these problems go away,” he said.
Users of the more advanced GPT-4 have reported fewer such issues, and later versions are sure to get even more accurate. And commercial products that help shape AI for certain tasks have already started to arrive. Microsoft, for instance, which is an investor in OpenAI, has launched its AI-powered Bing search engine and says it plans to add artificial intelligence features to such widely used programs as Outlook, PowerPoint, Excel and Word.
Specific to the legal community, the company Casetext recently unveiled CoCounsel, a GPT-4-powered program billed as “the world’s first reliable AI legal assistant.” In a webinar demonstration the day after GPT-4’s release, a company representative walked viewers through its legal research, document review and contract analysis abilities.
CoCounsel uses the company’s proprietary legal database as well as specific documents the lawyer uploads, which allows its results to be easily reviewed for accuracy. During the demonstration, the program reviewed a sample of emails from the Enron accounting scandal. The program was able to discern passages that showed evidence of concealment, even if the word “conceal” or a similar term didn’t appear. It also was able to dig up a joke that an Enron employee had written.
“We think this might be the most important legal technology advancement so far,” Casetext’s CEO, Jake Heller, said during the presentation.
How quickly lawyers and law firms adopt such tools remains to be seen. Katz described AI as a “Rorschach test” for lawyers: Some are pessimistic about its flaws; others sense the opportunities it presents.
“You can be the Henry Ford of something like this, where you can do a lot more work than you used to be able to so in a given unit of time because you have a capability like this,” he said.
As a writing teacher, Regalia wonders what will happen when the act of stringing together words and sentences becomes obsolete. But that prospect is outweighed, he said, by its potential to increase access to justice by allowing, say, a public defender or civil rights lawyer to spend more time building cases and less time writing motions.
He also noted that economic forces are likely to punish inefficient firms that insist on doing things the old-fashioned way.
“Twenty years ago, it was still largely a black box with clients, and the market wasn’t nearly as competitive as it is now,” he said. “These days, if the next competent firm can do the same motion, the same work for a third of the price because it takes them a third of the time, you can’t compete with that.”
In the short run, at least, AI appears to be poised to augment, rather than replace, lawyers.
“We’re still far from the time where you just hand stuff over to AI, for lots of reasons: It’s biased, it still will have accuracy problems, etc.” Regalia said. “But humans plus AI — or in our context, lawyers plus AI — is incredible.”
