The rise of artificial intelligence (AI) has many attorneys questioning how the technology may upend their profession.
In December 2022, the most advanced large language model at the time, GPT-3.5, was tasked with taking the bar exam; it failed. Less than a year later, GPT-4 took the bar exam and passed with flying colors.
What is GPT-4?
ChatGPT is a language model created by OpenAI. The program has been trained on a large amount of data for purposes of understanding and generating human-like language. To put it another way, ChatGPT is a computer program designed to have conversations with humans, answer questions, and provide information.
OpenAI has been releasing GPT language models since 2018. The third version, GPT-3, was released in 2022. GPT-4 is the latest and most powerful version. At the time of publication, GPT-4 is only available to paid subscribers of ChatGPT+.
What is the Uniform Bar Exam?
The Uniform Bar Exam (UBE) is a standardized test used by many U.S. jurisdictions as part of their bar admission process. The UBE consists of three parts:
- The Multistate Bar Examination (MBE): a multiple-choice test that assesses knowledge of common law principles in areas such as contracts, torts, and property.
- The Multistate Essay Examination (MEE): a series of essay questions that test a candidate’s ability to apply legal principles to specific fact patterns.
- The Multistate Performance Test (MPT): a set of tasks that assess a candidate’s ability to complete legal tasks such as drafting a memorandum or client letter.
The UBE is designed to allow attorneys to be licensed in multiple states without having to retake the bar exam for each state. Many states have adopted the UBE or a variation of it, and each jurisdiction sets its own passing score. However, UBE scores can be transferred to other UBE jurisdictions, subject to each jurisdiction’s rules and requirements.
Can GPT-4 pass the Uniform Bar Exam?
OpenAI used GPT-4 to take a number of exams designed for humans. The exams were sourced from publicly-available materials and included both multiple-choice and free-response questions. Among the exams taken by GPT-4 was the Uniform Bar Exam (UBE).
GPT-4 didn’t just pass the UBE; it passed the exam with flying colors. GPT-4 received a score in the top 10 percent of test takers, meaning it scored better than 90 percent of lawyers. For good measure, GPT-4 also passed the Law School Admissions Test (LSAT).
Here’s a look at GPT-4’s performance on a host of notable academic and professional exams:
| Exam | Score | Percentile |
|---|---|---|
| Uniform Bar Exam | 298-400 | 90th |
| LSAT | 163 | 88th |
| SAT Evidence-Based Reading & Writing | 710/800 | 93rd |
| SAT Math | 700/800 | 93rd |
| Graduate Record Examination (GRE) Quantitative | 163/170 | 80th |
| Graduate Record Examination (GRE) Verbal | 169/170 | 99th |
| Graduate Record Examination (GRE) Writing | 4/6 | 54th |
| Medical Knowledge Self-Assessment Program (MKSAP) | 75% | 75th |
| AP Biology | 5 | 85th – 100th |
| AP English Language and Composition | 2 | 14th-44th |
| Source: GPT-4 Technical Report | ||
What does the rise of AI mean for the future of the legal profession?
You shouldn’t expect a computer to hang a shingle and start representing clients anytime soon, but AI will undoubtedly have an impact on the practice of law— probably sooner than you think.
Casetext, a legal research platform that provides access to a wide variety of legal information, recently unveiled CoCounsel.
Casetext’s CoCounsel is an AI legal assistant powered by GPT-4. CoCounsel does document review, legal research memos, deposition preparation, and contract analysis in minutes.
Before attorneys start leaving the legal field to become chefs or school bus drivers, it’s worth noting that ChatGPT has been flat-out wrong about a number of legal issues.
ChatGPT can be easily confused if the questions it’s asked are too complicated or too niche. What’s more, ChatGPT can’t deal with concepts that are too recent.
It’s best to think of ChatGPT as autocomplete on steroids. The model is trained using massive amounts of data obtained from books, webtexts, and other pieces of writing on the internet. Of course, anyone who uses the internet knows that the internet is, well, not always accurate. What’s more, we don’t know precisely what information ChatGPT is being fed.
The University of Washington linguist and prominent AI critic summed up the problem with ChatGPT as follows: “We generally don’t eat food whose ingredients we don’t know or can’t find out.”
Will ChatGPT ultimately destroy the legal profession? Here’s what ChatGPT said when I asked the program:
No, ChatGPT is not likely to destroy the legal profession. While AI technology is becoming increasingly advanced and capable of performing certain legal tasks, there are many complex legal matters that require the expertise and judgment of human lawyers.
ChatGPT and other AI tools can assist lawyers with tasks such as legal research, document analysis, and drafting basic legal documents. However, they cannot replace human lawyers when it comes to providing legal advice, advocating for clients in court, or handling complex legal issues that require strategic thinking and nuanced interpretation of the law.
In fact, AI technology can actually help lawyers be more efficient and effective in their work, allowing them to focus on higher-level tasks that require human intelligence and creativity.
Therefore, while AI technology may change the nature of the legal profession, it is not likely to eliminate the need for human lawyers altogether.
Convinced?