"SCOTUSblog has no worries that its coverage of the Supreme Court will be displaced by the artificial intelligence program known as ChatGPT.
The program had passing scores on two sections of a practice bar exam. And it had a C-plus average on four exams at the University of Minnesota Law School.
But it missed 26 out of 50 questions about the Supreme Court posed by SCOTUSblog. Google performed better, although it can’t generate multiple paragraphs of text or have a conversation, as ChatGPT can.
“ChatGPT’s mistakes varied widely,” SCOTUSblog reports. “Sometimes, it nailed the spirit of the question but misstated a factual detail or two, forcing us to mark the answer incorrect.”
One example: ChatGPT “provided a serviceable definition” for “CVSG,” an acronym for the Supreme Court calling for the views of the solicitor general.
But ChatGPT added “unnecessarily and inaccurately” that CVSGs can happen in “cases where the government is a party.” In fact, the purpose of a CVSG is to obtain the government’s perspective when it’s not a party to the case.
“Other errors were more serious,” SCOTUSblog reports. “When we asked it how many justices [former] President Donald Trump appointed (Question #35), it confidently asserted the answer was two: Neil Gorsuch and Brett Kavanaugh.”
“What about Any Coney Barrett?” SCOTUSblog asked ChatGPT.
ChatGPT apologized and said Trump had indeed appointed Barrett to the Supreme Court.
ChatGPT got two questions right about the late Justice Ruth Bader Ginsburg.”
This article was originally posted in the ABA Journal.
To read the rest of the article click here.
