by Elana Gotkine

The natural language processing model Chat Generative Pretrained Transformer (ChatGPT) does not pass the American College of Gastroenterology (ACG) Self-Assessment Tests, according to a study published online May 22 in the American Journal of Gastroenterology.

Kelly Suchman, M.D., from the Zucker School of Medicine at Hofstra/Northwell and the Long Island Jewish Medical Center in New Hyde Park, New York, and colleagues assessed ChatGPT-3 and ChatGPT-4 performance on the 2022 and 2021 ACG Self-Assessment Tests. A 70% score or higher was required to pass the assessment.

The researchers found that ChatGPT-3 and ChatGPT-4 scored 65.1% and 62.4%, respectively, on 455 included questions across the two exams.

"Recently, there has been a lot of attention on ChatGPT and the use of artificial intelligence across various industries. When it comes to medical education, there is a lack of research around this potential ground-breaking tool," a co-author said in a statement. "Based on our research, ChatGPT should not be used for medical education in gastroenterology at this time and has a way to go before it should be implemented into the health care field."

More information: Kelly Suchman et al, ChatGPT Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test, American Journal of Gastroenterology (2023). DOI: 10.14309/ajg.0000000000002320

Journal information: American Journal of Gastroenterology 

Copyright © 2023 HealthDay. All rights reserved.