Study shows ChatGPT outperforms PubMedGPT on US medical exam with 60% score

ChatGPT has once again demonstrated its remarkable abilities, as a recent study conducted by Tiffany Kung and colleagues at AnsibleHealth in California showed that it could pass the United States Medical Licensing Exam (USMLE) with a score of around 60%. The USMLE is a highly standardised and regulated series of exams that are required for medical licensure in the US.

In the study, ChatGPT was tested on 350 of the 376 publicly available questions from the June 2022 USMLE release and received a score range of 52.4% to 75% across all exams. This result outperformed PubMedGPT, a rival model trained exclusively on biomedical literature, which scored 50.8% on an older set of USMLE-style questions.