(PO127) Evaluating the Accuracy of ChatGPT in Endodontic Diagnosis

Abstract: Chatbot Generative Pre-Trained Transformer (ChatGPT) is a large language model (LLM) developed by OpenAI. It is an advanced artificial intelligence (AI) language model that understands and generates human-like text, enabling diverse and complex interactions across various topics. We conducted a cross-sectional assessment of ChatGPT's endodontic diagnosis accuracy using the ChatGPT-4 September 25 version. Five clinical scenarios comprised of a diagnostic result table, a set of radiographs, and the patient's chief complaint were prompted into ChatGPT twenty times per case to assess accuracy. ChatGPT was explicitly prompted to provide the shortest answer possible using the AAE diagnostic terminology. In addition, we compared ChatGPT-4 accuracy to five different groups (D2, D3, and D4, postgraduate endodontic students (PG), general dentists (GD), and endodontists) from a previous similar project. A diagnostic score was generated and expressed as the percentage of cases answered correctly. The overall diagnostic accuracy of ChatGPT was 20%. The accuracy for pulpal diagnosis alone was 47%, and 43% for periapical diagnosis alone. One interesting finding was that ChatGPT used the diagnosis of "irreversible" pulpitis, knowing that it is not a diagnosis recognized by the AAE. Pairwise comparisons using t-tests with Bonferroni correction were performed, and ChatGPT-4's diagnosis accuracy was comparable to D2 and D3 students; however, it was outperformed by all the other groups (p < .05). These data suggest that even though ChatGPT identifies the accurate endodontic diagnosis at a rate similar to D2 and D3 students, it is not yet a reliable source of information for patients and clinicians.