The rapid growth in computing power and the vast amount of data have led to significant advancements in artificial intelligence (AI) in recent years, enabling its integration into various aspects of daily life.

The rapid growth in computing power and data availability has accelerated the development of artificial intelligence (AI), enabling its integration into diverse aspects of life (1). In 1943, McCulloch and Pitts introduced the first artificial neuron model, regarded as the foundation of AI. Later, Alan Turing proposed the Turing Test to determine whether machines could exhibit human-like reasoning. In 1956, John McCarthy coined the term artificial intelligence and developed LISP, the first AI programming language. After a period of limited progress, advances in computing power and algorithmic design reignited interest in artificial intelligence, marking the beginning of the deep learning era that enabled the development of large language models (2).

The Generative Pre-trained Transformer (GPT) model, introduced by OpenAI in 2018, is a deep learning–based system trained on massive text datasets to generate human-like responses. The ChatGPT interface, launched in 2022, made this technology accessible to the general public. The latest versions, such as GPT-4 and GPT-5, demonstrate enhanced accuracy and reasoning ability (3).

In healthcare, AI applications now contribute to diagnosis, image interpretation, and patient management with increased accuracy and reduced workload for clinicians. In education, AI-based learning systems are being tested for their potential to improve comprehension and self-directed learning (4–6).

In this study, questions from the Residency Training and Progress Examination (UEGS) and the Competency Examination, organized by the Turkish Society of Orthopedics and Traumatology (TOTBID) across various years, were presented to the ChatGPT-3.5 model. The results were compared with participant outcomes to evaluate whether ChatGPT performs better than resident physicians in the UEGS and whether it can achieve a passing score in the Competency Examination.

The findings aim to clarify whether ChatGPT can serve as a supplementary educational tool in residency training.

All questions from the UEGS and Competency Exams were presented to the ChatGPT-3.5 model (OpenAI, web interface; tested in March 2023). The free version of ChatGPT-3.5 web interface, which represents the most widely used configuration during the study period, was employed. The model was instructed to answer in Turkish to ensure linguistic compatibility with the original exam format.

Example of standardized prompt used for Competency Exam: “You are an orthopedic resident preparing for the national examination. Read the following question carefully and select the most appropriate answer among the options. Then, provide a brief (one-sentence) explanation for your choice. Answer in Turkish.”

The same standardized prompt structure and wording were used consistently for all questions to ensure reproducibility. No feedback or corrections were provided to the model during data entry.

For the UEGS, ChatGPT was prompted to respond to each item as either correct or incorrect with a brief explanation. Each interaction was designed following the Role–Goals–Context (RGC) Framework, which defines the AI’s role (exam participant), objective (select the most accurate answer), and contextual boundaries (question content and available options) [9].

To allow domain-specific performance analysis, all questions were classified into subcategories representing the major divisions of orthopedics and traumatology.

For the UEGS, nine subcategories were defined: general orthopedics, trauma, pediatric orthopedics, spinal surgery, hand and upper extremity surgery, foot and ankle surgery, sports trauma and knee arthroscopy, orthopedic oncology, and adult reconstructive surgery

For the Competency Exam, ten subcategories were used: basic sciences, pediatric orthopedics, pediatric trauma, adult trauma, upper extremity and hand surgery, lower extremity and foot surgery, arthroscopic and sports surgery, adult reconstructive surgery and arthroplasty, spinal surgery, and infections and tumors (Table 1).

Because ChatGPT-3.5 does not support image interpretation, the model was unable to answer questions containing radiographs, clinical photographs, or other visual material. In total, 56 such questions were skipped: 7 (2023), 8 (2022), 17 (2021), 5 (2020), 5 (2019), 4 (2018), 1 (2017), 5 (2016), 3 (2015), and 1 (2014). These were excluded from accuracy calculations.

All remaining UEGS questions (n = 2000) and non-visual Competency Exam questions (n = 944) received valid text-based responses.

This study examined the performance of the ChatGPT-3.5 artificial intelligence model in Türkiye’s Residency Training and Progress Examination (UEGS) and Competency Examination, both organized by the Turkish Society of Orthopedics and Traumatology (TOTBID).

ChatGPT achieved its highest accuracy in the General Orthopedics category (61%) and the lowest in the Adult Reconstructive Surgery category (39%). The model significantly outperformed participants only in the Spine Surgery category. This finding may be explained by the fact that Spine Surgery was also the category in which resident physicians had the lowest accuracy (7.98%). These results may provide insights for those involved in planning and improving orthopedic and traumatology residency training programs, as there is already a core curriculum committee established for Spine Surgery (11).

When comparing the results of ChatGPT and residents in the UEGS, no significant difference was found in the total number of correct answers. However, ChatGPT gave more incorrect responses in every category compared with the participants’ average. Based on these data, ChatGPT-3.5 performed worse than orthopedic and traumatology residents in the UEGS.

In the Competency Examination, ChatGPT passed four out of ten exams by correctly answering at least 60 questions out of 100. In the remaining six exams, the model failed to reach the required passing score. Among the 1,000 questions analyzed, ChatGPT-3.5 was unable to answer 56 image-based questions due to its lack of visual interpretation capability. With newer versions such as ChatGPT-4, which can interpret images, more questions could be answered, potentially resulting in higher success rates. In a comparative study using UEGS questions, ChatGPT-4 demonstrated significantly higher accuracy than ChatGPT-3.5 (12). Likewise, a recent study using questions from the Turkish Competency Exam reported that ChatGPT-4o achieved higher accuracy than human participants across all orthopedic subdomains (13). This indicates that our findings align with the overall trend of improving model performance in newer AI systems rather than contradicting it.

Our findings show that ChatGPT-3.5 performs worse than resident physicians in the UEGS and fails to pass most of the Competency Exams in orthopedics and traumatology. Similar studies have demonstrated that ChatGPT performs well in some exams but poorly in others (14–19). In certain cases, ChatGPT outperformed participants, whereas in others, as in our study, participants achieved better results.

ChatGPT became one of the fastest-growing computer programs in history, reaching 100 million active users within two months of its public release (20). Like other AI models, it draws from a wide range of data sources, including peer-reviewed journal articles, textbooks, and online content (21). As new versions are released, both the technical capabilities of the model and the size of its knowledge base expand. Thus, it is expected that future AI models will continue to improve their ability to evaluate and answer questions.

This study has certain limitations. The results of residents and specialists were evaluated using publicly available data from the TOTBID website, where not all exam years had standardized or complete analyses. Some exams were not conducted during the COVID-19 pandemic, and those years were therefore excluded. These limitations may have affected the comparison process.

Finally, the use of chatbots in medical education is an emerging trend supported by many educators and medical professionals. OpenAI’s ChatGPT offers several potential advantages for both students and teachers (22, 23). Recent reviews have highlighted that generative AI tools hold promise for orthopedic education and training but also pose challenges related to reliability, ethical use, and integration into curricula. It is important to remember that these systems are still evolving and have not yet reached perfection. There remain significant gaps in both theoretical and practical orthopedic education that AI tools cannot yet fill (24).

ChatGPT-3.5 showed variable accuracy in orthopedic examinations, performing comparably to residents only in the Spine Surgery category while underperforming in most others. Although ChatGPT offers potential educational benefits, it is not yet a reliable or valid resource for independent use in orthopedics and traumatology education.

As artificial intelligence systems continue to evolve, future versions with multimodal capabilities may become more effective tools in medical learning and assessment.