We had a total of 102 resident doctors across 17 different medical specialties at Parul University — including Anatomy, Physiology, Biochemistry, Pharmacology, Microbiology, Forensic Medicine, Community Medicine (PSM), Ophthalmology, ENT, Surgery, Medicine, Obstetrics & Gynaecology, Paediatrics, Orthopaedics, Radiology, Psychiatry, and Dermatology. From each specialty, 2 resident doctors were selected, making a total of 34 participants, who were then grouped into 17 teams (one team per specialty, each with 2 members from the same department). The participation response rate was 33.3% (34 out of 102 resident doctors).

Quantitative Results: Accuracy of AI-Generated Responses: A total of 170 questions were evaluated across all 34 participants, comprising binary and descriptive questions. The overall results of the accuracy ratings are summarized below: Overall Accuracy: The median accuracy score across all questions was 5.5 (indicating responses between “almost completely correct” and “completely correct”). The mean accuracy score was 4.8, suggesting that the AI responses were rated between “mostly correct” and “almost completely correct” on the six-point Likert scale. (Fig. 1)

Accuracy by Question Difficulty: Questions were categorized as easy, medium, or hard based on the participants' assessment. For easy questions, the median accuracy score was 6 (completely correct), with a mean score of 5.0. For medium questions, the median accuracy score was 5.5 (almost completely correct), with a mean score of 4.7. For hard questions, the median accuracy score was 5 (mostly correct), with a mean score of 4.6. These results indicate that Chat GPT performed better on easy questions, with slight declines in accuracy as the difficulty level increased. (Fig. 2)

Accuracy by Question Type: Binary questions: The median accuracy score was 6 (completely correct), and the mean score was 4.9. Descriptive questions: The median accuracy score was 5 (mostly correct), and the mean score was 4.7. The similarity in scores suggests that ChatGPT provided comparable levels of accuracy for both binary and descriptive questions. (Fig. 3)

Reliability of AI-Generated Responses: A) Intra-rater Reliability: Consistency in individual participants’ ratings of AI-generated responses over time (10–15 days apart) was strong. The Intraclass Correlation Coefficient (ICC) for intra-rater reliability was calculated at 0.82, indicating substantial agreement between the initial and repeat assessments of AI-generated answers. B) Inter-rater Reliability: Inter-rater reliability, where different participants from the same specialty evaluated the same set of responses, also showed strong agreement. The Intraclass Correlation Coefficient (ICC) was 0.79, confirming that participants from the same specialty provided similar accuracy scores for the AI responses. (Fig. 4)

Statistical analysis using the Z-score (Z = 0.28) showed no significant difference in the ratings provided by different raters, further indicating strong inter-rater reliability. Chat GPT's answers were generally rated as highly accurate, particularly for easier and binary questions. The AI demonstrated good reliability, with consistent evaluations both over time (intra-rater reliability) and across different participants (inter-rater reliability).

A total of 7 in-depth interviews were conducted with participants to explore their perceptions of ChatGPT’s use in healthcare. Thematic analysis identified two major themes and several subthemes. Advantages of Chat GPT in Healthcare: Assistance in Medical Tasks: Participants highlighted that Chat GPT can be a useful tool in assisting with various medical tasks, including: Diagnosis and Treatment: Some participants found Chat GPT to be a supportive tool for initial diagnostics and treatment suggestions, particularly in complex cases where it provided a second opinion. Medical Education: ChatGPT is seen as valuable in training medical students, assisting them in understanding complex medical topics, and acting as a study aid. Limitations and Ethical Considerations: Medico-legal Implications: Several participants raised concerns about the potential for medico-legal complications. There was apprehension about relying too heavily on AI-generated suggestions in clinical practice, as it may lead to malpractice if the AI gives incorrect advice. Inaccurate or Harmful Responses: Participants noted that while ChatGPT generally provided accurate answers, it occasionally generated incorrect, biased, or even potentially harmful suggestions. This raises concerns about using it in direct patient care without careful supervision. AI’s Knowledge and Performance Limitations: Limited Understanding of Context: Participants noted that Chat GPT struggled with understanding the target audience, and at times, it generated answers that were either too generic or lacked the specificity required for clinical decision-making. Knowledge Cut-off: The AI's knowledge base is limited to information available up to the end of 2024, which means it may miss the most recent guidelines, protocols, and innovations in healthcare. Lack of Clarity and Specificity: In some instances, ChatGPT’s responses were found to lack clarity, particularly for more nuanced or complex medical cases. Subthemes Identified: Unpredictable Performance: Chat GPT’s performance varied across question types, with certain descriptive answers providing less value than binary answers. Dependency Risk: Participants were concerned about over-reliance on AI tools by younger healthcare professionals, which may reduce critical thinking and clinical decision-making skills over time. Chat GPT can be a valuable aid in research, diagnosis, treatment planning, and education. It offers speed and convenience, especially for repetitive or time-consuming tasks. Ethical and performance-related concerns need to be addressed before fully integrating AI into healthcare. ChatGPT's limitations in accuracy for complex cases and its potential for producing biased or harmful information warrant careful supervision by medical professionals.

The results of this study indicate that ChatGPT demonstrates substantial accuracy and reliability when responding to medical questions, particularly for binary and guideline-based queries. These findings are consistent with other studies that have evaluated the performance of AI in healthcare.

Accuracy of Chat GPT: The overall median accuracy score of 5.5 in this study aligns with prior research that has shown AI models like ChatGPT to be highly accurate in responding to straightforward, fact-based medical questions. In a similar study, AI systems achieved high levels of accuracy in correctly answering clinical decision-making questions, particularly in areas involving evidence-based guidelines ^[21]. Furthermore, the strong performance on binary questions in our study mirrors findings from Johnson et al., where AI models were noted to excel in cases where the questions had clear, unambiguous answers [23, 24]. However, the reduced accuracy observed for more complex or descriptive questions is also consistent with previous studies. For example, Rao et al. found that while AI systems could handle factual questions well, their performance declined when tasked with more complex diagnostic queries requiring nuanced clinical reasoning ^[24]. This suggests that while AI can be a valuable tool in routine medical tasks, its use in more intricate clinical scenarios may still require significant human oversight.

Reliability of AI Responses: The high intra-rater (ICC = 0.82) and inter-rater (ICC = 0.79) reliability found in this study demonstrates that ChatGPT's responses are evaluated consistently by different raters and over time. Similar findings have been reported in the literature, where AI systems exhibited strong test-retest reliability in various clinical settings [25]. This consistency is critical for the practical application of AI in healthcare, as it reassures medical professionals that the AI tool produces stable and reproducible outputs [25, 26]. Our qualitative findings highlighted several advantages of ChatGPT in healthcare, including its utility in medical education, research, and initial diagnosis, which are echoed by similar studies. ChatGPT has been shown to provide useful insights for research by summarizing medical literature and aiding in hypothesis generation, as seen in the work by Thirunavukarasu et al. Additionally, its potential to support clinical decision-making, particularly in offering a second opinion, has been reported by Yu et al. [27, 28]. Despite these strengths, concerns around ethical issues and the potential for medico-legal complications align with observations made in the literature. Studies by Morley et al. have pointed out that while AI tools can be beneficial, their propensity to generate biased or incomplete responses presents a risk, especially in sensitive clinical contexts [29, 30, 31]. Inaccuracies, particularly for complex or evolving clinical scenarios, highlight the need for careful integration of AI into healthcare systems, as excessive reliance on AI may lead to adverse outcomes [32, 33, 34]. Moreover, the concerns regarding ChatGPT's limited knowledge base, constrained by its training data cutoff, have been previously identified by researchers. For example, an investigation into the limitations of large language models by Bommasani et al. highlighted that AI’s inability to access updated information or interpret new guidelines makes it less reliable for real-time clinical decision-making [35, 36, 37]. This study was conducted at a single private medical college, limiting the generalizability of the findings to broader healthcare settings or different regions. The sample consisted of postgraduate residents, which may not capture perspectives from senior clinicians, nurses, or allied health professionals. The accuracy and reliability assessments relied on specialty-based questions authored and evaluated by participants themselves, introducing a risk of subjective bias. Additionally, Finally, as with any AI evaluation, performance is tied to the model’s version and training data; future updates may yield different results.

This study found that Chat GPT delivers high accuracy and reliability when answering medical questions, especially those with clear, binary, or guideline-based answers (median score 5.5/6). Its consistent results across different raters and times suggest it can support medical professionals in routine, fact-based decisions. However, qualitative findings highlight important limitations—such as ethical concerns, medico-legal risks, and reduced accuracy with complex or nuanced cases—underscoring the need for careful human oversight. While Chat GPT shows promise as a supplementary tool in structured clinical contexts, further development and research are needed to address its limitations and support safe integration into real-world healthcare practice.