Accurate diagnosis is fundamental to effective medical treatment, yet diagnostic errors affect millions of patients annually. A meta-analysis of 22 studies involving 80,026 hospitalized patients found a harmful diagnostic error rate of 0.7% (95% CI, 0.5%-1.1%), translating to approximately 249,900 harmful diagnostic errors annually in the United States¹. The burden is substantially higher in outpatient settings, where diagnostic errors affect 5.08% of adults—approximately 12 million Americans each year—with at least 1 in 20 adults experiencing a diagnostic error and half of these errors potentially causing harm². This challenge is magnified for rare diseases, where patients endure a median diagnostic delay of 4.7 years and 40% receive multiple incorrect diagnoses before the correct one is identified³. For the estimated 300 million individuals worldwide affected by rare diseases, the scarcity of specialized expertise and the tendency of rare conditions to mimic common diseases create particularly formidable diagnostic obstacles⁴.

Recent advances in large language models (LLMs) have demonstrated remarkable performance on standardized medical examinations, suggesting potential diagnostic capabilities^5–7. GPT-4 exceeded the United States Medical Licensing Examination passing threshold by more than 20 points, achieving 86.5% accuracy on complex clinical questions⁸. Med-PaLM 2 reached 86.5% accuracy on medical question-answering datasets, with physician evaluators preferring its responses over human physicians' on eight of nine clinical utility metrics⁹. These achievements indicate that current LLMs possess substantial medical knowledge that could theoretically be applied to diagnostic challenges. However, translating examination performance into clinical utility reveals critical limitations. A 2024 systematic review found that while AI models achieved diagnostic accuracy comparable to non-expert physicians (52.1%), they remained significantly inferior to specialists by 15.8% (P = 0.007)¹⁰.

The recent development of multimodal LLMs capable of processing both images and text simultaneously now enables direct comparison of AI and physician diagnostic performance in a format that mirrors clinical practice^11–14. In a clinical evaluation of 150 dermatological cases, SkinGPT-4, a multimodal diagnostic system, achieved 80.63% accuracy validated by board-certified dermatologists¹⁵. Han et al. demonstrated that GPT-4V achieved superior diagnostic accuracy compared with unimodal predecessors (GPT-4, GPT-3.5) and contemporary models (Gemini Pro, Llama 2, Med42) across both JAMA Clinical Challenge and NEJM Image Challenge datasets, establishing that multimodal capabilities enable medical image interpretation without specialized fine-tuning¹⁶. However, GPT-4V demonstrated poor diagnostic performance in radiological contexts, achieving accuracy rates of only 8% without clinical context and 29% with contextualization when requiring the most likely diagnosis¹⁷. Fenglin Liu et al. developed Med-MLLM, a medical multimodal large language model evaluated across five COVID-19 datasets, demonstrating superior performance in COVID-19 reporting, diagnosis, and prognosis tasks even with minimal labeled data (1%)¹⁸. In COVID-19 diagnostic image-text classification tasks, the model achieved 90.3% diagnostic accuracy (AUC) when trained with complete datasets, indicating high efficiency and accuracy for rare disease diagnosis.

Given these limitations in image-only interpretation, we sought to evaluate the diagnostic performance of three state-of-the-art multimodal LLMs—GPT-4o, Claude 3.7, and Doubao—using both image-alone and image-plus-text modalities across NEJM Image Challenge cases, comparing their accuracy against global physician performance to determine the clinical potential of AI-assisted rare disease diagnosis.

Our study encompassed a diverse patient population spanning infancy to advanced age with balanced gender distribution (58.5% male, 41.5% female) and broad disease spectrum including infectious, immune-mediated, neoplastic, and genetic conditions. This comprehensive approach offers several clinical advantages over specialized evaluations. Unlike domain-specific studies focusing on single specialties such as neuroradiology or rheumatology, our broad case selection better reflects the diagnostic challenges encountered in general clinical practice where physicians must differentiate among diverse conditions with overlapping presentations. The consistent AI performance across age groups, particularly the perfect accuracy in infants under one year where physicians achieved only 49.6%, suggests robust generalizability across patient demographics—a critical consideration for real-world implementation. The wide disease spectrum evaluation demonstrates that multimodal AI capabilities extend beyond specialty-specific pattern recognition to general diagnostic reasoning, supporting potential applications in primary care and emergency medicine settings where diagnostic breadth rather than depth is often required.

Our findings diverge significantly from four recent evaluations using NEJM Image Challenge datasets. Han et al. reported GPT-4V achieving 88.7% accuracy on 348 NEJM cases versus 51.4% for human readers¹⁶, while Kaczmarczyk et al. found Claude 3 models reaching only 58.8–59.8% accuracy compared to 90.8% collective intelligence¹⁹, and Suh et al. demonstrated GPT-4o accuracy of 59.6% versus 80.9% for junior faculty radiologists²⁰. A rheumatology-focused evaluation showed Claude Sonnet 3.5 achieving 81.2% accuracy in multimodal tasks versus online participants' 51.6%²¹. Our Claude 3.7 achieved 89.0% accuracy against 46.7% physician majority vote. These disparities reflect critical methodological differences: human performance benchmarks varied from individual physician responses (Han, our study) to collective intelligence aggregation (Kaczmarczyk) or expert radiologist panels (Suh), representing fundamentally different clinical scenarios. Model generation advances likely contributed, as newer versions (Claude 3.7, GPT-4V) consistently outperformed earlier iterations, while dataset composition and evaluation periods differed across studies (Han: 2017–2023; Kaczmarczyk: 2005–2023; Suh: 2005–2024; ours: 2009–2025). The Kaczmarczyk collective intelligence benchmark at 90.8%, though statistically robust, represents an idealized scenario unattainable in clinical practice where individual physicians make diagnostic decisions, explaining apparent AI superiority in studies using realistic individual physician baselines.

Our findings contrast with Le Guellec et al.'s neuroradiology evaluation, where radiologists outperformed GPT-4o and Gemini 1.5 Pro in complete cases (48.0% vs 34.0%) with AI models showing minimal multimodal benefit, unlike our substantial 28–42 percentage point improvements and overall, AI superiority (Claude 3.7: 89.0% vs physicians: 46.7%)²². These disparities likely reflect domain-specific challenges, as neuroradiology requires specialized expertise in subtle imaging findings—an area where AI models failed in 81–94% of cases—and different human benchmarks (expert radiologists vs general physician majority vote), suggesting AI diagnostic capabilities vary significantly across medical specialties.

In approximately 2%–3% of cases, adding clinical text caused models such as GPT-4o and Claude 3.7 to shift from a correct image-based diagnosis to an incorrect one. Our conclusions are based on qualitative review and remain subjective; the specific reasons are unclear. Notably, this pattern was not observed with the Doubao model, and the errors rarely overlapped between GPT-4o and Claude 3.7. Most cases involved images with highly characteristic findings, where non-specific text may have misled models—especially those more reliant on textual cues. These observations highlight model differences and underscore the need for further study.

Several study limitations require acknowledgment. Selection bias in educational case collections may not reflect typical clinical practice complexity. Evaluation using static clinical vignettes differs from dynamic clinical encounters where physicians gather additional information and order sequential investigations. We cannot assess optimal AI-physician collaboration potential or account for real-world time pressures and resource constraints. Additionally, without access to proprietary model training data, possible data contamination cannot be excluded.

This study used only publicly available Internet data and did not involve human subjects. Therefore, no specific ethical considerations were required in this study.