The review process consisted of four stages: identification, screening, eligibility assessment, and final inclusion, following the PRISMA-ScR guidelines¹². The initial search yielded 3,697 articles (Fig. 1). Thirteen articles (0.35%) were removed by automatic deduplication using EndNote. Based on title and abstract screening, 3,569 articles (96.54%) were excluded according to the eligibility criteria. A full-text review was performed of the remaining 115 articles (3.11%), of which 78 (2.11%) were excluded because they did not meet the inclusion criteria. Finally, 37 articles (1.00%) were included in the analysis. An additional 18 articles were identified through forward snowballing by reviewing the citations of initially included 37 articles via Google Scholar, resulting in a final total of 55 studies included in this review.

The benchmark datasets were categorized based on their sources: medical licensing examinations (n = 17)^13–29, consumer health questions (CHQ) (n = 17)^{18,21,30–44}, and electronic health records (EHR) (n = 17)^{20,25,45–59}. There were slightly fewer datasets derived from biomedical literature (n = 14)^{18,29,41,48,49,56,60–67} (Table 1). Nine benchmarks utilized multisource data that incorporated two or more types of inputs (Table 1).

As shown in Fig. 2, the temporal trends revealed that only four benchmarks were introduced in 2018, increasing to nine in 2019. This number declined to three and five in 2020 and 2021, respectively, before rising to 11 in 2022 and peaking at 15 in 2023. Exam-, EHR-, and literature-based benchmarks showed limited development in earlier years but increased notably after 2022 or 2023. The CHQ-based benchmarks initially rose in 2019, declined thereafter, and resurged in 2022 and 2023. Multisource benchmarks were not introduced until 2021, with continued development in subsequent years.

The annotation presence, method, and specificity were assessed across 55 benchmarks (Fig. 3). Of these, 39 (70.9%) were annotated, 16 (29.1%) were not annotated, 25 (64.1%) were annotated manually, 5 (12.8%) were annotated using models, and 9 (23.1%) were annotated using a hybrid approach. GPT-4 was employed for annotation in CMExam and K-QA, whereas MedREQAL utilized GPT-3.5 for health-area classification.

Manual annotation in exam-based datasets sometimes targets only subsets of questions (Table 0). The annotations included the medical specialties, reasoning types, difficulty levels, and question types. In CHQ-based benchmarks, annotations involve generating answers or assessing response reliability. Some benchmarks include question type categorization, and the K-QA employed International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10) – based annotations. Literature-based benchmarks were annotated for answer generation and content classification. The EHR-based benchmarks included prototype clinical queries extracted from electronic medical records (EMRs), which were subsequently annotated manually or automatically.

The approaches used to evaluate the models against the benchmarks were subsequently analyzed. Code-based evaluation was the most common method (n = 38), followed by human evaluation (n = 3). Twelve studies employed both code-based and human evaluations. Two benchmarks, ExplainCPE and CMB, were used to assess the outputs generated by GPT-4.

Prior to 2020, benchmark validation primarily relied on neural network–based models such as convolutional neural networks (CNNs), long short-term memory networks, and bidirectional gated recurrent units, including several variants such as multi-scale CNNs, multilevel composite CNNs, and multi-scale attentive interaction networks. In the early 2020s, numerous BERT-based variants, such as RoBERTa, BioBERT, ALBERT, ClinicalBERT, and PubMedBERT, were widely adopted. Since 2023, GPT-based LLMs, including ChatGPT, GPT-4, and PaLM, have been increasingly employed alongside the growing use of open-source models, such as Llama.

The metrics used to evaluate the model performance were also examined. Accuracy (n = 27) and F1 score (n = 22) were the most frequently reported metrics commonly used to assess the overall performance. ROUGE (n = 17) and BLEU (n = 8) were used to evaluate the similarity between generated texts. Additional metrics, including exact match (n = 8), precision (n = 4), and recall (n = 4), were used to evaluate predictive accuracy. Other metrics, such as BERTScore (n = 7) and METEOR (n = 5), were also applied in several studies.

We developed a structured taxonomy of benchmarks to help researchers select benchmarks (Fig. 4). First, the benchmarks were divided into English and non-English based on language because language is important. Given that there were many Chinese benchmarks, we divided non-English into Chinese and other benchmarks. Next, as the source of the benchmark data is important, we distinguished whether the benchmark was based on multiple sources. If it was not multi-source, we further categorized it into Exam, EHR, CHQ, and Literature. Finally, we distinguished whether the benchmark was classified into detailed items.

.

The development framework consists of 24 questions based on five key principles: (1) Reliable, (2) Ethical, (3) Annotated, (4) Diverse, and (5) Yield-validated (Table 2). First, five questions were asked to confirm that the benchmark data could be sourced from reliable sources and were of high quality. This included explanations of the data source, composition, preprocessing methods, collection from a reliable institution, data recency, and data volume. Second, regarding ethics, the framework included the removal of patients’ personal information, data bias, compliance with the institutional review board, privacy protection laws, and reusability. Third, the framework checked whether a meaningful and consistent annotation system existed. The evaluation consisted of five questions regarding the clarity and consistency of annotations, annotation workflow, annotation usage model, annotation quality verification, and consistency rate among annotators. Fourth, diversity included whether the classification had been performed across multiple departments, ICD codes, or inference types. Finally, five questions were included regarding the evaluation: whether a benchmark evaluation was performed, the models and metrics used, whether reviews by external experts or clinicians were conducted, and whether the results were publicly available. All 24 questions can help researchers to confirm whether the benchmark is appropriately prepared.

The study was conducted in accordance with the PRISMA-ScR guidelines¹². A previous comprehensive review that examined 444 benchmarks across diverse domains identified seven benchmarks specific to the medical field⁹. Relevant keywords such as “clinical,” “medical,” “healthcare,” “benchmark,” “question answering,” and “QA dataset” were extracted from these studies. Additional insights were drawn from earlier scoping reviews targeting EHR and CHQ datasets^10,11. The literature search was restricted to studies published between January 1, 2017, and July 30, 2024. Searches were conducted in PubMed, the ACM Digital Library, and the ACL Anthology.

Study selection followed a two-step screening process, with inclusion and exclusion criteria derived from prior literature^9–11. In the first screening step, two reviewers independently assessed each record. Discrepancies were resolved by a third reviewer. The second step involved full-text screening to finalize the inclusion of studies.

The analytical framework was adapted from previous benchmark reviews and studies focusing on dataset development^9–11. It comprised two principal domains: dataset construction and evaluation.

Dataset construction was assessed based on dataset type, source, construction methods, annotation practices, categorization, and language. The dataset types were classified as follows: (1) medical licensing exams (e.g., United States Medical Licensing Examination, USMLE); (2) CHQs generated through patient–provider online interactions; (3) EHRs from EMRs; and (4) biomedical literature from textbooks, scientific articles, or websites such as Wikipedia. Datasets integrating multiple sources were categorized as multi-source.

Based on prior studies, construction methods were classified into three categories: (1) Human-generated datasets, constructed manually by annotators following predefined criteria, without LLM involvement. (2) Model-generated datasets, produced using language models prompted to generate desired outputs. (3) Collection and improvement of existing datasets, developed by compiling and modifying open-source datasets⁹. Annotation methods were categorized by whether they were manual or automated, and by the nature of the annotated content. Information regarding the number and characteristics of categories was extracted. The quantities of training, test, and validation subsets were recorded.

Evaluation analysis included evaluation methods, models used, metrics, and additional assessments. Consistent with prior literature, evaluation methods were classified into: (1) Code-based evaluation, quantitative assessment using predefined metrics. (2) Human evaluation, qualitative assessment by individuals with relevant expertise. (3) Model-based evaluation, assessment wherein LLMs evaluate outputs⁹. To determine model performance, we recorded the evaluation models used in each study and the metrics applied. Additional evaluations, including implementation-specific details, were also reviewed.

No new data was generated or analyzed in this study. All data supporting the findings of this study are available within the cited articles included in the scoping review.