\documentclass[fleqn,10pt]{wlscirep}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{multirow} 
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{subfigure}
\usepackage{subcaption}
\newcommand{\Fonte}[1]{\textbf{Fonte:} #1}
\linespread{1.25} % Espaçamento de 1,25 linhas
\title{Interoperable web platform based on large language models for medicals data analysis}

\author[1,2*]{Marcello Carvalho dos Reis}
\author[1+]{Rafaelly Rios dos Santos}
\author[3,+]{Md Rafiul Hassan}
\author[4,+]{Mohammad Mehedi Hassan}
\author[1,+]{Daniel Santos da Silva}
\author[1,+]{Pedro Lino Azevedo Landim}
\author[1,+]{Victor Hugo Costa de Albuquerque}
\affil[1]{Federal University of Ceará, Fortaleza, Brazil}
\affil[2]{Meteora, Fortaleza, Brazil}
\affil[3]{Department of Computer Science, Central Connecticut State University, New Britain, CT, USA}
\affil[4]{Information Systems Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia }

\affil[*]{marcelloreis@alu.ufc.br}

\affil[+]{these authors contributed equally to this work}

\keywords{Interoperability; Artificial Intelligence; Data Management}

\begin{abstract}
\textcolor{This paper introduces an interoperable web platform for managing medical data, prioritizing security, and integration of information from multiple sources using FHIR (Fast Healthcare Interoperability Resources). The objective is to optimize medical record analysis with artificial intelligence (AI) and machine learning, offering automatic alerts and preventive recommendations while complying with the General Data Protection Law (GDPL in English; LGPD in Portuguese). The platform facilitates efficient sharing of data among hospitals, clinics, remote devices, and healthcare systems, improving diagnostic and treatment accuracy.} The methodology involved creating a secure, LGPD-compliant web platform, the integration of data through FHIR to ensure interoperability. AI algorithms analyze medical data, generate alerts, and provide personalized recommendations. Performance was assessed in controlled and stress tests, focusing on scalability and security. Results highlighted promising performance of the Retrieve Augmentation Generation (RAG) technique with BAAI/bge-small-en embedding models. Metrics such as BertF1, BertP, and BertR ranged from 0.389 to 0.538, averaging 0.43, indicating moderate consistency. The average Bleu score was 0.442, reflecting diverse response quality, while Rouge metrics averaged 0.326, indicating lower
precision. Performance with Chest X-rays and MedQA datasets showed better results with Chest X-rays, achieving higher scores but higher perplexity (3.635e4), indicating challenges in generating clinical text. MedQA showed greater response diversity (0.807) but lower precision. In qualitative analysis, Chest X- rays demonstrated higher semantic similarity (mean 0.767) compared to MedQA (mean 0.754). During load testing, the platform remained stable as user numbers increased, but response times grew under stress, suggesting bottlenecks in high-demand scenarios. In conclusion, the platform is a promising tool for integrating medical data and supporting clinical decisions. The FHIR standard ensured interoperability, while AI effectively analyzed records and issued alerts. Adjustments are needed in response times under heavy loads and improvements in infrastructure and mobile experience to encourage greater patient adoption.
\end{abstract}
\begin{document}

\flushbottom
\maketitle
% * <john.hammersley@gmail.com> 2015-02-09T12:07:31.197Z:
%
%  Click the title above to edit the author information and abstract
%
%\thispagestyle{empty}

%\noindent Please note: Abbreviations should be introduced at the first mention in the main text – no abbreviations lists. Suggested structure of main text (not enforced) is provided below.

\section*{1 - Introduction}

The storage, management and sharing of health information represent complex challenges in the current healthcare landscape \cite{chukwu2020systematic,chen2024integration}. As medical care is provided across various institutions, including hospitals, diagnostic clinics, laboratories, and doctor’s offices, a single patient’s information is often recorded and stored in different information systems \cite{vardhan2024large}. The lack of interoperability among these systems hinders the efficient retrieval and sharing of data \cite{liu2021medical}.

The absence of a universal electronic health record prevents healthcare professionals from quickly accessing critical patient information during care, which can compromise diagnoses and treatments, especially in emergency situations \cite{cheng2022basic}. Even with advances in digital technologies, limited access to complete medical history remains a significant barrier to collecting essential information that could improve diagnostic and treatment effectiveness \cite{fang2022construction}.
Recent studies have explored the use of large language models (LLMs) to address these challenges. These models have been applied in various clinical contexts, such as triage, diagnostics, and decision support. Table 1 provides a summary of the key contributions of 10 studies that explored LLM applications in healthcare.

For example, the Jahan's study \cite{jahan2024comprehensive} evaluated the performance of LLMs in biomedical tasks, demonstrating high accuracy in specific contexts. Another study developed fine-tuned LLMs for digital mental health screening, highlighting their impact on personalization and diagnostic efficiency \cite{taylor2024model}. Applications of LLMs in pneumoconiosis diagnostics were explored in Song et al. \cite{song2024pneumollm}, while Chien et al. \cite{chien2024using} utilized these technologies to identify workload patterns among informal caregivers.

Additionally,  Xue, X. et al. \cite{xue2024xiaoqing} presented a question-and-answer system based on LLMs for glaucoma diagnosis, emphasizing improvements in medical education. Other studies, such as Tan et al.\cite{tan2024medchatzh} and Wu et al. \cite{wu2024llm}, demonstrated the effectiveness of LLMs in hybrid diagnosis and traditional Chinese medicine consultations. The use of LLMs for emotion recognition in mental health was explored in Zhang et al. \cite{zhang2024llm}, while Sarzaeim et al. \cite{sarzaeim2024framework} highlighted their application in intelligent policing systems, showcasing potential uses beyond healthcare with relevant contributions to public health.

In contrast, studies such as Chen et al. \cite{chen2024mass} highlighted the importance of integrating technologies such as space-air-ground networks to optimize decision-making processes and data analysis in complex scenarios. In Li et al. \cite{li2024benchmarking} investigated the use of LLMs in clinical
questioning and prompt engineering, while Yan et al. \cite{yan2024knownet} combined LLMs with knowledge graphs in the KNOWNET system to reduce biomedical misinformation. In Bonfigli et al. \cite{bonfigli2024pre} emphasized the importance of fine-tuning LLMs to maximize performance in diagnostics and triage.

\Despite advancements in AI and machine learning, implementing these technologies in healthcare presents significant challenges. Integrating LLMs requires robust infrastructure, effective data governance, and compliance with data protection laws such as the General Data Protection Regulation (GDPR) and the General LGPD. These frameworks ensure the ethical and secure handling of sensitive health data while addressing growing concerns about privacy, security, and bias in AI-driven decisions. Furthermore, ensuring the interoperability of diverse data sources, such as electronic health records (EHRs), imaging systems, and IoT devices, remains a critical obstacle to seamless data integration.

This study acknowledges the need for a patient-centered approach to healthcare, where individuals have greater control over their data and can actively participate in managing their health. By incorporating standards such as FHIR and leveraging cloud technologies, the proposed platform aims to bridge the gaps in data accessibility and interoperability. Beyond its technological contributions, this platform envisions fostering collaboration among healthcare providers, empowering patients, and promoting a preventive approach to healthcare delivery through advanced analytics and real-time insights.


Therefore, the remainder of this article is organized as follows: Section 2 presents the theoretical foundation, discussing concepts related to healthcare interoperability, language models, and artificial intelligence. Section 3 details the adopted methodology, describing the system architecture, the technologies used, and the validation processes. Section 4 presents the obtained results, including quantitative and qualitative analyses, as well as load and stress tests. Finally, Section 5 concludes the study, highlighting the main contributions and suggesting future research directions and improvements for the proposed system.

\section{2 - Literature Review}

\subsection*{2.1 - Healthcare Market}

The aging population, the shortage of qualified healthcare professionals, and rising costs present substantial challenges for healthcare systems worldwide \cite{vynamic2024health}. The growing demand for medical assistance due to the aging population increases pressure on medical resources, especially in high-income countries, which creates additional pressure on already overwhelmed healthcare infrastructures due to rising costs and demand for specialized care. Furthermore, the financial burden of long-term care is shifting to personal savings and pension funds, further complicating access to healthcare for older individuals \cite{vynamic2024health, roesler2023scaling}.

At the same time, digital innovations in healthcare, such as wearable devices, big data, and AI, are transforming the way healthcare is delivered, enhancing efficiency and personalization, and offering some relief by improving care delivery \cite{br2019nucleo}. However, this vast and complex global healthcare market encompasses medical services, pharmaceuticals, healthcare equipment, insurance, and technologies targeted at the sector \cite{roesler2023scaling}. It is important to note that many countries adopt mixed models, where public and private healthcare systems coexist, with regulations and attempts at interoperability that vary by country, depending on the adopted health policy.

In Brazil, specifically, the healthcare market is characterized by the coexistence of two systems: the public and universal Unified Health System (UHS) \cite{ministerio_saude_2024} and the supplementary healthcare sector, consisting of private health plans\cite{br2019nucleo}.
UHS, established by the Federal Constitution of 1988 and regulated by Laws No. 8.080/1990 and 8.142/1990 \cite{ministerio_saude_2024}, is based on the principles of universality, comprehensiveness, and equity, with the goal of ensuring that all Brazilian citizens have access to free healthcare. However, the private sector covers approximately twenty five per cent of the Brazilian population, using private health plans and insurance, regulated by the National Supplementary Health Agency (NSHA) and the Private Insurance Superintendence (PIS) \cite{vynamic2024health}.

Although UHS faces challenges such as underfunding and long waiting lists, there is increasing adoption of telemedicine technologies and electronic health records \cite{mahadik2024digital}. The quality of healthcare services in Brazil varies by region, with major urban centers offering better services compared to rural and poorer areas \cite{br2019nucleo}. The private sector, in turn, heavily invests in top-tier hospitals and diagnostic technologies. To regulate the supplementary healthcare sector in Brazil, NSHA was created by Law No. 9.961/2000, which includes private health plans \cite{ans_2024}. Its role is to ensure the financial balance of health plan operators, guarantee the quality of services provided, and protect consumer rights, particularly defining standards for Data Exchange in Supplementary Healthcare to attempt to ensure interoperability between service providers, health plan operators, and NSHA itself. To this end, standards such as Supplementary Health Information Exchange (SHIE) and Unified Terminology of Supplementary Health (UTSH) were developed \cite{saracco2020personal}.

\subsection*{2.2 - Interoperability in Healthcare
}

Health data interoperability refers to the ability of different healthcare systems, devices, and applications to exchange, interpret, and use information in an integrated and efficient manner \cite{zhou2024prognostic, kim2024phenoflow}. This is crucial to ensure that data generated in one healthcare system can be accessed, understood, and used in another, without the need for manual intervention or format conversion \cite{tabari2024state, nguyen2024}.

Health data interoperability consists of three types: First, Technical Interoperability, which is the ability of different systems to exchange data through common standards, such as communication protocols and data formats, for example, Health Level Seven (HL7), FHIR, and Digital Imaging and Communications in Medicine (DICOM) \cite{kumar2024}  This allows data to be sent and received between healthcare systems; Second, Semantic Interoperability, which ensures that the exchanged information is interpreted in the same way by all systems. 

This requires the use of standardized terminologies and taxonomies, such as Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT) and Logical Observation Identifiers Names and Codes (LOINC) \cite{patel2023,tabari2024state}, so that health data have the same meaning across different systems; Lastly, Organiza- tional Interoperability, which involves collaboration between different healthcare organizations, ensuring that policies, standards, and procedures are aligned so that data exchange, for example, SHIE and UTSH, happens smoothly, respecting privacy and security norms \cite{kumar2024,nguyen2024}.

HL7 is an organization dedicated to developing standards for the exchange, integration, sharing, and retrieval of electronic health information \cite{chen2024,martinez2024}. These standards are widely used to ensure that different healthcare information systems can communicate effectively, promoting interoperability among hospitals, clinics, laboratories, and other healthcare providers \cite{kumar2024}.

The standardized framework provided by HL7 enables the exchange of clinical and administrative data, reducing ambiguities and improving accuracy in communication between systems \cite{martinez2024}. The HL7 standards facilitate the integration of various healthcare information systems, such as Electronic Health Records (EHRs), laboratory systems, applications, and APIs \cite{chen2024}.
HL7 facilitates interoperability both at the syntactic level (how the data is formatted) and at the semantic level (what the data means), ensuring that systems understand the exchanged information consistently \cite{kumar2024}. Through this effective exchange of information, \textbf{HL7 standards contribute to better care coordination and more informed clinical decisions \cite{chen2024,martinez2024}.}

\textbf{FHIR is a standard developed by HL7 that establishes a standardized way to represent and exchange health information \cite{nguyen2023,chen2024}}. By using modern web technologies such as RESTful APIs, JSON, and XML, FHIR facilitates interoperability between healthcare information systems, allowing different applications to communicate efficiently and quickly \cite{martinez2024}. 

By enabling the fast and secure exchange of information, FHIR promotes continuity of care and improves communication between healthcare providers \cite{kumar2024}. With its modular approach, it allows the integration of various systems, such as EHRs, laboratory systems, and telemedicine applications. Additionally, the use of APIs and modern data formats makes it easier to implement new healthcare solutions, such as mobile apps and real-time data analytics tools \cite{chen2024}. Structured data via FHIR can also be applied in clinical research, aiding in generating insights and improvements in medical practice \cite{nguyen2023}.

Digital Imaging and Communications in Medicine (DICOM) is an international standard that defines the exchange, storage, and presentation of medical images and related information. This standard is crucial for ensuring interoperability between different devices and healthcare information systems, especially in areas such as radiology, cardiology, and other specialties that use medical imaging \cite{patel2023}. DICOM provides a standardized format for medical images, such as X-rays and MRIs, allowing different devices and software to interpret and display these images consistently. 

This standardization enables healthcare institutions to share images and reports efficiently, improving continuity of care and collaboration among professionals \cite{smith2023}.
 Moreover, DICOM facilitates the integration of medical imaging systems with EHRs, providing access to complete patient information. Medical images encoded in DICOM can be used in research and clinical analysis, {enabling the development of machine learning  and computer vision algorithms for diagnoses \cite{chen2024,martinez2024, sun2024transunet, 10456909, pan2024mutual }.


\textbf{ The interoperability provided by DICOM improves the efficiency of clinical workflows and reduces the risk of errors related to data exchange \cite{nguyen2024,wang2024}.}
Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT) is a comprehensive coding system that provides a standardized language for representing clinical concepts in electronic health records. \textbf{It allows clinical data to be accurately and consistently coded, facilitating communication between healthcare professionals, information systems, and institutions \cite{meyer2021,santos2022}.}

The main contributions of SNOMED CT to health data interoperability come through standardizing terminology and codes, ensuring consistency and quality in the description of diagnoses, procedures, and health conditions,\textbf{ which reduces ambiguities \cite{smith2023}\cite{kim2024}}. The use of SNOMED CT enables the integration of data from different sources, such as hospitals, clinics, and laboratories, facilitating large-scale data analysis. Finally, data coded with SNOMED CT can be used in clinical research, epidemiological studies, and public health analysis, \textbf{providing insights into trends and patterns \cite{santos2022,smith2023}.}

Logical Observation Identifiers Names and Codes (LOINC) is a standardized coding system that enables the identification and exchange of information on laboratory, \textbf{clinical, and other healthcare observations \cite{patel2023,martinez2024}.} This system ensures that health data can be consistently represented, regardless of the healthcare institution or information system used. \textbf{LOINC codes cover a wide range of clinical and laboratory tests, measurements, and observations, such as blood pressure readings, cholesterol levels, and radiology reports \cite{chen2024,patel2023}.} This allows healthcare providers to exchange data about patients’ medical conditions in a \textbf{standardized format, reducing errors and improving the accuracy of diagnoses and treatment decisions \cite{smith2023,martinez2024}.}

\subsection*{2.3 - Artificial Intelligence and LLM
}
Conceptually, LLM is a deep learning model architecture composed of an extremely considerable number of parameters, often in billions or even trillions. These models are trained on pre-training tasks such as Masked Language Modeling (MLM) and Auto-Regressive Prediction (ARP) \cite{basile2024can}. These models are fed massive corpora of text to capture linguistic, semantic, and syntactic patterns \cite{al2024blockchain}.

The LLM must be able to capture complex semantic relationships and syntactic dependencies in sentences and large texts. This includes the ability to understand anaphora and linguistic ambiguity, such as pronoun resolution \cite{roesler2023scaling}. The model uses its internal mechanisms to encode the contextual meanings of tokens in a nonlinear way, allowing it to understand not only the literal meaning of words but also their implicit meanings, metaphors, and other high-level linguistic aspects.

 The ability to generate fluent and coherent text that mimics the structure and fluency of human language is one of the central metrics for evaluating the quality of an LLM \cite{basile2024can}. Text generation involves the probabilistic selection of the most appropriate tokens, ensuring not only grammatical correctness but also contextual relevance. This is achieved through the probabilistic modeling of hidden Markov chains within the latent space of the model, where the transition probabilities between tokens are adjusted by the context and semantic relationships learned during pretraining.

Highly effective LLMs are trained on a wide range of topics and domains, enabling them to maintain sophisticated contextual awareness in knowledge-intensive areas such as medicine, law, and science \cite{al2024blockchain}. This requires the use of specialized datasets to adapt the model to technical fields. The ability to adjust the generated text to be coherent and relevant in very specific contexts, maintaining terminological and factual rigor, is achieved through techniques such as supervised fine-tuning and supervised instruction.

The ability to follow instructions efficiently and accurately, an increasingly desired feature in LLMs, is obtained through methods such as Reinforcement Learning with Human Feedback (RLHF) \cite{roesler2023scaling}. During this process, the model is refined based on human interactions, learning to align its output with the specific intentions and directions provided by human users. This makes LLMs effective in tasks involving problem-solving, decision-making, and executing complex instructions, particularly useful in customer support, assisted programming, and other industrial applications.

LLMs, with their advanced ability to understand, generate, and interpret natural language, have a broad range of applications in the healthcare sector. Due to their ability to handle large volumes of textual data and complex contexts, these models are being implemented in various areas of healthcare to improve care, clinical decision-making, and research \cite{basile2024can}.  For example, in Clinical Decision Support, LLMs can act as Clinical Decision Support Systems (Clinical Decision Support System (CDSS)), assisting doctors and other healthcare professionals in making more informed decisions, such as text-based diagnosis, treatment suggestions based on clinical guidelines, medical literature, and patient history, and predicting clinical outcomes, as LLMs can process large volumes of data to predict clinical outcomes based on historical patterns, \textbf{such as the risk of postoperative complications or the probability of hospital readmission \cite{al2024blockchain, pan2024enhancing}.}

Another application of LLMs is in the role of Virtual Assistants for Patients and Professionals, as they can be used to create intelligent virtual assistants that interact with patients and healthcare professionals \cite{roesler2023scaling}. An important application is in managing health data, where in the Analysis of Electronic Health Records and Natural Language Processing (NLP), LLMs play a crucial role in extracting information from Electronic Health Records (EHRs), which contain vast amounts of unstructured textual data. For example, in automatic coding, LLMs can perform the automated coding of diagnoses and medical procedures from clinical notes, facilitating the generation of classification codes such as International Classification of Diseases (ICD-10) and Current Procedural Terminology (CPT), which are essential for billing and reporting \cite{basile2024can}. 

Furthermore, in the detection of adverse events and clinical alerts, as we can see in Figure \ref{Fonte:fig1} in the architecture for disease identification, when processing admission notes and exam reports, LLMs can identify early signs of complications or adverse events, such as dangerous drug interactions, and issue alerts to the doctor team. Another key point is in summarizing medical records: language models can generate concise summaries of complex electronic health records, highlighting the most relevant points for healthcare professionals during consultations, optimizing time, and facilitating decision-making \cite{al2024blockchain}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.80\linewidth]{Artigo_Tese_001.png}\caption{Example of architecture for disease identification.}
    \label{Fonte:fig1}
\end{figure}
\newpage

Still, in data handling, LLMs are important in Automatic Documentation and Writing of Clinical Reports, where they are effective in automating medical reports and clinical documentation, helping to relieve the workload of healthcare professionals, and improving the accuracy of documentation. Using another type of file, such as DICOM, we can see its use in automatically generating image reports: LLMs can transform the analysis of medical images, such as X-rays or MRIs, into structured reports, assisting radiologists in quick and accurate documentation. Finally, in transcribing and summarizing medical consultations: using LLMs, it is possible to automatically generate transcriptions of medical consultations and summarize them into clinical notes, integrating this information directly into the electronic health record. This reduces the time doctors spend on administrative tasks and allows more focus on the patient \cite{roesler2023scaling}.

Health data is highly sensitive and requires stringent security measures\citeonline{leme2020lei}. For example, a healthcare device might implement encryption systems to protect patient information stored in electronic records. The use of multifactor authentication is another recommended practice to ensure that only authorized professionals have access to this data. In case of a breach, it is essential to have incident response plans that include clear protocols for notifying patients in the event of data leaks, in compliance with LGPD requirements \cite{BRASIL.CONSELHO.NACIONAL.DE.SAÚDE.2012}. These practices help prevent cyberattacks, which have become more frequent in the healthcare sector, with data being targeted by ransomware and other types of attacks.

The impact of LGPD directly reflects on the development of digital solutions in healthcare, such as health apps. When developing health monitoring apps, such as those collecting data on eating habits or chronic conditions, companies must incorporate data protection features from the start, such as clear notices about data collection and consent options . Additionally, telemedicine platforms must ensure that patient data, such as diagnoses and prescriptions, are handled in accordance with LGPD. This includes using secure systems that ensure confidentiality of consultations. Compliance with LGPD not only helps protect patient data but also becomes a competitive differentiator, as it increases user trust in using these technologies.

\subsection*{2.4 - Applications for Language Models in Healthcare: Advances and Limitations}

The use of Large Language Models (LLMs) has been increasingly integrated in several areas, bringing innovation and improving how complex tasks are performed. In the health sector, these models have stood out above all in tasks
such as diagnosis, screening and clinical support, we can evaluate this context in Figure \ref{fig:tabelasurvey} . For example, an integrated assessment by Jahan et al. analyzed the
effectiveness of LLMs, such as ChatGPT, PaLM and LLaMA, in biomedical contexts. The results indicate that these models
robust process performance medical texts, with significant potential to contribute to diagnostics and clinical care optimization \cite{jahan2024comprehensive}

Moreover, the adaptation of LLMs for specific clinical needs has been a major research focus. In a study focused on mental health, Taylor et al. developed LLMs tailored for digital triage, an application that can significantly enhance the personalization of care. By considering the particularities of each patient, these models help improve diagnostic accuracy and efficiency, ensuring a more individualized approach to care \cite{taylor2024model}. A similar application of LLMs in medical diagnosis was explored by Song et al., who created PneumoLLM, a model specialized in diagnosing pneumoconiosis. Besides being a valuable tool for physicians, this model also enhances the accuracy of respiratory disease diagnoses from medical images \cite{song2024pneumollm}.

The potential of LLMs extends beyond clinical diagnosis, encompassing care management and complex data analysis. An innovative study by Chien et al. demonstrated how LLMs can be used to identify overburdened informal caregivers in long-term care settings. The combination of these technologies with attention visualization techniques provides clearer in- sights into workload patterns, contributing to more efficient human resource management in this sector \cite{chien2024using}. In ophthalmology, Xue et al. developed Xiaoqing, an LLM-based question-and-answer system specialized in glaucoma diagnosis. 

This system not only improves patient and healthcare professional education but also serves as a  support tool for clarifying clinical doubts \cite{xue2024xiaoqing}. In parallel with these innovations in conventional healthcare, traditional medicine is also benefiting from LLM applications. Tan et al. fine-tuned a dialogue model for consultations on traditional Chinese medicine, creating MedChatZH. This model exemplifies the flexibility of LLMs in various cultural and medical contexts, expanding their application possibilities \cite{tan2024medchatzh}. The versatility of LLMs was also highlighted by Wu et al., who combined LLMs with Transformers in a hybrid system for medical diagnosis. By integrating fuzzy models and context selectors, this system improved diagnostic accuracy in hospital settings, making the process more reliable \cite{wu2024llm}.

Another significant advancement in the LLM field is the ability to handle incomplete data and human emotions. Zhang et al. proposed a multi-teacher knowledge distillation technique for emotion recognition, using LLMs to manage uncertainty and data scarcity in emotional health contexts. This application shows how LLMs can be used to enhance emotional intelligence in healthcare and patient support systems \cite{zhang2024llm}. Additionally, the application of LLMs in fields outside healthcare is also expanding. Sarzaeim et al. explored how these models can be used in smart policing systems to predict crimes, employing few-shot and zero-shot prompting techniques. This opens new possibilities for automating public safety\cite{sarzaeim2024framework}.

\begin{table}[htbp]
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}>{\raggedright}p{3cm} p{3cm} p{3cm} p{3cm} p{3cm}@{}}
\toprule
\textbf{Authors} & \textbf{Goals} & \textbf{Methods} & \textbf{Highlights} & \textbf{Limitations} \\ 
\midrule
Jahan et al. \cite{JAHAN2024108189} & Evaluate LLM performance in biomedical tasks & ChatGPT, PaLM, LLaMA & High accuracy in biomedical tasks & Limited focus on biomedical domain \\ 
Taylor et al. \cite{TAYLOR2024102988} & Develop LLMs for digital screening in mental health & LLM fine-tuned for clinical screening & Improvement in personalized care & Need for additional clinical data \\ 
Song et al. \cite{SONG2024103248} & Pneumoconiosis diagnosis with LLMs & LLMs in image diagnostics & High accuracy in radiological diagnoses & Dependence on high-quality images \\ 
Chien et al. \cite{CHIEN2024108329} & Identify overloaded informal caregivers & NLP, Visual attention & Visualization of workload patterns & Limited long-term application \\ 
Xue et al. \cite{XUE2024108399} & Create a Q\&A system for glaucoma & Q\&A with LLMs & Improvement in medical education & Exclusive focus on glaucoma \\ 
Tan et al. \cite{TAN2024108290} & Fine-tune LLMs for traditional Chinese medicine consultations & Fine-tuning, Medical dialogue & Integration of traditional medicine and LLMs & Need for greater generalization \\ 
Wu et al. \cite{10720422} & Hybrid medical diagnosis with LLMs & Transformers, Fuzzy selection & Better performance in complex diagnoses & Requires continuous model tuning \\ 
Zhang et al. \cite{10697478} & Emotion recognition with LLMs and multi-teacher distillation & Multi-teacher Knowledge Distillation & Accuracy in emotional recognition & Limitation in heterogeneous data \\ 
Sarzaeim et al. \cite{10538107} & Smart policing with LLMs & Few-shot, Zero-shot prompting & High efficiency in crime prediction & Dependence on reliable criminal data \\ 
Kim et al. \cite{10689638} & Visual analysis of stroke with LLMs & PhenoFlow, Visual analysis & Reduced cognitive load in diagnoses & Dependence on human-AI interaction \\ 
Chen et al. \cite{10720517} & Coordination for energy dispatch with connected vehicles & LLMs, SAGVNs & Cost optimization and energy decision-making & Complexity in integrated vehicular networks \\ 
Li et al. \cite{10723298} & Benchmarking LLMs in evidence-based medicine & Prompt engineering, Clinical NLP & Contribution to automated diagnostics & Need for more clinical samples \\ 
Yan et al. \cite{10670469} & Integration of LLMs with knowledge graphs for health & KNOWNET, Knowledge graphs & Reduction in biomedical misinformation & Need for comprehensive data \\ 
Bonfigli et al. \cite{BONFIGLI2024103003} & Adapt LLMs to the biomedical domain & Fine-tuning, Probing & Success in diagnostics and screening & High dependence on specialized data \\ 
\bottomrule
\end{tabular}
}
\caption{Summary of references on LLMs in various application areas.}
\label{tab:llms_summary}
\end{table}



%\begin{figure}[ht]
%    \centering
%    \includegraphics[width=0.65\linewidth]{image.png}
%    \caption{Summary of references on LLMs in various application areas.}
%    \label{fig:tabelasurvey}
%\end{figure}
The impact of LLMs is not limited to diagnostics and security but is also felt in areas such as clinical data analysis and process optimization. Kim et al. developed PhenoFlow, an LLM-based system that facilitates the visual analysis of stroke data, reducing the cognitive load on healthcare professionals and aiding in quick and accurate decision-making \cite{kim2024}. Similarly, Chen et al. utilized LLMs to optimize energy dispatch in space-air-ground networks, demonstrating how these models can be effectively applied in engineering and resource optimization \cite{chen2024integration}.

In evidence-based medicine, Li et al. conducted a study on the application of LLMs in tasks such as prompt engineering and clinical answering. These advancements not only aid in diagnostic automation but also improve therapy efficiency, making the medical process faster and more accessible \cite{li2024benchmarking}. Furthermore, Yan et al. integrated LLMs with knowledge graphs for the KNOWNET system, a health information search tool that helps minimize misinformation, ensuring safer and more efficient exploration of biomedical data \cite{yan2024knownet}.

Finally, Bonfigli et al. emphasized the importance of fine-tuning for the success of LLMs in biomedical contexts, demonstrating how these models can be adjusted for specific tasks such as diagnosis and medical triage. This type of adjustment allows LLMs to achieve superior performance, improving the accuracy and efficiency of medical services in various contexts \cite{bonfigli2024pre}. Together, these contributions reveal that LLMs are becoming indispensable tools in multiple fields, with transformative potential for healthcare, security, and beyond, driving significant advancements in how we interact with artificial intelligence.


\section*{3 - Methodology}

The process of developing the tool will be presented, covering the technologies and frameworks used, as well as the capacity and stress tests. The methodology adopted for the development of the web interface and its integration with our multimodal LLMOps pipeline involves the use of open-source LLMs (Large Language Models) and SLMs (Small Language Models), such as Ollama models, Microsoft-Phi, among others. The combination of these language models using Ensemble RAG (Retriever Augmentation Generation) will also be discussed, along with the incorporation of Autonomous Agents based on SLMs. The platform was built with a software architecture combining Flutter for the frontend and Flask API for the backend. The interface was designed to allow users to input text with questions and receive responses generated by the natural language model.

\subsection*{3.1 - Technologies Used}

The development process of the tool was divided into two main modules: the user interface and the resource server. The user interface, called FrontEnd, is responsible for consuming data provided by the server, which consists of an API and a MySQL database. Communication between the interface and the server was implemented using the REST standard, a widely adopted model to ensure system interoperability, scalability, and flexibility. The FrontEnd was developed using the Flutter framework, which supports the development of responsive applications for both web and mobile devices. The choice of Flutter was made because of its ability to create a rich user experience with a single codebase, simplifying system maintenance and evolution.  

The resource server, or BackEnd, was developed using Flask API, a lightweight and flexible microframework in Python. Flask was chosen for its simplicity in creating RESTful APIs and its ability to integrate with external libraries, such as machine learning tools. It also facilitates communication with the database and the organization of application routes. Additionally, the system was designed to ensure the security of user data. The login interface was implemented to validate credentials and verify access to the application, ensuring that only authorized users can use it.

The navigation flow begins with the login page, which checks the user's credentials before granting access to the application. This process is essential to ensure the security and privacy of the medical data processed by the tool.

\begin{itemize}
    \item Login Page: Contains two input fields (email and password) and a login button. After the backend verifies the credentials, the user is redirected to the main page of the application, provided the user is authorized.
    \item Text Input Field: On the text interaction page, the user inputs their question or comment related to the exam.
    \item Submit Button: Triggers the request to the backend, sending the text for processing.
    \item Response Area: Displays the response generated by the Ollama model.
\end{itemize}

The layout was designed with a focus on simplicity and usability, ensuring that interactions are direct and efficient. In Figure \ref{fig:aplicacaoweb}, the web application pages can be seen.
			\begin{figure}[h!]
    \centering
     \captionsetup{width=20cm} % Ajusta a largura da legenda
    \includegraphics[width=0.5\linewidth]{aplicacaoweb.png} % Insere a imagem
    \caption{Web Interface}
    \label{fig:aplicacaoweb}
\end{figure}

In figure \ref{fig:textoanalisado}, we can observe in detail the processing of a text. The analyzed text contained 100 characters. 

\begin{figure}[h!]
    \centering
    \captionsetup{width=8cm} % Ajusta a largura da legenda
    \includegraphics[width=12cm]{textoanalisado.png} % Insere a figura
    \caption{Text Analysis Interface} % Combina legenda e fonte
    \label{fig:textoanalisado} % Define o rótulo
\end{figure}

Our platform was designed to be fed with patient data as they use the platform or through public or private institutions that wish to acquire the system. Thus, the platform supports user registration, report creation, the patient's medical history, scheduled appointments, and other types of patient information.

Initially, we proposed feeding the platform with a dataset obtained from the National Library of Medicine website at the University of India. This dataset contains two tabular data files and 7,470 chest X-ray images in DICOM format, with different orientations (frontal and lateral). The tabular data includes information on various medical reports of patients and their respective diseases. In addition, this tabular data contains complementary information that aids in medical diagnosis, such as: images associated with the patient, indications, comparisons, findings, and impressions.

The database used has already undergone basic preprocessing to optimize performance during the processes carried out on it. The images were converted from DICOM to PNG, adjusting pixel values to the 0-255 range and resized to a limit of 2048 pixels on the shortest side.

\subsection*{3.2 - Agent Pipeline}

The pipeline for medical decision-making proposed by us consists of a collaborative approach involving multi-agent systems, covering four stages, as can be seen in Figure \ref{fig:processofdecision}:

Sending the query — Through our platform, the user will have access to our workflow that uses large natural language models to perform a complete decision-making process involving medical experts based on Agents. 
Query complexity check — The system classifies the user's query into two complexity levels: low and high, based on computational cost and context criteria.
Recruitment of Expert Agents — Based on the complexity level of the user's query, the system directs the PCC approach for simple problems or uses the MDT approach for complex problems.
Analysis and Synthesis — Simple queries use CoT ~\cite{wei2022chain, zhang2022automatic, sahoo2024systematic}, SC ~\cite{sahoo2024systematic}, and the combination of CoT-SC \cite{wang2022self} techniques. Complex queries, falling under the MDT approach, involve multiple LLM and SLM agents that form a consensus and make the decision.
Medical Decision-making — The final stage involves analyzing the user's query and providing a response based on clinical medical knowledge.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.85\textwidth]{Decision making.png}
\caption{Summarized process of decision making.}
\label{fig:processofdecision}
\end{figure}

\subsection*{3.3 - Multi-collaborative Agent: Medical Decision-making}

The entire flow can be followed in detail through the algorithm in algorithm  \ref{alg:MedicalDecision}.

Initially, the user submits an initial query via the chat prompt. The text is received by the API via a POST request and sent to the Agent Checker, the agent responsible for evaluating the complexity level of the query, as detailed in the section below. After determining the complexity of the query, it is directed to one of two approaches: PCC or MDT. Regardless of the selected approach, a new check is performed through the General Clinical Agent (CG), in order to simulate the process in detail as done in practice \cite{damarell2020general}, and is initially triggered to analyze the user's query. After the analysis, the General Clinical Agent, if capable, will respond to the query or forward it to the specialist team.

When the query is sent to the group of specialists, the process of recruiting experts begins through the Recruiting Agent. This agent is responsible for selecting specialists from each area based on the indications provided by the General Clinical Agent (CG). After recruitment, the MDT starts the collaborative discussion process, where specialists from each area will present their technical views on the problem until all or at least most of the expert agents reach a consensus. This process may occur over up to $N$ rounds.

For each round $n \in N$, the Agents $A_i$, where $i \in {1,2,3,...,M}$, participate in the discussions. If any Agent does not respond, the Moderator Agent will point out the need for the specialist to make their comments. During the discussion session, a turn-based speaking system is adopted. A turn ends when all specialists reach a consensus on an important aspect of the patient's query, although they may not be fully convinced about the specific problem. If there is no consensus and the agents agree to continue, a new round of discussion is initiated based on the previous conversations.

For each round, consensus within the MDT is determined by analyzing and comparing the experts' opinions, which is ensured by the Moderator Agent. In case of disagreement, the Moderator Agent analyzes the current round of conversation and previous discussions, pointing out the facts discussed earlier to each disagreeing agent. If the disagreement persists, the discussion round is closed. The Moderator Agent then summarizes all the information raised by the specialists and makes a final decision.

\begin{algorithm}
\footnotesize % Reduces font size
\caption{Medical Decision-making Pipeline}
\label{alg:MedicalDecision}
\makeatletter
\makeatother
    \begin{algorithmic}[1]

    \STATE \textbf{Function} \textsc{receiveQuery}(\textit{userInput})
    \STATE \hspace{1em} POST to API with \textit{userInput}
    \STATE \hspace{1em} Receive query text
    \STATE \hspace{1em} \textbf{Return} text
    \STATE \textbf{End}
    
    \STATE \textbf{Function} \textsc{evaluateComplexity}(\textit{query})
    \STATE \hspace{1em} \textit{COMPLEXITY\_LEVEL} $\gets$ \textsc{AgentChecker}(\textit{query})
    \STATE \hspace{1em} \textbf{Return} \textit{COMPLEXITY\_LEVEL}
    \STATE \textbf{End}
    
    \STATE \textbf{Function} \textsc{directQuery}(\textit{query}, \textit{LEVEL})
    \STATE \hspace{1em} \textbf{If} \textit{LEVEL} = "simple"
    \STATE \hspace{2em} \textit{APPROACH} $\gets$ PCC
    \STATE \hspace{1em} \hspace{0.5em} \textsc{GeneralClinicalAgent}(\textit{query})
    \STATE \hspace{1em} \textbf{Else}
    \STATE \hspace{2em} \textit{APPROACH} $\gets$ MDT
    \STATE \hspace{1em} \textbf{Return} \textit{APPROACH}
    \STATE \textbf{End}
    
    \STATE \textbf{Function} \textsc{GeneralClinicalAgent}(\textit{query})
    \STATE \hspace{1em} Analyze query
    \STATE \hspace{1em} \textbf{If} capable \textbf{then} Respond
    \STATE \hspace{1em} \textbf{Else} \hfill\Comment{\#Forward to specialist team (MDT APPROACH)}
    \STATE \hspace{2em} \textit{LIST} $\gets$ \textsc{recruitSpecialists}()
    \STATE \hspace{2em} \textsc{discussWithSpecialists}(\textit{LIST}, \textit{query})
    \STATE \textbf{End}
    
    \STATE \textbf{Function} \textsc{recruitSpecialists}()
    \STATE \hspace{1em} \textit{LIST} $\gets$ Recruiting Agent recruits specialists
    \STATE \hspace{1em} \textbf{Return} \textit{LIST}
    \STATE \textbf{End}
    
    \STATE \textbf{Function} \textsc{Discussions}(\textit{LIST}, \textit{query})
    \STATE \hspace{1em} \textbf{For} \textit{round} from 1 to N \textbf{do}
    \STATE \hspace{2em} \textbf{For each} \textit{Ai} in \textit{LIST}
    \STATE \hspace{3em} \textbf{If} \textit{Ai} commented \textbf{then} Add contribution
    \STATE \hspace{3em} \textbf{Else} Notify \textit{Ai}
    \STATE \hspace{3em} \textbf{If} consensus \textbf{then} Move to next
    \STATE \hspace{3em} \textbf{Else} \textbf{If} continue \textbf{then} New round
    \STATE \hspace{3em} \textbf{Else} End round
    \STATE \textbf{End For}
    \STATE \textbf{End}
    
    \STATE \textit{query} $\gets$ \textsc{receiveQuery}(\textit{userInput})
    \STATE \textit{level} $\gets$ \textsc{evaluateComplexity}(\textit{query})
    \STATE \textit{APPROACH} $\gets$ \textsc{directQuery}(\textit{query}, \textit{level})
    \STATE \textbf{If} \textit{APPROACH} = MDT \textbf{then}
    \STATE \hspace{1em} \textit{LIST} $\gets$ \textsc{recruitSpecialists}()
    \STATE \hspace{1em} \textsc{Discussions}(\textit{LIST}, \textit{query})
    \STATE \textbf{If} \textit{APPROACH} = PCC \textbf{then}
    \STATE \hspace{1em} \hspace{0.5em} \textsc{GeneralClinicalAgent}(\textit{query})
    
    \end{algorithmic}
%\label{alg:Medical Decision-making Pipeline}
\end{algorithm}



%\begin{figure}
%    \centering
%    \includegraphics[width=0.7\linewidth]{Figures/Figura6_Algoritmo.png}
%    \caption{Algorithm 1}
%    \label{fig:Algorithm 1}
%\end{figure}


\begin{figure}
\centering
    
    \includegraphics[width=0.95\textwidth]{Decision making - example.png}
   
\caption{Illustrative example of our collaborative Multi Agent approach to medical decision making using the Chest-X-Ray dataset (reports only).}
 \label{fig:multiagent}
\end{figure}



\subsection*{3.4 - Query Complexity}
To analyze the complexity of the medical query requested by the patient, a specialized verifier agent is triggered to direct the query to the appropriate Agentic workflow based on the complexity of the query. The verifier agent is based on an architecture of small natural language models. Tests were conducted using Qwen 2.5 (with its 3 billion parameter version) from Alibaba. The agent receives instructions on how the complexity of a medical query should be defined and classifies it according to the given instructions, assigning the patient's query to one of the complexity levels, as we can see in Figure \ref{fig:multiagent} :

\begin{enumerate}
    \item \textbf{Simple} — The patient's query is classified as simple when the patient has a predictable and well-organized medical history \cite{kilian2020reorganizing}, clear and widely known symptoms, the presence of visual data such as CT scans, X-rays, ultrasound, among others. Direct, clear, well-defined questions with a limited context (up to 50 words) during the consultation.
    \item \textbf{Complex} — The patient's medical query is assigned to this level due to vague symptoms presented during the consultation, multiple comorbidities or complex and rare conditions, patients with a tendency toward chronic diseases, lack of medical history, or disorganized data. Patients with communication issues, queries with large contexts (over 50 words).
\end{enumerate}


\subsection*{3.5 - Medical Agent Roles}
\textbf{General Doctor} — An agent based on large language models that acts as a general practitioner, initially analyzing the patient's query forwarded by the verifier agent and responding if it's a simple issue or directing it to the recruiter agent to recruit a team of specialist doctors if it is a complex question.

\textbf{Recruiter} — An agent based on small language models responsible for recruiting the team of specialist doctors after the prior analysis of the patient's query conducted by the general practitioner.

\textbf{Specialist Doctor} — These agents are specialists based on a specific context observed by the general practitioner, recruited by the recruiter agent. These specialist agents work in a group with other specialists, discuss among themselves, reach a consensus, and ultimately make the medical decision regarding the patient's query, complementing with the provision of technical reports on possible anomalies and pathologies related to the patient's query.

\textbf{Moderator} — This agent moderates discussions, summarizes information, and encourages opinions among the specialist agents in a multidisciplinary team setting. The moderator is mainly responsible in cases of disagreement among the specialists, mediating discussions divided into rounds and speaking turns. The moderator records the MDT meeting report, highlighting all the important information addressed during the meeting.


\paragraph{{Database Description}}
The databases used in this work were Chest-X-Ray and MedQA. The first dataset is a large two-part dataset, one with tabular data (medical reports) and images. The images in the dataset are from MRI scans of the thorax and the tabular data is the data we are using in this work. While the second dataset is a multi-choice question and answer dataset, this dataset is more focused on correct and well-defined answers while the first is more of a clinical case study. In addition, the first dataset contains very valuable information for clinical cases, which are divided into: patient problem, images (the tests carried out), indication, comparison, findings (of the tests) and impression. It's worth remembering that not all of these data are present for all patients, so there's a need to treat these data as well.


\paragraph{Data Cleaning}
A deeper investigation was conducted on the tabular data of the addressed dataset, identifying that it is more likely to contain noise. It was observed that additional preprocessing was required, beyond what was applied to the original dataset. To ensure that the Ensemble RAG works effectively, it was necessary to analyze in detail the files $indiana_reports.csv$ and $indiana_projections.csv$ of the dataset, in order to eliminate the noise present in these data. A traditional data cleaning approach was proposed, distributed in three stages: noise removal, normalization, and stopword removal.

\noindent\textbf{Noise Removal} — Removal of special characters (such as: \#!@\$\%¨\*\_+*), emojis, and meaningless repeated characters, such as: XXXX and x-XXXX.

\noindent\textbf{Normalization} — Conversion of all words in a sentence to lowercase letters.

\noindent\textbf{Stopword Removal} — Discarding common or repeated words that do not add real meaning to the sentence, such as “i”, “a", “and”, “of”, among others.

In addition, during the verifier agent call, data cleaning and structuring in JSON format are applied to facilitate the understanding of the patient’s query for the subsequent stages of medical decision-making.


\subsection*{3.6 - Validation Metrics}
Considering all stages of the platform and the processes that compose it, three types of evaluations were proposed to assess the system in detail, as well as its parts, at different stages of our platform: 1) System Evaluation e 2) RAG Evaluation.
%3) Evaluation of the response generated by the collaborative multi-agent autonomous approach.

\subsection*{3.6.1 - System Evaluation}
In the context of Big Data, the novelty and uncertainties inherent in many language model-based resources are inevitable. Therefore, it is crucial to maintain data privacy standards and social responsibility. Currently, large data platforms and models are trying to apply LLMLOps (Large Language Models Operations, a niche of MLOps, Machine Learning Operations), being heavily conditioned to keep these platforms running continuously and monitoring their life cycles \cite{wang2024towards}.

Thus, it is essential to analyze and measure some aspects of the usage of these large systems focused on natural language models, such as: monitoring the structure of the data provided for training these models, quality of LLMs, SLMs, RAGs, autonomous agents, and the entire life cycle of the models. These analytics enable developers and companies to gain valuable insights into the real-world usage of these technologies, ensuring the reliability and efficiency of LLMs through controlled and automated evaluations.

This work, therefore, focused on online evaluations, which are conducted in production environment scenarios exposed to the real world. This process involves automatic evaluators triggered by new log entries derived from live production. Online evaluation is excellent for reflecting the complexities of real-world use and integrates valuable user feedback, making it ideal for continuous performance monitoring. The following metric categories were analyzed: Cost metrics and Performance metrics.

\paragraph{\textbf{Cost Metrics}}
\begin{itemize} 
    \item \textbf{GPU Utilization} This metric aims to measure the GPU usage configured in the platform backend based on the total number of tokens and the number of responses received.
    \item \textbf{LLM Calls Cost} We decided measures the cost price of a commercial API (i.e., Anthropic, OpenAI) or when opting to purchase a machine with excellent configurations to run the full server.
    \item \textbf{Infrastructure Cost} This is an unconventional metric but quite important for the LLM serving deployment context. The cost of your application's infrastructure becomes a crucial factor in production contexts. The system addressed here has not been tested in production with numerous users using the platform simultaneously and continuously, as it should be. Therefore, this metric aims to demonstrate a base cost for maintaining a high-level LLMOps-based structure in an early production version (2/3 simultaneous users on the platform) for one month. Costs for vector storage (pinecone), backend server (AWS EC2 instance — t3.medium or machine local), storage (logs, patient data, results, LLM models) and network will be evaluated.
\end{itemize} 


\section*{Performance Metrics}

\begin{itemize}
    \item \textbf{Tokens per second:} 
    Esta métrica calcula o número de tokens gerados por segundo por um LLM/SLM durante sua resposta. Esses grandes ou pequenos modelos de linguagem geram tokens durante suas previsões. Analogamente, isso se assemelha à construção de uma sentença em uma linguagem humana, onde o token seria uma letra. Um conjunto de tokens agrupados pode formar palavras e, por consequência, uma sentença. A fórmula básica para calcular esse tempo é fornecida por:
    \begin{equation}
      T = \text{const.} + k \cdot N  
    \end{equation}
    Onde $T$ é o tempo total de resposta para uma consulta enviada a um LLM. A variável $\text{const.}$ depende de fatores como pesquisas de DNS, proxies, enfileiramento e processamento de tokens de entrada. O $N$ é o número de tokens de saída gerados pelo modelo durante uma solicitação. Já o $k$ mede o tempo necessário para gerar um token. Essa variável depende de vários fatores, como o tamanho do modelo, quantização, otimizações aplicadas e o ambiente de execução (hardware, interconexões, drivers CUDA, etc.).

    \item \textbf{Latency:} 
    Diferentemente da métrica anterior, a latência está focada em mapear principalmente o Time To First Token (TTFT), o Time Per Output Token (TPOT) e o número de tokens que serão gerados. 
    - O TTFT busca medir o tempo que o usuário leva para ver a saída do modelo após inserir sua consulta. Esse tempo depende do processamento do prompt e da geração do primeiro token de saída. 
    - O TPOT mede o tempo para gerar um token de saída para cada usuário que consulta o sistema. Essa métrica reflete como cada usuário percebe a "velocidade" do modelo. Por exemplo, um TPOT de 100 milissegundos/token equivale a 10 tokens por segundo por usuário, ou aproximadamente 450 palavras por minuto. 

    A equação para calcular a latência do LLM é dada por:
    \begin{equation}
        \text{latency} = TTFT + TPOT \cdot \text{Nº de tokens a serem gerados}
    \end{equation}
\end{itemize}


\subsection*{3.6.2 - RAG Evaluation}

For the evaluation of the RAG technique employed in this work, it was initially divided into two key conventional areas widely used in RAG systems and another alternative to conventional methods. Evaluations based on RAG systems are made through retrieval evaluation \cite{es2023ragas}, generation evaluation \cite{es2023ragas}, and finally, evaluation by an LLM-as-a-Judge \cite{zheng2023judging}.

\textbf{Evaluation Retrieval} is a crucial step to identify the potential and robustness of the system during the generation of responses based on the retrieval of documents stored in the database. Nowadays, two key metrics are employed to measure these aspects: context relevancy and context recall.

\begin{itemize}
    
    \item \textbf{Semantic Textual Similarity (STS)} This metric uses modern language models to generate vector representations (embeddings) to compare the semantic similarities between the retrieved documents and the user's query. The comparison between the embeddings is performed by calculating cosine similarity \cite{rahutomo2012semantic}. The similarity formula is as follows:
    \begin{equation}
        Cosine Similarity = \frac{\vec{v_1}\cdot\vec{v_2}}{||\vec{v_1}||||\vec{v_2}||}
    \end{equation}

\end{itemize}

\textbf{Evaluation Generation} is the second step after retrieving data from the database. Once the correct information is retrieved from the patient's query, it is necessary to evaluate not only if the natural language model was able to respond to the query, but also how clear and accurate the response is. Therefore, it is crucial to evaluate these key points.

\begin{itemize}
    \item \textbf{BLEU} is a metric that compares n-gram precision, a sentence of $n$ words, and applies penalties to sentences that are either too long or too short, known as the brevity penalty. It takes the geometric mean of the modified precision scores from the test corpus and then multiplies the result by an exponential brevity penalty factor, $BP$. First, the geometric mean of the modified n-gram precisions, $pn$, is calculated using n-grams up to length $N$ and positive weights $w_n$ summing to one. Then, let $c$ be the length of the candidate translation and $r$ the effective length of the reference corpus \cite{papineni2002bleu}. Initially, the calculation of $BP$ is formally described by:
    \begin{equation}
    \text{BP} =
    \begin{cases} 
    1, & \text{if } c > r \\
    e^{(1 - r / c)}, & \text{if } c \leq r
    \end{cases}
    \end{equation}
    Then, from the equation of $BP$, we obtain the equation for the BLEU metric:
    \begin{equation}
        BLEU = BP*exp(\sum_{n=1}^N W_nlogp_n)
    \end{equation}
    \item \textbf{Rouge} or Recall-Oriented Understudy for Gisting Evaluation is actually a set of metrics, widely used for automatic summary evaluation, meaning it focuses more on recall, the ability to retrieve n-grams present in the reference. The set of metrics compares a summary or translation automatically produced with reference summaries or translations (gold-standard high-quality, human-produced). For this work, ROUGE-1 (unigram), ROUGE-2 (bigrams), and ROUGE-L (longest common subsequence - LCS) were used \cite{lin2004rouge}. The calculation for ROUGE-N, for n-grams, is given by:

    \begin{equation}
        ROUGE-N = \frac{\sum_{S\in{ReferenceSummaries}} \sum_{gram_n\in S}Count_{match}(gram_n)}{\sum_{S\in{ReferenceSummaries}} \sum_{gram_n\in S}Count_{gram_n}}
    \end{equation}

    Meanwhile, ROUGE-L differs from ROUGE-N, as it is based on the longest common subsequence (LCS) between the model's output and the reference, i.e., the longest sequence of words shared between both. A longer shared sequence should indicate greater similarity between the two sequences \cite{lin2004rouge}. The formal notation of this metric is:
    
    \begin{equation}
        F_{lcs} = \frac{(1+b^{2})R_{lcs}P_{lcs}}{R_{lcs}+b^2P_{lcs}}
    \end{equation}
    
    \item \textbf{BERTScore} is a widely used metric based on modern natural language models, such as BERT\cite{devlin2018bert}. Unlike n-gram-based approaches like Rouge\cite{lin2004rouge} and BLEU\cite{papineni2002bleu}, it aims to analyze semantic representations by generating what we call word $embeddings$, obtained from pre-trained language models or fine-tuned models, in order to calculate the similarities between the generated response and the gold-standard \cite{zhang2019bertscore}. In this work, we chose to use the "bert-base-uncased" model and recorded the metrics calculated for recall, precision, and F1.

    \item \textbf{Perplexity} is a key metric widely used in probabilistic language models. It is used to evaluate how well the language model can predict a sequence of words in a response, i.e., it measures the perplexity or confusion of the model during the generation of the next token \cite{bengio2000neural}. Perplexity is calculated as follows:

    \begin{equation}
        Perplexity = 2^{H_(p)} = exp\left(\frac{-1}{N}\sum_{i=1}^{N}logP(w_i|w_1,...,w_{i-1})\right)
    \end{equation}


Where $P(w_i|w_1,...,w_{i-1})$ is the predicted probability for the word $w_i$ given the previous sequence.
\end{itemize}
\textbf{LLM-as-a-Judge} is one of the most commonly used unconventional techniques for evaluation, offering two key benefits: scalability and explainability \cite{zheng2023judging}. In the context of Big Data, large volumes of data are routinely stored in data lakes, making manual annotation unfeasible in these scenarios. Therefore, the need arises to train LLMs to act as judges and automatically assess how accurate and precise the responses from other LLMs are. Additionally, LLM-as-a-Judge provides not only scores but also explanations, making their results interpretable \cite{zheng2023judging}, as shown in \ref{fig:llm judge - example}.

As used in other stages of this work, we employ this technique to analyze the response from our RAG-based system in order to generate analytics about our system. It was proposed to use an LLM-as-a-Judge for single-response classification with a few-shot motivational prompt, meaning the LLM judge is asked to directly assign a score to a single response.  Few-shot prompts use examples to help improve classification response accuracy, usually between 20 and 30 examples to base the analysis on, and then compare the results provided by the prompt with those generated by the system to analyze if they align. After the analysis, it returns feedback, scoring from 1 to 4 on how accurate the response is when compared to the gold standard. Additionally, it calculates the Pearson correlation between the response generated by the LLM and the gold standard. The classification levels generated by the LLM Judge are:

\begin{enumerate}
    \item The system's response is awful: completely irrelevant to the question asked or highly biased;
    \item The system's response is mostly unhelpful: does not address some important aspects of the question;
    \item The system's response is mostly helpful: provides support but can still be improved;
    \item The system's response is excellent: relevant, direct, detailed, and addresses all the issues raised in the patient's query.
\end{enumerate}


\subsection*{3.7 - BackEnd}

The backend of the application was developed using Flask, a lightweight framework that allows for the rapid creation of APIs. One of the central features of the API is the endpoint $/$api$/$analyze$\_text$, which processes user text queries using the Ollama model in conjunction with a Retrieval-Augmented Generation (RAG) approach.

\begin{itemize}
    \item  Prompt Template: The model's prompt is configured to provide context and guide the model's behavior. The template includes information about the model's role, for example, "You are a very experienced cardiologist. You need to answer the question related to a patient's symptoms."

    \item  RAG (Retrieval-Augmented Generation): To improve the quality of the responses, we use a RAG approach. This technique combines document retrieval with text generation. The implementation loads a database of medical reports from the dataset indian$\_$projections.csv (an open collection of chest X-rays from Indiana University).
    \begin{itemize}
        \item  The data was pre-processed to optimize performance by converting DICOM images to PNG, adjusting pixel values to 0-255, and resizing the images to a maximum of 2048 pixels on the shortest side.
    
        \item  The pipeline loads these data as vectors using the embedding technique generated by the Hugging Face model "BAAI/bge-small-en." The embeddings are indexed with FAISS, an optimized library for vector search, enabling the application to quickly retrieve the most relevant documents based on the user's query.
    \end{itemize}
        
    \item  Query Pipeline: The pipeline performs retrieval and text generation in parallel. The most relevant information is retrieved from the database (using FAISS), and the context is then combined with the Ollama language model, which generates a final response based on the query.

    \item  Result Storage: All queries, along with execution time, the model used, and generated responses, are saved in a CSV file for auditing and further analysis.
\end{itemize}

\subsection*{3.7.1 - Credential Verification}

Before accessing the text analysis functionality with the Ollama model, users go through a verification process on the login page. This process ensures that only authenticated users can interact with the tool. The endpoint /api/login receives the user's credentials (email and password) via a POST request, and the data is verified against a user database stored in MySQL.

The password is verified against a stored hash, and, upon success, the user's information, such as ID and email, is returned in the response. If the credentials are incorrect, an error message is sent. Only after successful authentication can the user access the analysis and query functionalities using the Ollama model with RAG. This mechanism ensures security and access control for the tool.


\subsection*{3.7.2 - Architecture}

The backend architecture was structured in a REST model, and the Figure \ref{summarizedprocess} shows the graphical representation of the backend. The architecture follows a modular pattern, with specific functions for loading vectors, pipelines, and processing. The separation between the model logic, vectors, and endpoints contributes to a layered system, a characteristic of the REST model. The architecture is marked by several components: Flask API, Model Layer, Vector Database, Pipeline Process, and Result Storage.

\begin{itemize}
        \item  Model Layer (Llama): Performs the text analysis using a model called Llama. The model uses a prompt template that guides the model's response.
        \item Vector Database (FAISS + .csv Files): A repository that stores vectors for semantic similarity based on pre-processed data. The vectors are loaded from the configured path $(/$modelo$\_$llama$/$vectors$/$llama3$\_$indiana-reports$\_$.CSV$)$ and $(/$modelo$\_$llama$/$vectors$/$llama3$\_$medQA$\_$.CSV$)$, and the vectors generated with FAISS are saved.
        \item Pipeline Process: The pipeline process involves executing the text analysis with the LLM. The Vector Retriever retrieves relevant documents from the vectors, inserts the data into the template, and invokes the model, returning the response to the client.
\end{itemize}

\begin{figure}[h!]
    \centering
			\includegraphics[width=16cm]{fig8_appwebarquitetura.png}
		\caption{Summarized process of decision making}
\label{summarizedprocess}
\end{figure}



\subsection*{3.7.3 - Libraries and Tools Used}

Among the key libraries for the development of the endpoint responsible for the model integration, we have Langchain, FAISS, Hugging Face Embeddings, and Ollama.

\begin{itemize}
    \item  Langchain: Used to create modular and scalable pipelines for retrieval and text generation. Langchain allows for easy connection between the natural language model and document retrieval with FAISS.

    \item FAISS: A library developed by Facebook for high-performance similarity search. In this project, FAISS was used to index the embeddings of the medical reports and retrieve relevant documents quickly.

    \item Hugging Face Embeddings: The embeddings were generated using the "BAAI/bge-small-en" model, optimizing query processing in an environment with or without a GPU.

    \item Ollama: The natural language model that answers user queries, using a specific model (llama3) trained to generate high-quality natural language responses.
\end{itemize}


\section*{4 - Results}

The Results is divided into two parts. The first part addresses the results related to our pipeline, which integrates multi-Agents based on LLMs/SLMs collaborative approach, while the second part focuses on analyzing the robustness of the platform, from the perspective of the application infrastructure and the performance tests conducted in load and stress scenarios. The application snapshots were taken in a local development environment, using Python 3.11, version 0.3.12 of Ollama, and a client-server architecture based on Flask 3.0.3 for the backend and Flutter 3.22.3 for the frontend.

\subsection{4.1 - Agent-LLM Based Judge: Is it a Good Judge?}


The use of assistants based on large language models, initially proposed by Zheng \textit{et al.} \cite{zheng2023judging}, has emerged as a promising practice. These language models can perform various functions, ranging from conversational interactions to serving as medical specialists. Although there are several benchmarks for evaluating language models, most focus on closed questions with short answers, requiring well-defined reference standards. In situations where reference answers are unavailable, traditional evaluation metrics, which rely on the similarity between results and reference answers, such as ROUGE \cite{lin2004rouge} and BLEU \cite{papineni2002bleu}, become ineffective. The LLM Judge drastically reduces the need for human intervention in evaluations, enabling scalable benchmarks and rapid iterations. Additionally, LLM-based judges not only provide scores but also explanations, making their results interpretable, as illustrated in Figure \ref{fig:llm judge - example}.


\begin{table}[!ht]
\centering

\begin{tabular}{cclllll}
\hline
Judge                      & \multicolumn{2}{c}{Prompt}    & Pearson                              & RMSE & MAE & Hit Rate \\ \hline
\multirow{3}{*}{Mixtral}   & \multicolumn{2}{c}{short}     & \multicolumn{1}{c}{0.098} & \multicolumn{1}{c}{1.779} & \multicolumn{1}{c}{1.346} & \multicolumn{1}{c}{6/28} \\ 
                           & \multicolumn{2}{c}{zero-shot} & \multicolumn{1}{c}{0.174} & \multicolumn{1}{c}{1.618} & \multicolumn{1}{c}{1.293} & \multicolumn{1}{c}{\textbf{15/28}} \\
                           & \multicolumn{2}{c}{few-shot}  & \multicolumn{1}{c}{0.475} & \multicolumn{1}{c}{1.673} & \multicolumn{1}{c}{1.367} & \multicolumn{1}{c}{9/28}  \\ \hline
\multirow{3}{*}{LLaMA 3}   & \multicolumn{2}{c}{short}     & \multicolumn{1}{c}{0.485} & \multicolumn{1}{c}{1.617} & \multicolumn{1}{c}{1.314} & \multicolumn{1}{c}{11/28} \\
                           & \multicolumn{2}{c}{zero-shot} & \multicolumn{1}{c}{0.479} & \multicolumn{1}{c}{\textbf{1.614}} & \multicolumn{1}{c}{1.314} & \multicolumn{1}{c}{10/28} \\
                           & \multicolumn{2}{c}{few-shot}  & \multicolumn{1}{c}{0.425} & \multicolumn{1}{c}{1.652} & \multicolumn{1}{c}{1.339} & \multicolumn{1}{c}{13/28}  \\ \hline
\multirow{3}{*}{LLaMA 3.1} & \multicolumn{2}{c}{short} & \multicolumn{1}{c}{0.349} & \multicolumn{1}{c}{1.621} & \multicolumn{1}{c}{1.318} & \multicolumn{1}{c}{06/28} \\ 
                           & \multicolumn{2}{c}{zero-shot} & \multicolumn{1}{c}{\textbf{0.680}} & \multicolumn{1}{c}{\textbf{1.614}} & \multicolumn{1}{c}{1.307} & \multicolumn{1}{c}{12/28} \\
                           & \multicolumn{2}{c}{few-shot}  & \multicolumn{1}{c}{0.408} & \multicolumn{1}{c}{1.243} & \multicolumn{1}{c}{\textbf{0.886}} & \multicolumn{1}{c}{11/28} \\ \hline

\end{tabular}
\caption{Results table using Pearson correlation, RMSE, MAE metrics, for two types of prompts, using few-shot, zero-shot, and short prompts.}
\label{tab:LLM-judge results}
\end{table}


It was defined that the calculation of agreement between two types of judges represents the probability that randomly selected (but not identical) individuals from each type agree on a randomly selected question. That is, a question is randomly selected, and then two Agent-LLM Judge models, $n_i, n_j \in N$, where $N$ is the set of LLM Judge models, are randomly chosen. The agreement between these two Agent-LLM Judges is determined by the probability that both agree on their verdict for the selected question.

If the comparison is between an Agent-LLM Judge and human reviewers, the agreement is defined as the probability that an Agent-LLM Judge and a randomly selected human reviewer have the same verdict for a randomly selected question. Finally, the agreement between the human reviewers themselves is the probability that two randomly selected but distinct humans come to the same verdict for a randomly selected question, as described in the prompt in Figure \ref{fig:llm judge - example}





\begin{figure}[!ht]
    \centering

    \includegraphics[width=\linewidth]{Figures/LLM Judge - example.png}
    \caption{Evaluations, containing the classification and the explanation of the classification, by two reviewers regarding how correct the answer is in relation to the question. Additionally, the feedback from the Agent Judge containing the classification and the explanation of the classification.}
    \label{fig:llm judge - example}
\end{figure}

Therefore, to evaluate the robustness and accuracy of our LLM-based Agent Judge, we calculated the Pearson correlation metrics to measure how close or correlated the result of the final answer classification proposed by our Agent Judge was in relation to the correct answer, the hit rate of correct classifications (all correct classifications in relation to all classifications) and we measured the error related to the Pearson correlation values. To do this, we performed a calculation involving $MAE$ and $RMSE$, which measure the average difference between the correct classification and the classification predicted by our Agent and the other calculates the average difference between the predicted value and the actual value, respectively. 

The results of all results can be summarized through the following graphics presented on figure \ref{fig:maeeros}.


\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{Figures/Fig9_9graficos de Resultado.png}
    \caption{Graphic results of values for Pearson correlation and Hit Rate metrics and RMSE and
MAE errors}
    \label{fig:maeeros}
\end{figure}

The violin-shaped graph can provide important information about the behavior of the model during the evaluation of our LLM-based judgment agent. The element of the graph with the larger “belly” represents that there is a set of close values, representing low error variability (MAE and RMSE), accompanied by maximum and minimum values delimited by the lower and upper tail of the violin, which generally indicates an optimal signal when accompanied with the position of this element (violin) on the graph. 
Table \ref{tab:LLM-judge results} in question is able to provide valuable information about our model instructed through prompts using various prompt engineering techniques, such as: Few-shot, Zero-shot and Short. The figure provides a visualization of the distribution of the values obtained from the metrics calculations (Spearman correlation, MAE, RMSE).

Looking at the graphs in the figures, we can see a high degree of error variability when comparing the LLaMA3 and LLaMA3.1 models in relation to Mixtral. On the other hand, the Mixtral model showed a certain superiority in relation to Pearson's correlation metric, i.e. it can be inferred that the Mixtral model, despite obtaining a higher value for the correlation metric and, as observed in the table (cite table), the hit rate, shows a lot of inconsistency in its classifications, which can be very harmful in medical contexts. The LLaMA models as well as Mixtrall showed Pearson correlation values very close to 100\%, however, these were accompanied by slightly variable error values, which makes a model consistent, but not necessarily the best, because unlike Mixtral, the error values are higher. I believe that the great impasse in the choice of model and for our analysis would be to know which would be the best trade-off between consistency and trends.



\subsection*{4.2 - Intra-class correlation coefficient (ICC)
}

A quantitative intra-class analysis was performed, with the calculation of the Intra-class Correlation Coefficient (ICC), due to the central role it plays in the analysis of consistency and reliability between reviewers and the Agent-LLM Judge, especially when multiple individuals or systems evaluate the same issues. The goal is to quantify the degree of agreement and reveal the accuracy of evaluations in subjective judgment tasks, such as the analysis of answer quality, evaluation scores, or any other type of qualitative classification between the Agent-LLM Judge and the reviewers \cite{zheng2023judging, chaturvedi2015evaluation}.

The ICC analysis in this context seeks to answer fundamental questions such as: "To what extent do the model and the reviewer evaluate consistently?" and "Could the averages of these evaluations be trusted as a final judgment metric?". Each type of ICC provides distinct and complementary information about the consistency and reliability of the evaluations made.

\subsubsection*{Types of ICC analyses}

\noindent \textbf{ICC1 — Absolute Reliability of Individual Evaluators}

\noindent ICC1 measures the reliability of individual evaluators in absolute terms, i.e., it assesses how consistent each evaluator is individually when judging the items. In contexts where evaluators are unique and the rating scale is crucial for judgment, it is essential to determine whether the evaluators maintain standardization in their evaluations.
%Insight: A low ICC1 would indicate considerable variability between the model and the reviewer, suggesting that both are unreliable in maintaining a standard evaluation for each item. In contrast, a high ICC1 would imply high individual accuracy of the evaluators, pointing out that both (LLM and human) are aligned in individually judging each item with consistent standards.

\noindent \textbf{ICC2 — Reliability of Random Evaluators}

\noindent The ICC2 metric is used to estimate reliability when it is assumed that the evaluators are representative of a larger and random set of evaluators, that is, when the identity of the evaluators is not relevant to the consistency of the judgment. This metric indicates whether the observed consistency would be maintained if other reviewers replaced the LLM or the human.
%Insight: If ICC2 shows a low value, the reliability of the judgment strongly depends on specific evaluators, suggesting that consistency would not be replicable by other evaluators. High values, on the other hand, indicate that the consistency between the LLM and the human reviewer remains solid regardless of who evaluates, implying that the model could be replaced by other systems, or the human reviewer by another, without affecting the reliability of the evaluations.

\noindent \textbf{ICC3 — Reliability of Fixed Evaluators}

\noindent ICC3 is used when the consistency of specific evaluators (in this case, the Agent-LLM Judge and the human reviewer) is of interest and there is no expectation to replace these evaluators. By measuring reliability between fixed evaluators, ICC3 allows understanding the consistency of evaluations between these specific evaluators.
%Insight: High ICC3 values indicate good agreement between the LLM and the human reviewer, suggesting that both follow a reliable judgment pattern specific to the items. This is valuable in scenarios where the fixed evaluators will be the ones performing all future evaluations, as it shows that the system and the human evaluate in a complementary and consistent manner. A low ICC3 value would indicate the need to adjust the model or the reviewer’s training to increase joint reliability.

\noindent \textbf{ICC1k, ICC2k, and ICC3k — Mean Metrics of Evaluations}

\noindent The index $k$ in ICC1, ICC2, and ICC3 indicates the average ICC when there is a combination of reviewer evaluations, i.e., it is the scenario where the average of evaluations is used as a final measure, which can be useful in cases of high variability between individual evaluations.
%Insight: Valores altos de ICC1k, ICC2k e ICC3k sugerem que a média das avaliações é uma medida confiável e robusta, permitindo que a média entre o LLM e o revisor seja utilizada como uma avaliação final confiável. Isso tem implicações práticas significativas, pois, ao se utilizar a média das avaliações, um valor elevado dessas métricas indicaria que o LLM e o revisor humano podem ser usados em conjunto, fornecendo uma medida de consenso confiável.

% --------------------------------------------------------------------

%Significância Estatística (p-valores)

%Os p-valores associados a cada cálculo de ICC informam se os coeficientes obtidos são estatisticamente significativos. Valores de p < 0.05 indicam que a consistência observada entre os avaliadores é improvável de ter ocorrido ao acaso.
%Insight: A significância estatística reforça a confiabilidade das métricas obtidas. P-valores baixos confirmam que a concordância entre o LLM e o revisor humano é estatisticamente robusta, enquanto valores altos sugerem que a aparente consistência pode ser devida a fatores aleatórios. Em termos práticos, um ICC elevado com significância estatística elevada (p < 0.05) validaria a utilização conjunta do LLM e do revisor humano nas avaliações.

%Intervalos de Confiança (CI95%)

%Os intervalos de confiança (CI95\%) indicam a precisão dos valores de ICC calculados. Intervalos mais estreitos sugerem maior precisão e menor incerteza nos coeficientes de confiabilidade, enquanto intervalos mais amplos indicam variabilidade potencial.
%Insight: Intervalos de confiança estreitos aumentam a confiabilidade nos resultados, indicando que as métricas de ICC obtidas são representativas de uma consistência sólida entre os avaliadores. Intervalos amplos sugerem que a consistência observada pode variar, indicando a necessidade de mais dados ou de ajustes nos critérios de avaliação.

% --------------------------------------------------------------------


We performed the calculation by simulating 3 different LLMs, all with the same number of parameters (llama3 — 8B, llama3.1 — 8B, and Mixtral — 8B) and open-sourced, initially designing 3 prompt techniques: few-shot, zero-shot, and short. The ICC calculation was performed only on a subsample of 28 questions, which reflect the agreement between the human reviewers themselves (first and second). In this scenario, an Agent-LLM Judge is introduced to infer its judgment based on this subsample. One of the two human reviewers is removed, and a quantitative analysis is performed based on 6 intra-class scenarios (ICC1, ICC2, ICC3, ICC1k, ICC2k, ICC3k) between the Agent-LLM Judge and the human reviewer.

\begin{table}[H]
\centering

\begin{tabular}{ccllllll}
\hline
Judge                     & \multicolumn{1}{l}{Prompt} & ICC1 & ICC2 & ICC3 & ICC1k & ICC2k & ICC3K \\ \hline
\multirow{3}{*}{Mixtral}  & short                      & 0.136 & 0.164 & 0.182 & 0.320 & 0.370 & 0.400 \\
                          & zero-shot                  & 0.280 & 0.353 & 0.507 & 0.539 & 0.621 & 0.755 \\
                          & few-shot                   & 0.224 & 0.323 & 0.522 & 0.463 & 0.588 & 0.766 \\ \hline
\multirow{3}{*}{LLaMA3}   & short                      & 0.234 & 0.331 & 0.532 & 0.479 & 0.597 & 0.773 \\
                          & zero-shot                  & 0.244 & 0.340 & 0.551 & 0.492 & 0.607 & 0.786 \\
                          & few-shot                   & 0.218 & 0.321 & 0.531 & 0.456 & 0.587 & 0.772 \\ \hline
\multirow{3}{*}{LLaMA3.1} & short                      & 0.521 & 0.525 & 0.537 & 0.766 & 0.768 & 0.777 \\
                          & zero-shot                  & 0.246 & 0.343 & 0.560 & 0.495 & 0.611 & 0.792 \\
                          & few-shot                   & \textbf{0.599} & \textbf{0.597} & \textbf{0.589} & \textbf{0.817} & \textbf{0.816} & \textbf{0.811} \\ \hline
\end{tabular}
\caption{Quantitative intra-class analysis involving both reviewers and their respective feedback on answer quality.}
\label{tab:resultados icc - tradicionais prompts}
\end{table}

As shown in Table \ref{tab:resultados icc - tradicionais prompts} above, the Agent-LLM based on LLaMA3.1 achieved the best intra-class results in all scenarios involving ICC calculation. These indicators demonstrate the robustness of this model, making it fully viable to use it as a judge to assess complex questions. The first scenario analyzes the absolute reliability of individual evaluators, and the results obtained confirm the relatively high degree of reliability of our Agent-LLM and the synchronization maintained between it and the human reviewer. This reflects the accuracy and consistency of both in evaluating each question properly. The second scenario analyzes the reliability of randomly selected evaluators, showing high values for the LLaMA3.1 model. Unlike the Mixtral and LLaMA3 models, the order of who evaluates, whether it is the Agent-LLM Judge or the human reviewer, does not affect the consistency and accuracy of the results. The third important scenario analyzes the reliability of fixed evaluators, i.e., in settings where the evaluators and the Agent-LLM Judge do not change. As seen in the results table above, the results remained consistent with scenarios 1 and 2, emphasizing high reliability, consistency, and accuracy in this intra-class evaluation. Finally, the indicators from the last three scenarios confirm that the average of the evaluations is a robust measure, providing a reliable consensus metric.

Quantitatively speaking, these analyses have a significant impact on speed, reliability, and scalability, as an Agent-LLM Judge has proven to be more capable of evaluating questions properly than human reviewers specialized in the field. The two tables \ref{tab: analisa quantitativa intra-revisadores}, \ref{tab: analisa quantitativa intra-revisador-llm} below show the same calculations performed but comparing the two human reviewers based on the complete dataset, which contains 5660 samples of questions, answers, evaluations from each reviewer, and their explanations of their evaluations \cite{li2204using}. 



\begin{table}[ht]
\centering

\begin{tabular}{ccccccc}
\hline
Method         & ICC1 & ICC2 & ICC3 & ICC1k & ICC2k & ICC3K \\ \hline
Intra-reviewer & 0.483 & 0.486 & 0.493 & 0.651 & 0.654 & 0.660 \\ \hline
\end{tabular}
\caption{ICC calculation results for two human reviewers.}
\label{tab: analisa quantitativa intra-revisadores}
\end{table}

As shown earlier, these results demonstrate greater variability compared to those of the Agent-LLM Judge, proving that at least one of the two human reviewers is in disagreement with the other, showing inconsistency in the responses. Next, the calculation involving the following metrics: Pearson Correlation, RMSE, and MAE is observed.

\begin{table}[ht]
\centering

\begin{tabular}{clclc}
\hline
\multicolumn{2}{c}{Method}                                   & \multicolumn{1}{l}{Pearson} & RMSE                   & \multicolumn{1}{l}{MAE} \\ \hline
\multicolumn{2}{c}{\multirow{2}{*}{Intra-reviewer}} & \multirow{2}{*}{0.563}      & \multirow{2}{*}{1.165} & \multirow{2}{*}{0.765}  \\
\multicolumn{2}{c}{}                                         &                             &                        &          \\ \hline  
\end{tabular}

\caption{Results table using the Pearson correlation, RMSE, and MAE metrics, based on two human reviewers.
}
\label{tab: analisa quantitativa intra-revisador-llm}
\end{table}


\subsection*{4.3 - Motivation is all you need}
An additional study proposed in this work focused on improving the accuracy of LLM responses through prompt engineering, a technique that blends knowledge from four major areas that define prompt engineering: the combination of linguistics, creative writing, artificial intelligence, and cognitive psychology ~\cite{singh2024exploring, sahoo2024systematic}. In particular, during experimentation and the use of prompt engineering techniques ~\cite{singh2024exploring, sahoo2024systematic, wei2022chain, zhang2022automatic, wang2022self}, it was observed that the final response of Agent Judge based on LLM, the responses significantly improved in cases where creative prompts were used. Some results obtained from these experiments can be seen in Figure \ref{9graficos2}


\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\linewidth]{fig10_9graficos_2.png}
    \caption{Distribution of metrics results for Motivational technique of engineering prompt.}
    \label{9graficos2}
\end{figure}
\newpage

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|c|c|}
\hline
\textbf{Judge}            & \multicolumn{2}{c|}{\textbf{Prompt}}       & \textbf{Pearson} & \textbf{RMSE}  & \textbf{MAE}   & \textbf{Hit Rate} \\ \hline
\multirow{2}{*}{Mixtral}   & \multicolumn{2}{c|}{motivational}         & 0.268            & 1.628          & 1.364          & 9/28              \\ \cline{2-7} 
                           & \multicolumn{2}{c|}{few-shot + motivational} & \textbf{0.806}   & 1.686          & 1.408          & 12/28             \\ \hline
\multirow{2}{*}{LLaMA 3}   & \multicolumn{2}{c|}{motivational}         & 0.561            & \textbf{1.599} & \textbf{1.296} & \textbf{14/28}    \\ \cline{2-7} 
                           & \multicolumn{2}{c|}{few-shot + motivational} & 0.413            & 1.641          & 1.346          & 13/28             \\ \hline
\multirow{2}{*}{LLaMA 3.1} & \multicolumn{2}{c|}{motivational}         & 0.205            & 2.449          & 1.567          & 13/28             \\ \cline{2-7} 
                           & \multicolumn{2}{c|}{few-shot + motivational} & 0.678            & 1.604          & 1.289          & 12/28             \\ \hline
\end{tabular}
\caption{Results table using the Pearson correlation, RMSE, MAE, and Hit Rate metrics for two types of prompts: motivational prompt and motivational combined with few-shot prompt.}
\label{tab:resultados-few-shot-motivational}
\end{table}


As observed in the table \ref{tab:resultados-few-shot-motivational} above, excellent results were obtained, especially when considering the set of metrics from using the motivational prompt or the combination of few-shot + motivational prompt compared to the purely traditional prompts (zero-shot, few-shot, short) discussed earlier.

The use of creative writing can elevate the standards we know today in the development of generative artificial intelligence. Using motivation to inspire language models sparks interest, making it worthwhile to explore this prompt engineering technique in a detailed and thorough manner. It is worth conducting experiments testing various contexts of questions (QA, queries, clinical exams, among others) and other domains beyond healthcare.

\begin{table}[ht]
\centering

\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
\textbf{Judge} & \textbf{Prompt} & \textbf{ICC1} & \textbf{ICC2} & \textbf{ICC3} & \textbf{ICC1k} & \textbf{ICC2k} & \textbf{ICC3K} \\ \hline
\multirow{2}{*}{Mixtral}  & motivational            & 0.176   & 0.281   & 0.453   & 0.391    & 0.539    & 0.713    \\ \cline{2-8}
                          & few-shot + motivational & -0.346  & 0.056   & 0.529   & -3.358   & 0.150    & 0.771    \\ \hline
\multirow{2}{*}{LLaMA3}   & motivational            & 0.249   & 0.343   & 0.548   & 0.500    & 0.610    & 0.784    \\ \cline{2-8}
                          & few-shot + motivational & 0.223   & 0.324   & 0.531   & 0.463    & 0.590    & 0.772    \\ \hline
\multirow{2}{*}{LLaMA3.1} & motivational            & \textbf{0.271}   & \textbf{0.353}   & 0.533   & \textbf{0.527}    & \textbf{0.620}    & 0.774    \\ \cline{2-8}
                          & few-shot + motivational & 0.249   & 0.344   & \textbf{0.555}   & 0.499    & 0.611    & \textbf{0.789}    \\ \hline
\end{tabular}
\vspace{0.5em} % Espaço entre a tabela e o texto adicional
\caption{Quantitative intra-class analysis involving both reviewers and their respective feedback on answer quality using motivational and combination of motivational and few-shot prompts.}
\label{tab:intra-class-analysis}
\end{table}


However, unlike the ICC results obtained from responses generated by traditional methods, creative prompts (using motivation) showed some bias. Using the LLaMA3.1 model, we obtained the best results in all scenarios when compared to other models (LLaMA3 and Mixtral). Although we did not achieve unanimous results in all scenarios, both approaches, motivational and few-shot + motivational, were impressive, with a slight advantage for the motivational-only prompt, as it won 4 out of 6 scenarios, even though it did not outperform the few-shot-only approach. The response using the motivational prompt was more robust in the ICC1, ICC2, ICC1k, and ICC2k scenarios, while the combined prompt approach performed better in the ICC3 and ICC3k scenarios. Based on each metric, it can be concluded that the first approach is superior, with a high degree of reliability in the Judge-based Agent and maintaining synchronization between randomly chosen human reviewers, as also shown in the superior ICC2 scenario, confirmed by the ICC1k and ICC2k scenarios.

The second approach showed greater robustness only in the third and last scenario, which concerns when both human evaluators and the Agent are fixed. 
Overall, the first approach proved to be more robust, not only in the number of scenarios it won but also in real scenarios that more closely reflect ICC1 and ICC2, where reviewers are chosen randomly, as evidenced in Table \ref{tab:intra-class-analysis}.


\subsection*{4.4 - System Evaluation}
The evaluation of the system was based on some of the key factors to consider when trying to build a platform like this from scratch. We divided the analysis of this section into 3 parts, 1) Analysis of the use of the GPU during the execution of our pipeline, as well as monitoring its temperature and power consumption, 2) Analysis of the costs of the most popular commercial APIs for large language models and 3) Costs related to the operational infrastructure as a whole (use of data storage services, vector storage, network and others).

\subsubsection*{4.4.1 - GPU Utilization}

When creating a high-level platform, it is essential to consider a good GPU for running the experiments, especially when the idea is to maintain a local server. Therefore, we decided to list some important information analyzed in real time during a 2-minute pipeline execution section. Data was collected every second during the execution period, measuring factors such as: GPU usage (\%), GPU temperature (°C) and GPU consumption (W).

\begin{figure}[!ht]
    \centering
    
    \includegraphics[width=\linewidth]{Figures/gpu_performance_chart.png}
 \caption{GPU stress utilization during workflow of MDT pipeline}
    \label{gpuperf}
\end{figure}

Figure  \ref{gpuperf} shows in detail the behavior of our GPU that was used during the execution process. It is important to note that the GPU used was the Nvidia RTX 3070 TI 12GB model. With regard to temperature, we had sharp increases in temperature at approximately the same time as we had high levels of GPU consumption, which would be the expected behavior and, consequently, the power consumption figures also follow the same behavior. 

Although our GPU model isn't the best, we managed to run the entire pipeline, with the Ollama server running the Llhama3.1 model, integrating the agents into the entire workflow without using the entire GPU, only at a few points in the execution did we have execution problems (we got close to 100\%), which represents the start of the generation of responses between the agents. The behavior of the graph, in Figure \ref{fig:costs1}, visually resembles a step function, where the beginning represents the start of an agent's execution, hence the “almost periodic” behavior, as it is the behavior of an agent's response generation, then the time it takes to send the response from one to the other until it starts generating it, and so on until we finish the flow. Therefore, we were able to show that our entire pipeline can be run even with a video card that is not even in the top 5 video cards intended for this type of experiment, indicating the cost-effectiveness of our system as a whole and how optimized it is.

\subsubsection*{4.4.2 - Cost of LLM API calls}
One of the main analyses that we decided to include in this work was the cost of using commercial LLM APIs per call, after all, for any platform that uses a cloud hosting approach it has to worry about the costs of the most commercialized APIs in the area of generative AI, such as: Anthropic, OpenAI. In addition to the world's most popular commercial APIs, we decided to add to the analysis a Brazilian team that develops an API that also focuses on large language models, Maritacaca AI. Although we don't use any approach using commercial APIs in our current work, it is interesting to show the possible costs we would incur if we migrated to cloud hosting, for example.

Two different API usage scenarios were carried out, one involving LLMs and the other involving SLMs. For the first scenario, we will show three important analyses: the cost of 1 million input and output tokens, and a scalability analysis involving the cost per volume of accesses. The information involving token costs per input, both for LLMs and SLMs, is shown in the figure below:

\begin{figure}
    \centering
     
    \includegraphics[width=0.65\linewidth]{Figures/costs1.png}
   \caption{Costs of 1 million input tokens for different language model APIs}
    \label{fig:costs1}
\end{figure}

\begin{figure}
    \centering
   
    \includegraphics[width=0.65\linewidth]{Figures/costs2.png}
     \caption{ Costs of 1 million output tokens for different language model APIs}
    \label{fig:costs2}
\end{figure}

As seen in the figures above, the most expensive way to use APIs for large language models is to use Anthropic's Claude 3 Sonnet model, while the cheapest is also Anthropic's Claude 3 Haiku. Then for \$0.03 we have the second cheapest model, the value of the API for the use of our local model, Sabiá 3 from Maritacaca AI, while the other models (GPT-4o from OpenAI and Claude 3 Sonnet from Anthropic) are much more expensive when compared to these two. In the second graph of the image, which refers to the use of small language models, the GPT-4o mini model has a \$0.095 advantage over the cost of our native API. 

In the second analysis of Figure \ref{fig:costs2} involving the values of the output tokens, our native API took a big lead over the runners-up, at just \$2.35 the cost of the API per 1 million output tokens the Sabiá 3 model is provided by Maritacaca AI. In the second analysis involving the values of the output tokens, our native API took a big lead over the runners-up, at just \$2.35 the cost of the API per 1 million output tokens the Sabiá 3 model is provided by Maritacaca AI. While in the small language model version, OpenAI with the price of its API using the GPT-4o mini model took the lead over the price provided by the competitor, of \$0.2.

Finally, we have the scalability analysis, which shows in Figure \ref{fig:costs3}  us what the price of using the API would be on a daily, weekly and monthly basis, based on a scenario of the server in production.

In a production scenario, as exemplified in the graph, the most cost-effective option would be to use SLMs, in general, specifically the GPT-4o mini version. Because it is mini, this version of the model can reduce the quality and accuracy of the answers a little, but among the large language models, the most cost-effective is the GPT-4o mini. 

\begin{figure}[ht]
    \centering

    \includegraphics[width=0.65\linewidth]{costs3.png}
        \caption{API cost analysis from a production perspective}
    \label{fig:costs3}
\end{figure}
\newpage

\subsubsection*{4.4.3 - Infrastructure Cost}
An estimated cost for the infrastructure of the entire platform was calculated, considering a hybrid setup, cloud + local. The values of some elements shown in the table may vary depending on the location. Below, in Table \ref{tab:intra-cost}, we present an infrastructure cost for this platform assembled with the prices provided in Brazil. All elements that depend on the locality factor are marked with an “*” and are subject to price variation.

\begin{table}[!ht]
\centering

\begin{tabular}{cc}
\hline
Infrastructure          & Cost         \\ \hline
Cloud Machine (AWS EC2) & $\sim$840.00 \\
Local Machine \textbf{*} & 1,893.69     \\
API Calls               & $\sim$135.12 \\
Storage                 & 23           \\
Network \textbf{*}      & 22.60        \\
Vector Store            & 191.4        \\
Total cost              & 3,084.81     \\ \hline
\end{tabular}
\caption{Average cost of building a similar infrastructure}
\label{tab:intra-cost}
\end{table}

\section*{4.5 - RAG Evaluation}
We incorporated the Retrieve Augmentation Generation (RAG) technique into each responsible Agent that will diagnose the patient's query. Both specialist Agents and the General Clinical Medical Agent share this additional context, which assists in providing more accurate and precise responses, based on data extracted from real clinical scenarios and situations.

Initially, we propose incorporating this RAG technique using a set of two tabular datasets: Chest X-rays (version containing only reports) and MedQA. The BAAI/bge-small-en model was used for embedding generation, transforming the vector space of textual representations of the tabular data to the plane, then storing these embeddings through indexing with FAISS.

During data retrieval, similarity between the indexed tabular data (in embedding format, i.e., vector) and the user query is sought. This similarity search is performed through cosine similarity calculation. When the similarity is high, documents with high relevance scores are retrieved to be used as additional context during the response generation by the language model or Agent, in our case.

\subsubsection*{4.5.1 - Comparative Quantitative Analysis}

When analyzing the results, there is a significant impact on the correct choice of datasets; they must be completely aligned with their application domain or at least close to what is expected. The results of metrics can be observed in table \ref{tab:results quantitative metrics} Analyzing the metrics $Bert_{F1}$, $Bert_P$, $Bert_R$, they tend to vary from 0.389 to 0.538, with averages around 0.43, indicating consistent and moderate performance. The average Bleu score is 0.442, with a minimum of 0.133 and a maximum of 0.732, suggesting good match in some cases but also considerable variation, meaning low consistency. Rouge had an average of 0.326, with a lower precision than Bleu for this dataset. High values were observed for Diversity (with an average of 0.762), which is positive, as the model generates varied responses. Meanwhile, perplexity had a high average of $3.635e^4$, indicating higher difficulty in predicting sequences in clinical text.

A superior performance of the model was observed for the Chest X-rays dataset. When analyzing Bleu and Rouge metrics, superiority was noted compared to the other dataset. Additionally, higher perplexity was observed, indicating more challenges in modeling clinical text. While MedQA showed lower performance for Bleu and Rouge metrics, it had similar results in Bert metrics and higher diversity values (0.807 × 0.762), suggesting more varied responses despite the lower performance.

It is noticeable that when analyzing the diversity and perplexity metrics, there is a weighting between the fluency of the language model's responses and its variability due to its standard deviation. Therefore, despite the model being good at generating varied responses, the high perplexity indicates that it faces significant challenges in generating precise textual sequences.

Additionally, the model seems to work better with texts that share a similar structure to medical reports (Chest X-rays) but struggles with multiple-choice question formats (MedQA).

A later analysis would involve investigating other uses of Embeddings, whether based on neural networks, transformers, among others. Perhaps the impact of the embedding model size has also directly contributed to the performance of the quantitative metrics, as observed in the table above, because it influences both information retrieval and generation.

\begin{table}[!ht]
\centering
\resizebox{\textwidth}{!}{

\begin{tabular}{ccccccccc}
\hline
Method     & \multicolumn{1}{l}{Embedding model} & Dataset                                      & Min                       & Max                       & Mean                      & Mode                     & Median                   & Std                       \\ \hline
Bert\_F1   & \multirow{14}{*}{BAAI/bge-small-en} & \multirow{7}{*}{Chest X-rays (only reports)} & 0.404                     & 0.538                     & 0.438                     & 0.404                    & 0.439                    & 0.025                     \\
Bert\_P    &                                     &                                              & 0.389                     & 0.49                      & 0.433                     & 0.389                    & 0.434                    & 0.018                     \\
Bert\_R    &                                     &                                              & 0.403                     & 0.591                     & 0.443                     & 0.403                    & 0.444                    & 0.035                     \\
Bleu       &                                     &                                              & 0.133                     & 0.732                     & 0.442                     & 0.245                    & 0.249                    & 0.915                     \\
Rouge      &                                     &                                              & 0.000                     & 0.250                     & 0.025                     & 0.013                    & 0.015                    & 0.049                     \\
Diversity  &                                     &                                              & 0.637                     & 0.936                     & 0.762                     & 0.637                    & 0.748                    & 0.129                     \\
Perplexity &                                     &                                              & 1.605e\textasciicircum{}1 & 3.305e\textasciicircum{}6 & 3.635\textasciicircum{}4  & 1.286\textasciicircum{}4 & 1.352\textasciicircum{}3 & 2.919e\textasciicircum{}5 \\ \cline{1-1} \cline{3-9} 
Bert\_F1   &                                     & \multirow{7}{*}{MedQA}                       & 0.348                     & 0.614                     & 0.409                     & 0.348                    & 0.38                     & 0.073                     \\
Bert\_P    &                                     &                                              & 0.365                     & 0.786                     & 0.433                     & 0.365                    & 0.386                    & 0.104                     \\
Bert\_R    &                                     &                                              & 0.325                     & 0.609                     & 0.395                     & 0.325                    & 0.368                    & 0.078                     \\
Bleu       &                                     &                                              & 0.000                     & 0.857                     & 0.083                     & 0.001                    & 0.026                    & 0.176                     \\
Rouge      &                                     &                                              & 0.014                     & 0.398                     & 0.091                     & 0.014                    & 0.049                    & 0.113                     \\
Diversity  &                                     &                                              & 0.423                     & 0.981                     & 0.807                     & 0.423                    & 0.835                    & 0.123                     \\
Perplexity &                                     &                                              & 16.050                    & 4.346e\textasciicircum{}4 & 3.069e\textasciicircum{}3 & 51.539                   & 51.539                   & 4.408e\textasciicircum{}4
\end{tabular}


}
\caption{Analyze comparative of results obtained from generation metrics}
\label{tab:results quantitative metrics}
\end{table}


\subsubsection*{4.5.2 - Comparative Qualitative Analysis}

The qualitative results obtained are shown in Table \ref{tab:Results of qualitative metrics}. It can be seen from the table that there are values that allow a detailed qualitative analysis of the values we obtained for semantic textual similarity (STS) and average semantic textual similarity from the two datasets used for the RAG: Chest X-rays (its version containing only reports) and MedQA, using the BAAI/bge-small-en embedding model. The analysis considers statistical metrics such as minimum value (Min), maximum value (Max), mean (Mean), mode (Mode), median (Median) and standard deviation (Std).

\begin{table}[!ht]
\centering

\resizebox{\textwidth}{!}{
\begin{tabular}{ccccccccc}
\hline
Method                             & \multicolumn{1}{l}{Embedding model} & Dataset                                      & Min                       & Max                                & Mean                               & Mode                               & Median                    & Std                       \\ \hline
Semantic Textual Similarity        & \multirow{4}{*}{BAAI/bge-small-en}  & \multirow{2}{*}{Chest X-rays (only reports)} & \textbf{0.759}            & 0.776                              & 0.767                              & 0.76                               & \textbf{0.767}            & \textbf{0.004}            \\
Semantic Textual Similarity Avg\_g &                                     &                                              & \multicolumn{1}{l}{0.723} & \multicolumn{1}{l}{0.827}          & \multicolumn{1}{l}{0.762}          & \multicolumn{1}{l}{\textbf{0.827}} & \multicolumn{1}{l}{0.753} & \multicolumn{1}{l}{0.030} \\ \cline{1-1} \cline{3-9} 
Semantic Textual Similarity        &                                     & \multirow{2}{*}{MedQA}                       & 0.733                     & 0.775                              & 0.754                              & 0.733                              & 0.758                     & 0.011                     \\
Semantic Textual Similarity Avg\_g &                                     &                                              & \multicolumn{1}{l}{0.701} & \multicolumn{1}{l}{\textbf{0.846}} & \multicolumn{1}{l}{\textbf{0.773}} & \multicolumn{1}{l}{0.701}          & \multicolumn{1}{l}{0.763} & \multicolumn{1}{l}{0.036}
\end{tabular}

}
\caption{Analyze comparative of results obtained from retrieval metrics
}
\label{tab:Results of qualitative metrics}
\end{table}

\subsection*{4.6 - Prompt Templates}

This section lists the models used to generate responses from Agents/LLMs/SLMs in the system, as well as example prompts employing prompt engineering techniques.

\subsubsection*{4.6.1 - Medical Agents - General Practioner}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.65\linewidth]{Figures/prompt2.png}
    \caption{The Chain-of-thought prompt for analyzing the complexity of a consultation made
by a general practitioner.}
    \label{fig:prompt2}
\end{figure}

Figure \ref{fig:prompt2} illustrates the proposed process for developing an effective prompt for the analysis of the patient's complexity consultation by the practitioner clinician, highlighting the interaction between the technical criteria required for a very accurate classification. Using a strategic visual design, the figure organizes the main stages, from identifying the desired profile to formulating the messages of interest. Analysis of the graphic representation of this prompt allows us to see how the agent is instructed on how to act, think and behave. Furthermore, in particular, the figure representing the general practitioner's prompt has a part highlighted in blue. This section of the prompt represents the use of the Chain-of-Throught technique, which would be the instruction for the Agent to divide the problem into $n$ parts and solve the problem part by part.

\begin{figure}
    \centering
    \includegraphics[width=0.65\linewidth]{Figures/prompt3.png}
    \caption{The standard prompt for analyzing the complexity of a consultation made by a
specialist physician.}
    \label{fig:prompt3}
\end{figure}

In contrast with figure \ref{fig:prompt2}, the figure \ref{fig:prompt3} above it's almost the same prompt use. The difference between them is that standard approach, without CoT technique. 

\subsubsection*{4.6.2 - Specialist, Moderator, Recruiter}
In this section, we represent the systematic prompts used to maximize the quality of the system's response. The proposed prompts have been constructed to resemble the roles of Expert, Moderator, Recruiter and Verifier.

Figure \ref{fig:prompt4} \ref{fig:prompt5} illustrates the proposed process for creating an effective prompt for recruiting agents, based on instructions. As shown above, the figure shows the ability to translate recruitment requirements into an accessible and clear structure, ensuring that the prompt fulfills its objective of recruiting qualified experts for a given type of problem, considered difficult.
\begin{figure}[h!] 
    \centering
    \includegraphics[width=0.55\linewidth]{Figures/prompt4.png}
    \caption{Prompt used for the Moderator Agent}
    \label{fig:prompt4}
\end{figure}
\begin{figure}[h!] 
    \centering
    \includegraphics[width=0.55\linewidth]{Figures/prompt5.png}
    \caption{Prompt used for the Recruiter Agent}
    \label{fig:prompt5}
\end{figure}

\subsubsection*{4.6.3 - LLM Judge}

The performance tests were carried out using the tool Gatling 3.11.5, run in a Java 17.0.11 environment, with the gatling-maven-plugin-demo-java as the base for orchestrating the tests. The physical environment used for the tests consisted of a computer equipped with an AMD FX-8300 Eight-Core processor at 3.30 GHz, 16 GB of RAM, and an NVIDIA GeForce RTX 3060 graphics card with 12 GB of VRAM, ensuring optimized performance for graphical processing and machine learning tasks. 

Load testing was performed to assess the application's behavior under different usage scenarios, using the Ollama machine learning model with RAG. The focus was to evaluate the system's response under high demand, particularly when processing data related to medical images. The testing environment ensured a realistic simulation of operations in an intensive usage scenario, both in terms of data processing and communication between the Flutter interface and the Flask server.

The results demonstrated the effectiveness of the model in processing complex data, highlighting the system's robustness even in higher stress scenarios. For the tests, a 100-character prompt was used, and the average processing time for text analysis was around 155 seconds. The prompt used in the tests was "How would you interpret an abnormal ECG result with ST elevation in a 50-year-old patient?"

\subsubsection*{4.7 - Load Testing}

The load test was configured to simulate the behavior of the application under an increasing number of users. Initially, the scenario was set to gradually inject 5 users over 30 seconds, with each request containing a 100-character text for processing. The wait time between requests was set to 5 minutes, allowing for the analysis of the application's behavior in a controlled load scenario.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress1.png}
    \caption{Distribution of request times in load testing}
    \label{fig:stress1}
\end{figure}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress2.png}
    \caption{Request sampling table by endpoint in load testing}
    \label{fig:stress2}
\end{figure}

As shown in Figure \ref{fig:stress1}, we can visualize the distribution of response times for the requests, highlighting the system's stability while processing the requests. Next, Figure  \ref{fig:stress2} shows the request sampling table by endpoint, allowing us to observe the average and maximum times for each endpoint under load.


\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress3.png}
    \caption{Active user distribution graph in load testing}
    \label{fig:stress3}
\end{figure}

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress4.png}
    \caption{Histogram of response times during load testing}
    \label{fig:stress4}
\end{figure}

Figure  \ref{fig:stress3} illustrates the distribution of active users during the load test. We note that the test proceeded linearly without abrupt peaks, indicating that the application handled the expected number of users well. Figure \ref{fig:stress4} presents the histogram of response times, providing a detailed view of the variation in request processing times. This histogram shows a well-balanced distribution of response times, with small variations throughout the test, suggesting good application performance.

The results indicate that the application was able to handle the proposed load, processing the majority of the requests within the acceptable parameters for load testing.



\subsubsection{
4.8 - Stress Test}


In the first scenario, a number of users equivalent to twice the amount used in the load test was employed. In this experiment, 10 users were injected over 30 seconds, with each request containing a 100-character text for processing and a 5-minute wait time per request. This test was designed to simulate overload situations and evaluate how the application responds when subjected to a higher number of simultaneous users.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress5.png}
    \caption{Distribution of request times in stress testing}
    \label{fig:stress5}
\end{figure}

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress6.png}
    \caption{Request sampling table by endpoint in stress testing}
    \label{fig:stress6}
\end{figure}

In Figure \ref{fig:stress5}, it is possible to analyze the distribution of response times for the requests, highlighting a distinct behavior compared to the previous load test. Figure \ref{fig:stress6} shows the request sampling table by endpoint, where the average and maximum times for each endpoint can be observed under this stress condition.

Figure \ref{fig:stress7} shows the distribution of active users during the stress test. Unlike the load test, in this scenario, it is possible to observe a more abrupt increase in the number of users, which allowed for an accurate assessment of the impact of this growth on the application's performance.The histogram of response times, shown in Figure \ref{fig:stress8}, provides a detailed view of the variations in request processing times. The difference compared to the load test becomes evident here, with significantly higher response times, indicating a possible bottleneck in processing.


\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress7.png}
    \caption{Active user distribution graph in stress testing}
    \label{fig:stress7}
\end{figure}

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth]{Figures/stress8.png}
    \caption{Histogram of response times during stress testing}
    \label{fig:stress8}
\end{figure}

These results highlight the points of attention that the application should consider in order to handle extreme stress scenarios, where the increase in the number of simultaneous users can lead to processing queues and prolonged response times.

\section*{5 - Conclusions and Future Work}
\

This thesis addresses the development of an innovative computational tool for monitoring and managing medical data, with an emphasis on interoperability and information security. The proposed web platform integrates data from various sources using the Fast Healthcare Interoperability Resources (FHIR) standard, aiming to optimize the analysis of medical records while providing automatic alerts and preventive recommendations. The adoption of FHIR was crucial to ensure efficient communication between hospitals, clinics, and remote monitoring devices, enabling faster and more accurate health data management.

Although the platform demonstrated good performance in controlled tests, stress testing revealed the need for improvements in the infrastructure, especially in real-time high-volume data scenarios, such as those generated by remote monitoring devices. The efficiency of communication between systems and the scalability of the system need to be improved to meet this growing demand. AI was successfully applied to the analysis of medical records, using models like BERT for text processing and ResNet for image analysis. However, response time emerged as a challenge, especially under high data load, and the efficiency of the models must be improved to ensure real-time responses.

The platform strictly follows the requirements of the General Data Protection Law (LGPD), ensuring the security and control of patients' personal data. However, tests revealed vulnerabilities in communication between systems, particularly under high load scenarios. In response, the implementation of more robust security protocols and optimization of the infrastructure are essential to protect sensitive data across multiple environments and platforms, ensuring compliance with current regulations.

The platform's interface was designed with a focus on usability, aiming to facilitate data management for both healthcare professionals and patients. However, improvements are needed in mobile device optimization and user experience personalization to increase patient adoption, especially considering the diversity of profiles and technological skills. The user experience can be decisive in the acceptance of the platform, and a more accessible and intuitive design is necessary.

As for future work, the research suggests the implementation of network optimization techniques and model compression to improve the scalability and efficiency of the platform, as well as enhance real-time response. The application of multimodal AI, although effective, needs to be refined to minimize latency in high-demand environments. Furthermore, strengthening security measures and adapting the interface for mobile devices and different user profiles are essential for ensuring the platform's continuous adoption. The combination of AI, interoperability, and multimodal models has the potential to transform medical data management, but significant advances in scalability, security, and user experience are required.


\bibliography{sample}

%\noindent LaTeX formats citations and references automatically using the bibliography records in your .bib file, which you can edit via the project menu. Use the cite command for an inline citation, e.g..


\section*{Acknowledgements }

The authors are grateful to King Saud University, Riyadh, Saudi Arabia for funding this work through Researchers Supporting Project Number (RSP2025R18)..

\section*{Author contributions statement}

M.C.R. conceived the experiments and supervised the project. R.R.S. and D.S.S. conducted the experiments. M.R.H. and M.M.H. analyzed the results. P.L.A.L. and V.H.C.A. reviewed and edited the manuscript. All authors reviewed the manuscript.

\section*{Data Availability Statement}

The datasets used and analyzed during the research are available in the repository at the links https://www.kaggle.com/datasets/raddar/chest-xrays-indiana-university/data/indianaprojections.csv and https://github.com/jind11/MedQA.


\end{document}