Predictive analytics with the integration of Electronic Health Records (EHR) and wearable data stands at the forefront of a transformative shift in healthcare [1]. This approach not only enhances disease prediction for individual patients but also scales to provide insights for public health monitoring and epidemiology. At the heart of this transformation is the potential use of advanced models, like foundational Large Language Models (LLMs), that can analyze complex, multimodal data to reveal underlying health risks [2–4].

The use of EHR and wearable data integration for predictive analytics relies on the ability to ingest and interpret complex data from multiple sources. Wearable devices offer continuous, real-time monitoring of physiological parameters, such as heart rate, activity, and sleep quality, while EHRs provide a longitudinal view of a patient’s health, including lab results, diagnoses, and clinical notes [1]. Integrating these data streams offers a unique opportunity to build predictive models that can identify at-risk patients early, enabling timely and personalized interventions. By combining the continuous flow of real-time wearable data with the depth of historical EHR information, predictive analytics provides a comprehensive health profile for each patient, enhancing proactive healthcare models that emphasize prevention.

The development of foundational LLMs has shown promise for various healthcare applications, including disease prediction. These models have the capability to process and synthesize multimodal, individual-specific data, ranging from textual clinical notes to time-series data from wearables, giving them a nuanced understanding of a patient's potential health risks [2]. LLMs can be fine-tuned to adapt their responses based on a patient’s risk profile, integrating contextual data such as lab values, genetic predispositions, and other indicators [2–7]. This approach goes beyond traditional models by leveraging the extensive prior medical knowledge encoded within LLMs, which can identify connections between conditions, such as recognizing that "hypertension" and "high blood pressure" are synonymous and understanding the clinical significance of various lab ranges.

Unlike classical predictive models, which often require explicit coding and complex data handling to incorporate prior knowledge, LLMs offer a more flexible and adaptive framework. Through prompt engineering, they can dynamically incorporate new information, making them highly versatile across a wide range of tasks and domains. Additionally, LLM-based architectures are well-suited for handling diverse and incomplete healthcare datasets. Traditional models often struggle with missing data, requiring imputation or exclusion. In contrast, LLM-based systems can work effectively with partial inputs by focusing on available information, without needing architectural modifications. This makes them ideal for modeling heterogeneous data sources such as real-time wearable signals, structured EHRs, and even unstructured clinical notes.

Predictive analytics using LLMs still faces the complex challenge of integrating diverse health data types. Health data is often stored in varied formats and modalities, from numerical lab results to visual data such as medical imaging. Traditional methods have long sought to address this through multimodal fusion techniques, such as logistic regression models that integrate several data inputs or fusion architectures combining features from disparate sources. However, these approaches often lack the flexibility and scalability of LLMs. For instance, traditional models require cumbersome methods to incorporate prior knowledge or to deal with fuzzy, non-standardized labels [2]. The challenge lies in how to efficiently integrate these data into a format that LLMs can process while retaining its richness and clinical relevance. In healthcare, crucial indicators of disease often span multiple data modalities, making it necessary to combine wearable sensor readings with diagnostic information for accurate prediction.

Recent advances in multimodal LLMs have explored integrating complex data types such as images and time-series signals, though applications in healthcare are still emerging. This strategy enriches the model's contextual capacity, as shown in recent studies where models like HealthAlpaca improve predictions by combining contextual and physiological data [3, 8]. Another example is HeLM, a framework that uses modality-specific encoders to combine clinical data such as demographic and laboratory data, with spirogram curves, which are used to evaluate lung function, for disease risk prediction [2]. Through modality specific encoders, HeLM creates unified representations suitable for LLM processing, demonstrating the potential of large models to handle diverse biomedical inputs.

Building on similar principles, we use modality-specific encoders tailored for more complex inputs, longitudinal EHR and wearable data, to generate unified representations for LLM-based prediction. This enables integration of detailed medical history with real-time physiological patterns to support dynamic health modeling. Figure 1 shows a high-level overview of our proposed system, which integrates structured EHR data, wearable sensor data, and optionally, other available information through modality-specific encoders and an LLM for personalized disease prediction. Our approach is akin to enhancing a language model's vocabulary with new, specialized words and retraining it to understand them in context. Here, the EHR and wearable embeddings act as newly introduced "words" that enrich the model's understanding of health-related information. By embedding these novel data types into the shared embedding space, we effectively retrain the model to interpret these embeddings with the same familiarity as its existing knowledge base. This allows the system to capture complex interactions and nuanced meanings between clinical history and real-time physiological signals, thereby deepening the model's predictive and contextual capabilities. This process empowers the system to "learn" from both structured EHR data and dynamic activity patterns, much like adding new vocabulary terms expands a language model's understanding and contextual relevance.

For this study, we utilized data from the UK Biobank, a large-scale biomedical database that contains in-depth genetic and health information on approximately 500,000 participants from across the United Kingdom [9]. This extensive resource provides a rich dataset for health research, including longitudinal EHR and accelerometer data, enabling comprehensive analysis of both historical health events and real-time movement patterns. For each individual, the dataset included a sequence of diagnosis codes across health events, capturing a detailed timeline of clinical diagnoses, treatments, and health outcomes. In addition, accelerometer data was collected for a subset of participants over a continuous one-week period. The accelerometer data was preprocessed to extract minute-by-minute step counts, forming a structured time series representation of each participant’s activity. In total, we identified 69,326 individuals with both activity and EHR data available meeting the criteria for our model training and evaluation. Among these, approximately 3% (or 2,193 participants) had recorded mortality events.

Our diagnostic system utilizes a multimodal, end-to-end training approach designed to integrate EHR and wearable time-series data to improve disease prediction. By combining historical health records with real-time physiological data, the system leverages both long-term and immediate health indicators, capturing a comprehensive view of patient health. The method employs distinct encoding models: one tailored to the structured nature of EHR data, capturing longitudinal health patterns, and another optimized for wearable data, identifying activity-based trends relevant to disease risk. The latent representations from each encoder are then integrated into a shared embedding space, along with contextual information, to enhance prediction accuracy. This section details each component of the system, from input data processing to disease probability prediction, demonstrating how each element contributes to a robust, adaptive model for personalized disease assessment, as illustrated in Fig. 2.

The model is trained end-to-end using a weighted binary cross-entropy loss function, designed to handle class imbalances often present in disease prediction tasks. This loss function adjusts the penalty for misclassification by applying higher weights to the minority class (e.g., true disease cases), encouraging the model to learn from both positive and negative cases effectively. Training proceeds until the loss function converges. During training, we monitor AUROC (Area Under the Receiver Operating Characteristic) in addition to minimizing loss, as it is a critical metric for evaluating model performance in clinical prediction. This approach helps ensure the model performs well across key metrics as training progresses.

The model is trained jointly on both EHR and wearable data. To evaluate the contribution of each modality, we assess performance under three test-time configurations: (1) using both modalities, (2) using only EHR data, and (3) using only wearable data. We also compare the model’s performance to baseline models, as detailed in the Results section.

This section presents the model's performance under three test configurations: (1) using both EHR and wearable activity data, (2) using only EHR data (with activity data replaced by zeros), and (3) using only activity data (with EHR data replaced by zeros). Additionally, these results are compared to single-modality encoder models (Med-BERT and Activity Encoder) each paired with a simple classifier and trained specifically on EHR or activity data alone to assess the standalone predictive power of each data modality.

When evaluated with both EHR and activity data, the model achieved the results shown in Table 1.

These high accuracy and AUROC values demonstrate strong predictive performance, indicating the benefit of combining both EHR and wearable data to capture comprehensive health insights.

In this setup, the activity data was replaced with zeros, effectively isolating the EHR data's contribution. The results are shown in Table 2.

These results indicate that EHR data alone provides substantial predictive power, achieving accuracy and AUROC values close to the full model’s performance. This outcome underscores EHR data's importance as a primary source of health insights.

In this configuration, EHR data was replaced with zeros, leaving only wearable activity data for predictions. The model performance is shown in Table 3.

To contextualize the above results, individual encoder models (Med-BERT and Activity Encoder) were evaluated with simple MLP layers on their latent spaces. This approach provides baseline performance metrics for each modality used independently, offering a point of comparison for the multimodal configurations.

The Med-BERT encoder shows relatively high accuracy and AUROC, as shown in Table 4, which are comparable to the results of the model evaluated with only EHR data in the multimodal setup (Table 2). This result highlights the strength of EHR data as a robust predictor on its own.

The activity encoder alone performs similarly to the activity-only configuration of the multimodal model, as shown in Table 5 and Table 3, respectively. Despite lower accuracy and AUROC than the EHR modality, it demonstrates the utility of activity data in providing predictive insights, particularly for dynamic health changes.

While the accuracy and AUROC values for activity data alone are lower than those for EHR data, they still demonstrate the unique predictive contributions of wearable data, especially for capturing dynamic physiological patterns.

In this work, we developed a multimodal LLM-based framework that integrates EHR and wearable data for disease prediction. The model was trained jointly on both data types and evaluated under different input configurations to assess the contribution of each modality. We also compared the model's performance against single-modality baselines to better understand the predictive strength of each input source. The results highlight key findings regarding model performance across these configurations and the independent predictive power of each modality.

The model achieved the highest accuracy and AUROC when combining both EHR and activity data, underscoring the advantage of integrating historical health records with real-time physiological signals. This multimodal integration provides a more comprehensive and nuanced assessment of disease risk than either data source alone. By capturing both long-term health patterns and real-time activity, the model offers a fuller view of patient health. For example, a patient with cardiovascular indicators in their EHR may also exhibit low physical activity and irregular sleep in their wearable data. The encoders process these modalities separately and integrate them to produce a cardiovascular risk prediction, which can be used by an LLM to generate tailored recommendations such as increased activity or dietary adjustments.

The model’s architecture demonstrates resilience by adapting to different data availability scenarios. A notable finding is that when either EHR or activity data is zeroed out, the model’s performance aligns closely with that of single-modality encoder models. This suggests that each data type retains its predictive power independently, even within a multimodal framework. The model’s ability to preserve performance when only partial data is available highlights its adaptability, a practical advantage in healthcare where complete data may not always be available.

The EHR-only configuration yielded relatively high accuracy and AUROC values, comparable to the full model’s performance. This result suggests that EHR data provides foundational predictive insights due to its comprehensive historical information. The model’s strong performance with EHR alone underscores the critical role of structured health records in disease prediction.

Although activity data alone produced lower accuracy and AUROC compared to EHR data, it still provided valuable predictive insights, capturing real-time physiological and behavioral patterns. The performance of the model evaluated with only activity data closely matches that of the activity-specific encoder model, confirming the wearable data's distinct contributions. To further quantify its impact, we analyzed the differences in predictions between the full multimodal system (Configuration 1: EHR and Activity Data) and the EHR-only system (Configuration 2). Specifically, we investigated whether the changes in the full model's predictions relative to the EHR-only model aligned with the predictions from the activity-only system (Configuration 3). Our analysis revealed that 79.8% of cases exhibited this alignment, indicating that the changes in predictions from the full model (both increases and decreases) corresponded with the activity-only predictions. This suggests that the activity data meaningfully contributes to the adjustments in risk predictions in the full system, either enhancing or attenuating the predicted risk based on the activity patterns. Additionally, we observed a moderately strong positive correlation (r = 0.771) between the prediction differences (Configuration 1 vs. Configuration 2) and the predictions from the activity-only system (Configuration 3). This correlation underscores the significant role that activity data plays in enriching the model's predictive capabilities when combined with EHR data.

Through the end-to-end training approach, the EHR encoder and the activity encoder learn collaboratively, with each encoder adapting to shared features to provide complementary health insights. By leveraging these advanced encoders, the model effectively combines clinical and activity data with additional contextual inputs, resulting in a robust, accurate, and personalized system for disease prediction.

Our system can also incorporate other available information and patient‑specific factors, such as genetic history and lifestyle choices, into prompts, making disease prediction and health management more precise. Building on these predictions, the same multimodal backbone or a separate conversational LLM operating on its outputs, can be used to generate actionable insights and personalized recommendations. These models can analyze complex data patterns across multiple sources to support clinical decision‑making and enhance patient care. For example, LLM‑based systems like PhysioLLM engage users in personalized health conversations, utilizing physiological and contextual data to deepen individuals’ understanding of their health [14]. Another example, the Personal Health Insights Agent (PHIA), leverages LLMs, code generation, and web search to interpret wearable health data, providing accurate responses to health‑related questions and empowering users to develop data‑driven wellness strategies [15]. Our proposed multimodal framework can adopt these capabilities: by examining the aggregated multimodal data, comprehensive risk scores, and the model’s attention weights, it can reveal the relative importance of each data source, surface key cross‑modal interactions, and convert those findings into tailored, patient‑specific recommendations. Operating across scales, it can flag high‑risk groups for targeted public‑health measures while simultaneously delivering individualized guidance, thereby advancing proactive, data‑driven care for clinicians and patients.

In summary, the analysis illustrates the robustness and flexibility of the multimodal approach, which captures the complementary strengths of EHR and wearable data. While each modality independently contributes predictive value, their integration provides a fuller picture of patient health, enhancing the potential for proactive and precise disease risk assessments. The model’s ability to perform well with partial data further highlights its resilience, making it suitable for diverse healthcare applications focused on personalized, preventive care. By bridging the gap between clinical data and real-world lifestyle behaviors, the system represents an advancement in preventive healthcare, offering healthcare providers and patients deeper, more actionable insights for disease management and health risk mitigation.

The data used in this study were obtained from the UK Biobank under approved application number 94364. Access to the UK Biobank dataset is restricted to approved researchers who meet the access criteria set by the UK Biobank Access Management System. Due to legal and ethical considerations, individual-level data are not publicly available. Interested parties may apply for access directly through the UK Biobank portal (Apply for access - UK Biobank, https://www.ukbiobank.ac.uk/enable-your-research/apply-for-access).

This research was conducted using the UK Biobank Resource under Application Number 94364. UK Biobank received ethical approval from the North West Multi-centre Research Ethics Committee (MREC; approval number 11/NW/0382). Under UK Biobank's Research Tissue Bank (RTB) approval, researchers using approved applications do not require separate institutional ethical approval. All participants provided written informed consent at recruitment. All methods were carried out in accordance with relevant guidelines and regulations, including the Declaration of Helsinki. All experimental protocols were approved by UK Biobank's Access Management System under the aforementioned application.