For rare inherited retinal diseases (IRDs), determining the mode of inheritance (i.e. autosomal versus X-linked inheritance) is crucial for providing accurate genetic counseling, guiding family planning, and predicting disease progression. While genetic testing is now routinely performed for many IRD patients, determining inheritance patterns through mutational analysis continues to present significant challenges (Britten-Jones et al., 2024; Xu et al., 2014). The most accessible and cost-effective option, whole-exome sequencing (WES), focuses on the protein-coding region, but coverage is limited and may miss disease-causing variants in non-coding regions (Burdick et al., 2020; Ross et al., 2020). Though more comprehensive, whole-genome sequencing (WGS) has other limitations, such as difficulty in detecting certain types of variants and higher costs (Marian, 2012). Furthermore, the diagnostic yield of next-generation sequencing for patients with RP ranges from 50–75% (Lynn et al., 2024; Consugar et al., 2015), and approximately half of non-syndromic RP patients are cases with no family history of disease, which complicates the determination of inheritance patterns (Jin et al., 2008). Although determining inheritance patterns through sequencing and family history remains difficult, research indicates that distinct genetic inheritance patterns may result in subtle morphological features that are unfortunately challenging to detect by the human eye alone (Currant et al., 2021; Ortin Vela et al., 2024).

Artificial intelligence (AI) enhanced imaging tools may offer a direct, non-invasive alternative. In clinical medicine, contemporary deep learning (DL) models have achieved disease diagnostic accuracies comparable to those of experienced physicians (Zhou et al., 2023; Men et al., 2023; Hwang et al., 2023). More remarkable, some DL models can further discern sub-visual features—such as inferring biological sex or age from fundus photographs—that elude human observers (Berk et al., 2023; Korot et al., 2021; Nusinovici et al., 2022). This success, however, has come at the cost of prodigious data demands. In response, the field is converging on foundation-model strategies—both general and task-specific—that couple resource efficiency with strong cross-task generalization (Gani et al., 2020). General-purpose vision encoders typically require 142–300 million heterogeneous images to attain competitive performance (Oquab et al., 2023; Dosovitskiy et al., 2020), whereas task-tailored variants achieve state-of-the-art (SOTA) accuracy with 1.6–3.4 million curated ophthalmic images (Zhou et al., 2023; Shi et al., 2024; Qiu et al., 2024). Transfer learning techniques can further shrink data requirements by orders of magnitude to 70,000 to 100,000 images (Yang et al., 2024; Cohen et al., 2025). While these AI techniques have empowered detection and grading of common retinal conditions (Silva-Rodriguez et al., 2024; Sevgi et al., 2024), such as diabetic retinopathy (DR) (Men et al., 2023) and age-related macular degeneration (AMD) (Du et al., 2024), reduced data thresholds still pose a prohibitive barrier for fields like IRDs, where annotated datasets seldom exceed a few hundred cases owing to low prevalence and fragmented data stewardship (Decherchi, 2021).

One promising strategy for curbing data demands is to leverage generative AI, which can augment existing datasets with high-fidelity synthetic images. (Chaurasia et al., 2024; Chen et al., 2024; Kumar et al., 2022). Synthetic data has been generated and utilized in multiple settings that are challenged by scarce datasets, including the development of multiracial facial recognition models and the curation of customized organ models for surgical simulation and training (Park et al., 2024; Kimura et al., 2025). While showing great potential, many generative AI methods, particularly diffusion-based models, are prone to hallucination, where the produced outputs are too far detached from reality, creating synthetic images that are plausible but nonsensical (Rajpurkar et al., 2018). To address this, we explored the use of variational autoencoder (VAE). VAE is a generative model that learns to encode input data into a latent space defined by a probability distribution, typically a multi-variate Gaussian. During training, the model optimizes a loss function that balances reconstruction accuracy with regularization, ensuring the latent space conforms to a known prior distribution. To generate synthetic data, new samples are drawn from this prior distribution and passed through the decoder network to produce novel but statistically consistent outputs (Wei & Mahmood, 2021). By generating outputs that adhere to specific input data, VAE is a more controllable generative model than Diffusion, mitigating the problem of hallucination. Previously, we demonstrated that VAE-enhanced synthetic datasets significantly improved glaucoma detection by ViT (Chen et al., 2024). Here we report the development of a second-generation VAE-based data enhancement workflow to deliver dataset diversity beyond previous methods and explore how this new functionality can enable access of DL methods to rare diseases.

For the purpose of confirming laterality, only fundus photos with clearly visible macula and optic nerve head (ONH) structures were included. If the patient had multiple imaging dates, only the most recent date was included for review. Fundus photos from the most recent date were manually assessed for image quality (excessive blur, artifact, sufficient field of view). In addition, all right eye (OD) images were horizontally flipped to match the image orientation of the left eye (OS) prior to model training. Eyes without images meeting these quality criteria were excluded.

To address the problem of overfitting caused by data scarcity, we employed a variational autoencoder framework to generate synthetic data. Two variations of autoencoder expansion were investigated: random noise expansion (Gen 1) and pair-wise combinatorial expansion (Gen 2).

I. Random Noise Expansion (Gen 1) VAE

Details of this first-generation expansion and training procedure were previously published (Chen et al., 2024). Briefly, synthetic images were generated by introducing noise into the latent space using four noise distributions: constant, Gaussian, uniform, and sinusoidal. For each image, one strategy was randomly selected and applied with a randomly sampled strength parameter from .05 to 1 to the embedding, leading to additional variations in the output images. Each image along with their random noise expansion was added to the training set resulting in a two-fold expansion of the training data. Our VAE utilized a dual-level ResNet-based encoder-decoder structure trained on ImageNet with pixel-wise reconstruction loss.

II. Pairwise Combinatorial Expansion (Gen 2) VAE

To expand data enhancement capabilities in terms of both quantity and diversity, we developed a 2nd generation framework to perform pairwise combinatorial expansions using VAE. Here, synthetic images were generated by combining the latent representations of every possible pair of training images that share the same genotype label (Fig. 1). This pairwise structure ensures that every unique two-image combination within a genotype class contributes to the synthetic dataset. For each image pair, we encoded both images using a pretrained variational autoencoder (AutoencoderKL) and linearly combined their latent vectors using a predefined set of mixing ratios of 0.1, 0.3, 0.5, 0.7, and 0.9, as illustrated in Fig. 2. Each mixing ratio determines the relative weight between the original image and its paired image in the latent space. This process is repeated for n x n combinations for each label, where n is the number of images with the same label, augmenting each inheritance mode by C (n, 2) images. The resulting composite latent vector is then decoded to produce a synthetic image.

The pretrained foundation model used in this study (Google’s vit-base-patch16-224-in21k) was initialized with pretrained weights from ImageNet-21k and modified for binary classification. We applied an 80/20 train-test data split, ensuring that the fundus images of both eyes from the same patient are incorporated into the same split, and trained two ViT models: one using only the original fundus images and the other incorporating both original and synthetic images (Fig. 3). To prepare the input images for training and evaluation, all images were resized to 224×224 pixels, matching the input size expected by the ViT model. Resizing images to a resolution of 224x224 pixels is a standardized preprocessing step in Vision Transformer (ViT) model, as demonstrated in multiple ViT-based retinal imaging studies (Wang et al., 2024; Powroznik et al., 2025). Training was performed for 30 epochs, using the AdamW optimizer with a learning rate of 5e-05. During training, we applied data augmentations using a randomized resizing crop followed by a randomized horizontal flip, introducing variability and reducing overfitting. Cross validation was performed by re-sampling to generate representative train/test splits. Synthetic images were generated from and added to only the training sets. To calculate mean accuracy, recall, and specificity, model performances were averaged over five-fold cross-validation. Pooled AUCs were calculated by aggregating labels and predictions generated across all validation sets.

We first evaluated our retinitis pigmentosa inheritance Vision Transformer (RP-ViT) base model trained on color fundus photo dataset without synthetic training data enhancements. For the purpose of binary model classification, we combined the four inheritance modes into two classes, autosomal (AR and AD) and X-linked (XR and XLC). For the RP-ViT base model, pooled AUC was 0.67, mean accuracy was 0.62 ± 0.05, and mean specificity was 0.55 ± 0.07 (Fig. 5, Table 2).

I. Random Noise Expansion (Gen 1)

We trained and evaluated RP-ViT using a two-fold augmented dataset with synthetic images produced by the first-generation VAE method (Chen et al., 2024). This improved pooled AUC to 0.75, mean accuracy to 0.69 ± 0.05, and mean specificity to 0.64 ± 0.07 (Fig. 6, Table 2).

II. Pair-wise Combinatorial Expansion (Gen 2)

Finally, we evaluated RP-ViT trained on a combinatorial pair-wise expanded dataset with synthetic images produced by our second-generation VAE method. This final model outperformed both the base model and the random noise-expansion RP-ViT models on all measured metrics with a pooled AUC of 0.79, mean accuracy of 0.71 ± 0.10, and mean specificity of 0.68 ± 0.10 (Fig. 7, Table 2).

¹Pooled AUC refers to AUC calculated from combined classifications across all folds.

Deep-learning systems have achieved near-expert performance in detecting prevalent retinal disorders—most notably diabetic retinopathy and age-related macular degeneration—powered by the vast imaging datasets generated through population-wide screening initiatives (Aggarwal et al., 2021; Cen et al., 2021; Shoaib et al., 2025). Yet their translation into subspecialty clinics has been limited, with few algorithms undergoing rigorous, real-world validation in expert settings where diagnostic subtleties matter most (Abràmoff et al., 2018; Li et al., 2023). The gap widens further for rare inherited disorders such as retinitis pigmentosa, where patient numbers are small and existing datasets fall several orders of magnitude below typical deep-learning requirements. It thus remains an open question for the field whether AI models trained on such limited cohorts can deliver actionable insights.

In this study, we report on the application of generative AI and deep learning to classify genetic inheritance patterns in rare retinal diseases. Specifically, we demonstrate the feasibility of using a state-of-the-art (SOTA) vision-based deep learning model (ViT) to identify modes of inheritance from retinitis pigmentosa (RP) fundus images. To enable application of ViT on an inherently scarce dataset that is orders of magnitude smaller than previously reported deep learning usage cases, we developed novel methods of training data enhancement. Using a variational autoencoder, we introduced both random-noise (Gen 1) and pair-wise combinatorial (Gen 2) expansions of training data. Results show that while both methods can improve ViT performance, Gen 2 (which fuses pairs of labeled images to simulate biologically plausible variants) offers greater range of quantity and diversity in its synthetic data, leading to improvements beyond Gen 1 methods. By generating synthetic data via the fusion of real patient images rather than using diffusion-based methods, we minimize hallucination and maximize clinical fidelity and relevance of the resultant synthetic training data.

Our findings introduce two important concepts in clinical AI methodology: first, the ability to identify hidden patterns that correlate with inheritance mode and are invisible to human detection, and second, an encoder-decoder generative AI approach to alleviate scarce-data limitations of rare diseases. To our knowledge, this is the first report to use deep learning to classify images by mode of inheritance in retinitis pigmentosa (RP). We chose a binary classification approach based on both clinical and model design considerations. Classification of inheritance patterns is an important first step for diagnosis and for selecting appropriate genetic testing approaches and interpretation, which may inform the patient about relevant clinical trials or treatment options (Ullah et al., 2025; Lam et al., 2021; Gocuk et al., 2023). Distinguishing between X-linked vs. autosomal inheritance for rare IRDs can improve genetic counselling and family planning, and also enable more targeted diagnostic testing approaches, potentially improving the yield of diagnostic testing. In addition to its clinical relevance, our binary classifier model demonstrated better accuracy compared to a multi-class classifier. We suspect this difference may be due to the inherent limitations of classifiers in handling multiclass problems, as well as the small size of rare disease datasets (Allwein et al., 2000). Multiclass classifiers are intrinsically more data-hungry, a limitation that becomes pronounced in rare-disease settings where sample counts are low. While our data augmentation pipelines aim to address this data scarcity problem, their benefits, particularly that of Gen 2, scale with the size of the native dataset because each new image arises from two distinct originals. Consequently, the smallest cohorts (e.g. <20 images) still pose a challenge. Ongoing efforts therefore aim to further advance data expansion techniques to enable AI tools that perform robustly even with very small datasets, thereby improving classification accuracy in multi-class problems.

Beyond building a more robust data-enhancement architecture, it is our hope that a discovery-focused approach to AI tool development may uncover previously unrecognized deep biological signatures. Pathology unfolds across many measurement axes (e.g. structural, functional, and molecular), each contributing a slice of a high-dimensional landscape. Integrating complementary modalities, such as optical-coherence tomography, fundus imaging, and electrophysiology, provides a more complete representation of inherited retinal disorders and, accordingly, may improve DL classification performance (Hwang et al., 2025).

We acknowledge several limitations to our approach. First, we relied on a limited dataset from a single institution. While UCSF is a tertiary referral center and attracts patients worldwide, our cohort for this initial report is nevertheless limited. Our current work serves as a proof-of-concept, laying groundwork for promoting the development of future multi-center studies to validate the reported framework across more diverse populations.

Second, the autosomal and X-linked groups in our cohort differed significantly in mean age—an expected reflection of the earlier onset typical of X-linked IRDs. Although this age disparity may serve as a confounding factor, it also represents a genuine clinical distinction in onset and progression between inheritance patterns; the differences we observed likely reflect differences in disease severity that correlate with inheritance pattern. Third, similar to other deep learning models, our method involves a level of complexity that can obscure the interpretability of its decision-making process —making it difficult to determine which features were most relevant to the model predictions. While established interpretability tools exist for earlier architectures such as CNNs, the self-attention mechanisms used in Vision Transformers (ViTs) make it more challenging to generate intuitive or clinically recognizable explanations (Doncevic & Herrmann, 2023). To address this limitation, there is ongoing work to develop a multimodal large language–vision model to articulate its visual reasoning for medical image analyses.

In conclusion, this study highlights the potential of deep learning to contribute to the diagnostic workflow for inherited retinal diseases. By integrating VAE-powered data enhancement with ViT-driven binary classification, we report a novel and effective framework for mitigating the limitations imposed on AI tool development by inherently scarce data and may provide information about disease severity in resource-limited settings where genetic testing is not available or results are indeterminate. The reported framework enabled classification of RP inheritance mode using a small dataset, fulfilling an important clinical objective and underscoring the potential of AI to enhance healthcare for all patient populations, irrespective of disease prevalence.