In the early 2010s, experts began to see that social media could be helpful in spotting signs of depression, since it captures how people think and feel in real time. The words people use on X can reveal signs of depression, according to one of the first studies by bib15. Their dictionary-based method demonstrated that a person's language frequently reflects their mood, providing a foundation for further computer research. Subsequently, researchers began to go beyond text alone by incorporating social activity features, such as network size and interaction levels, and user details, including age, gender, and location bib39,bib43. Depressive symptoms show correlation with behavioural and environmental factors such as posting time, weather, and daily activity patterns bib13,bib21,bib43. According to these results, combining behavioural and linguistic indicators rather than just language is the most effective way to understand depression.

After the initial text-based studies, researchers examined how depressive signs change. bib39 noticed that people showing depressive behaviour often post at unusual hours or follow irregular daily patterns.bib33 found that when people post, monitoring their activity over days or weeks allows the model to detect mood swings more accurately. bib10 also demonstrated that predictions can be more accurate when using time patterns. They achieved an F1-score of 0.956 by applying a time-aware attention model to Instagram posts. Simultaneously, other researchers started concentrating on outward manifestations of depression. bib15 found that users with depressive tendencies often shared darker and less colourful images, while bib31 achieved more than 70% accuracy with image-based machine learning.

Research started integrating text and image data as the field developed to create richer and more trustworthy models. \cite{bib5} attempted to enhance the model's understanding of its significance by incorporating topic modelling. In a separate study, bib19 employed a Bi-GRU network for text and a VGGNet for images to integrate text and image features. More recently, bib18 examined videos to identify body language and facial expressions as nonverbal signs of emotional distress. Although their study included video data, the material came from mixed sources: clinical recordings for videos and social media content for images and text. However, work in this area is still limited as many datasets are private and hard to reproduce. These studies show that multimodal learning is steadily advancing bib11,bib34. Another ongoing challenge is the limited language variety in available data. Most existing research still focuses mainly on English-language datasets. Only a handful of studies have looked into other languages, including Chinese bib43,bib40, Russian bib37, Arabic \cite{bib2}, and Bengali bib25.

In contrast, mixed-language settings like Hinglish, which naturally blends Hindi and English, have received little attention. Such texts are often complex for regular NLP systems to interpret, including mixed scripts, informal wording, and inconsistent grammar bib32. Overcoming this language gap is crucial for developing mental health detection models that are realistic, inclusive, and more accurately reflect how people communicate online.

In computer science, the study of depression detection moved steadily from hand-crafted statistical methods to deep learning architectures capable of learning complex, non-linear relationships in multimodal and multilingual data. Early work primarily relied on manual feature extraction, where researchers identified linguistic markers, behavioral signals, or profile attributes and then trained models such as support vector machines, logistic regression, or random forests bib14,bib21. These methods progressively added visual cues, temporal patterns, and social network indicators as richer data became available. However, most still handled each signal separately rather than using a single framework. Attempts are made to model time explicitly. bib10 investigated the relationship between the timing of Instagram posts and people's emotions. Their model focused on user post patterns and found links to emotional changes. Earlier, bib33 also examined timing in conjunction with the text to observe how moods evolve. Despite these advances, few large-scale public datasets include text, images, and timestamps, especially in multilingual and code-switched environments, which remain largely unexplored.

The transition to deep learning significantly improved the ability to capture such dependencies. Early research employed recurrent models, such as LSTMs and GRUs, to investigate how words in text relate to each other over time bib35,bib43. Convolutional networks, such as VGGNet and ResNet, gained popularity for image data analysis because they can capture emotional and contextual details from visual content bib19. Later, transformer models such as BERT, RoBERTa, and XLM-RoBERTa revolutionised text analysis, enabling the learning of deep language patterns that transfer across different languages bib12,bib28. Recent studies have also started to focus on model transparency and multilingual learning. bib40 explored ways to make depression prediction models more understandable by showing where the model focuses its attention and adjusting it for different languages. \cite{bib7} took a different approach, utilising a Time2Vec method to connect feelings and timing information from other types of data.

Our work combines XLM-RoBERTa for handling text and ResNet-18 for processing visual content. XLM-RoBERTa is a multilingual transformer trained on approximately 2.5 TB of text from CommonCrawl, covering 100 languages bib12. It expands the masked language modelling (MLM) idea from BERT to multilingual data by predicting hidden words based on their context. The model minimizes the following cross-entropy loss:

where represents the masked tokens, and denotes the surrounding words used as context.ResNet-18 is a convolutional neural network designed to overcome the vanishing gradient issue using shortcut, or skip, connections bib22.These connections allow the input feature map to bypass one or more layers and be added directly to the output. The relationship can be represented mathematically as follows:

where denotes the residual function, represents the input feature map, and is the resulting output after the residual connection. This setup preserves important visual details while enabling the deep network to recognize complex patterns that relate to emotions and mood. Combining these two models, our system captures proper signals from text, images, and posting behavior, leading to a more realistic picture of how depressive signs appear on social media. Building on earlier multimodal research bib11,bib10,bib7, our framework also explores areas that have received less attention, especially multilingual and code-mixed communication challenges in real-world depression detection. To contextualize our contribution, Table summarizes studies that have used non-English content or a mix of English with other languages for depression detection.

begin{sidewaystable*}\caption{Summary of studies using non-English or multilingual data for depression detection.}\tiny\label{tab:relatedwork}\renewcommand{\arraystretch}{1.2}\setlength{\tabcolsep}{2pt}\begin{tabular*}{\textheight}{@{\extracolsep\fill}p{2.5cm}p{2.8cm}p{1.6cm}p{1.6cm}p{2.2cm}p{1.6cm}p{0.8cm}p{5.2cm}}\topruleAuthor & Year & Models & Data Model & Model Type & Platform & Language(s) & F1 & Key Highlights \\\midruleOur Work (2025) & XLM-RoBERTa + ResNet18 + Temporal Embeddings & Text, Images, Time & Multimodal & X & English, Hindi, Hinglish & 0.79 & Multilingual, code-switched data, early fusion, time-aware design \\Xiao et al. (2024)bib44 & Hierarchical Transformer Network (HTN) & Text & Unimodal & Sina Weibo & Chinese & 0.87 & Hierarchical text modeling across social posts \\Yang Liu (2024)bib28 & Weighted Graph-RoBERTa Neural Network (WGRNN) & Text & Unimodal & Online Reviews & Chinese & 0.84 & Graph-based contextual representation \\Bucur et al. (2023)\cite{bib7} & Multimodal Transformer (Vanilla, Set, Time2Vec) & Text, Images & Multimodal & Reddit, X & English, Spanish & 0.81 & Temporal embeddings and multimodal fusion \\Kabir et al. (2022)bib25 & KSVM, RF, LR, KNN, NB, LSTM, GRU, RNN & Text & Unimodal & Blogs \& Microblogs & Bangla & 0.77 & Traditional + deep text models for low-resource language \\Abdulqader M. Almars (2022)\cite{bib3} & Attention-based Bi-LSTM & Text & Unimodal & X & Arabic & 0.83 & Attention for contextualized Arabic depression cues \\Cheng et al. (2022)bib10 & Multimodal Time-Aware Attention Networks (MTAN) & Text, Image, Time & Multimodal & Instagram & English & 0.956 & Time-aware LSTM + attention for interpretability \\El-Ramly et al. (2021)bib16 & Fine-tuned ARABERT, MARBERT & Text & Unimodal & X & Arabic & 0.82 & Domain-specific BERT adaptation \\Mumu et al. (2021)bib30 & CNN-LSTM Hybrid & Text & Unimodal & Facebook & Bangla & 0.75 & CNN-LSTM synergy for Bangla \\Safa et al. (2021)bib34 & CNN + BERT Fusion & Text, Image, Profile & Multimodal & X & English & 0.74 & User-level multimodal analysis \\Wang et al. (2020)bib40 & BERT, XLNet, RoBERTa & Text & Unimodal & Sina Weibo & Chinese & 0.86 & Transformer-based comparative analysis \\Chiu et al.(2020)bib11 & LSTM with Temporal Gaps & Text, Images, Time & Multimodal & Instagram & English, Chinese & 0.83 & Time interval modeling for user posts \\Eatedal Alabdulkreem (2020)\cite{bib2} & RNN + LSTM Cascaded DNN & Text & Unimodal & X & Arabic & 0.81 & Layered deep neural structure \\Wang et al. (2019)bib40 & CNN, LSTM, BERT, RoBERTa, XLNet & Text & Unimodal & Sina Weibo & Chinese & 0.80 & Early transformer benchmark \\Stankevich et al. (2019)bib37 & MLP & Text & Unimodal & VKontakte & Russian & 0.78 & Depression detection from short social posts \\Shen et al. (2018)bib35 & DNN & Text & Unimodal & X, Weibo & English, Chinese & 0.72 & Cross-lingual early deep model \\Wu et al. (2018)bib43 & RNN-inspired DNN & Text & Unimodal & Facebook & Chinese & 0.76 & Early deep architecture for behavioral signals \\\botrule\end{tabular*}\footnotemark\footnotetext{\textbf{Abbreviations:} CNN – Convolutional Neural Network; RNN – Recurrent Neural Network; LSTM – Long Short-Term Memory; Bi-LSTM – Bidirectional Long Short-Term Memory; DNN – Deep Neural Network; MLP – Multilayer Perceptron; BERT – Bidirectional Encoder Representations from Transformers; XLNet – Generalized Autoregressive Transformer; RoBERTa – Robustly Optimized BERT Pretraining Approach; KSVM – Kernel Support Vector Machine; RF – Random Forest; LR – Logistic Regression; KNN – K-Nearest Neighbors; NB – Naïve Bayes; GRU – Gated Recurrent Unit.}\footnotetext{Hinglish – a Hindi–English mixed language commonly used on Indian social media platforms, combining words or phrases from both languages within the same post.}\end{sidewaystable*}

As shown in Table 6, most earlier studies focused on a single type of data, mainly text in languages such as Chinese bib43,bib40,bib28,bib44, Arabic bib16, Russian bib37 and Bangla bib30,bib25. Even the more recent multimodal studies [bib1,bib10,bib7,bib27 still face key limitations, such as the limited availability of high-quality image data, restricted language coverage, and a lack of focus on multilingual or code-mixed settings.

So far, no research directly explores Hinglish or other mixed languages, which create unique challenges in tokenisation, meaning understanding, and connecting text with images.. Our work addresses these shortcomings by proposing a time-aware, multilingual, and multimodal framework that jointly learns from text, images, and timing information drawn from real social media posts. Previous studies have mainly examined text and pictures separately, without considering their interaction. Our model takes a different path; it studies both side by side to see how words and visuals influence each other. Using transformers for text and CNNs for images, the system learns subtle language details alongside the visual mood or setting. This combination helps it build a more precise and more realistic picture of how people express emotions or signs of depression on social platforms.

This work examines how various types of information, including text, images, and posting time, can be combined to detect signs of depression in social media posts. Unlike standard text classification, our dataset includes multiple languages and diverse input types, which introduces additional layers of complexity. To clarify the research motivation, we first identify the major sources of challenges—such as multilinguality, code-mixing, and multimodal imbalance—and then describe the specific objectives of the work.

Our study gathered publicly available posts from the X platform. The idea is to collect real online content that might show signs of depression, written in different languages and styles. Every post is marked with one of two labels: Depression or Normal. These posts reflect how people usually share their feelings online, often switching between languages in a single post. Each post is assigned one of two labels: Depression or Normal. The dataset covers four major linguistic forms: English (EN), Hindi written in the Devanagari script (HD), Hindi written using Roman letters (HR), and Hinglish (HN), which combines Hindi and English words within the same post. This diversity makes the data linguistically rich and realistic, adding complexity for automated analysis bib32.

To identify relevant posts, the strategy proposed by bib14 is followed, but broadened the focus beyond explicit self-reports of depression. The collection process not only utilised explicit statements like ``I am depressed'' but also considered posts that showed quieter forms of distress, such as feelings of emptiness, loss of hope, or emotional fatigue, an approach consistent with earlier findings on implicit markers of mental health issues in social media text bib15, bib20.

To ensure systematic collection, a multilingual lexicon is developed guided by three major clinical frameworks: the Diagnostic and Statistical Manual of Mental Disorders (DSM-5)cite{bib4}, the Patient Health Questionnaire-9 (PHQ-9)bib26, and the International Classification of Diseases (ICD-10)bib41. These guidelines do not serve for medical diagnosis but help in finding repeated language patterns linked to depressive feelings. Similar to earlier work bib43, bib10, our work pulled out common words and short phrases that reflect sadness, tiredness, low motivation, and even self-blaming or suicidal thoughts. Each keyword is written in EN, HD, and HR, while frequent misspellings and social media abbreviations are also included to reflect informal online communication styles. Frequent social media misspellings and abbreviations are incorporated for broader coverage.

The collection followed a staged process, drawing from methods used in previous multimodal studies bib11. The initial step is to create a multilingual lexicon that captures variations in script and spelling. This list is then used to search for posts, taking into account informal or mixed-language writing that is common on social media. The entire workflow is summarised in Figure 2, which illustrates the stages of multilingual lexicon creation, data retrieval, cleaning, and multimodal alignment.

After that, the data is cleaned, removing retweets, ads, and bot-generated content, so the final set consisted of real user posts and genuine feelings bib31. For users with multiple depression-related posts, historical tweets for up to 60 days are also collected to provide temporal continuity and behavioural context, following the practice used in earlier work bib14.

After collection, both textual and visual data are carefully preprocessed. The text is standardised through lowercasing and cleaned by removing URLs, emojis, hashtags, mentions, repeated characters, and other noise consistent with previous preprocessing pipelines bib43, bib10. To distinguish between HD and HR, a fastText-based language identification model bib24 is applied, supported by script-specific heuristics. All images are downloaded and passed through a perceptual hashing filter to remove duplicates. Images are collected and cleaned using perceptual hashing to avoid duplicates and then resized to 224 × 224 pixels and normalised to fit CNN-based models bib22. Further metadata, such as posting time, is also retained to support later analysis of users' temporal activity patterns.

As the study involves sensitive mental health content, special care is taken to ensure the ethical handling of data, user privacy, and protection against any risk of re-identification. The study follows established best practices for privacy-preserving social media research bib13 and built a custom anonymisation pipeline in Python using the \texttt{pandas} library. Before starting anonymisation, several quasi-identifiers (QIDs) are identified: details that do not directly reveal a person's identity but could still be used to trace them when combined with other information. These include screen names, display names, and exact posting timestamps. Although our dataset lacks direct identifiers such as email addresses or phone numbers, earlier research shows that even indirect data points can facilitate re-identification, especially in sensitive areas like mental health bib31.

To address this, a three-step anonymisation process is applied. In the first step, all identifiable details are removed. Usernames, personal names, and tagged handles are deleted, and all links are cleared to ensure that posts cannot be traced back to their original profiles. Next, the posting time is generalised to the nearest calendar day. This reduces timestamp precision but maintains the natural posting order, helping to preserve temporal patterns useful for later analysis bib10.

This study also applies k-anonymity with , ensuring that every record matches at least five others in basic features such as language and posting date. This step follows privacy protection guidelines widely used in social media research bib38. These three layers; personal information masking, temporal generalisation, and controlled anonymisation—create an ethically secure and research-ready dataset. It retains enough linguistic and temporal detail for mental health modelling without compromising privacy. In future work, automated language detection tools such as fastText or langdetect may be used to further refine linguistic consistency across anonymised data entries.

Our study utilises both machine and human effort to tag the data. Initially, posts are selected based on a list of words commonly associated with sadness or depression. Each post is manually reviewed to ensure label accuracy. To build this list, earlier studies are reviewed for how people express mental health issues on social media, and related terms from various languages (EN, HD, HR, HN) are incorporated bib15, bib35, bib11, bib10.

Similar to lexicon-driven methods used by bib43 and bib14, our term list is grounded in established clinical frameworks such as the Diagnostic and Statistical Manual of Mental Disorders (DSM-5)cite{bib4}, the Patient Health Questionnaire-9 (PHQ-9)bib26, and the International Classification of Diseases (ICD-10)bib41. These sources outline medically recognised symptoms like ongoing sadness, lack of interest, tiredness, hopelessness, and suicidal thoughts, which ensure that the labelling focuses on genuine clinical signs rather than regular mood changes. Afterwards, each post is read manually and marked as either Depression or Normal. Posts expressing clear or implied signs of distress, such as loneliness, emptiness, worthlessness, or self-blame, are marked as Depression. Posts representing neutral discussions, humour, or casual conversation are labelled Normal. In cases where meaning is uncertain, the broader context is considered, including the repeated use of negative language, tone of expression, or emotional intensity. This approach follows earlier manual labelling strategies applied to Twitter and Reddit data bib23.

Although labelling is primarily done at the post level, user-level patterns are also considered. If a user's posts show a clear emotional pattern over time, that background is also considered while deciding the final label. This idea aligns with the eRisk and CLPsych shared-task project methodology bib29, bib36. When available, attached images are also reviewed, as previous multimodal analyses demonstrated that visual cues often reinforce or clarify emotional states expressed in text bib31, bib21. After annotation, roughly 41% of posts are categorised as Depression and 59% as Normal. This pattern matches what earlier social media studies show: posts about depression are usually fewer than normal ones bib10. By combining automatic keyword search with careful human review, the final set provides a more authentic and culturally diverse representation of how people discuss depression in various languages, encompassing both text and images.

Accurately identifying depressive signals requires drawing information from multiple input forms such as text, images, and posting behaviour. To achieve this, a multimodal feature extraction pipeline is developed that converts each modality into compact numerical representations suitable for deep learning models.

The proposed model combines three types of information; text, images, and posting time to gain a more comprehensive understanding of signs of depression. Each component contributes a unique perspective: the text reflects what users express, the images reveal how emotions are visually portrayed, and the temporal dimension shows when these expressions occur. The overall structure of the model is illustrated in Figure 3 which shows the overall design of the proposed multimodal feature extraction and fusion framework.

The final dataset comprises 15,739 posts, among which 1,595 posts (approximately 10.1%) contain valid images. Most of the data still comes from text, which aligns with findings from earlier social media studies. People mainly express themselves through words, but sometimes add pictures to highlight or support their feelings.While relatively small, the multimodal portion of the dataset adds valuable diversity, enabling the evaluation of models that combine linguistic and visual signals, a dimension often underrepresented in earlier depression detection datasets.

Our model is trained using 80% of the data and retains the remaining 20% to evaluate its performance. The number of Depression and Normal posts is roughly equal in both sets. The model is trained with the Adam optimiser and binary cross-entropy loss. The learning rate is 0.0001, and the batch size is 8. Training lasted approximately 30 rounds. When no improvement showed after a few checks, it stopped early to avoid overfitting. The learning rate and batch size are selected after a few trial runs, allowing the training to remain steady and not fluctuate excessively. Every setup ran with different random seeds three times to ensure the results were not merely a matter of luck. The study utilised Accuracy, Precision, Recall, F1-score, and AUC to evaluate the model's performance. By resampling the data, standard deviations and confidence ranges are also calculated to increase the accuracy of the results. Score differences are verified to determine if they reflected real variation or random noise. Overall, this setup gave a precise and repeatable way to test performance across all data types.

The encoders described in Section 3.3 are jointly optimised under an early-fusion paradigm. The concatenated representation

is used as input to the classifier, enabling joint learning of cross-modal interactions. We trained the architecture end-to-end using the binary cross-entropy (BCE) loss:

where is the ground-truth label and is the predicted probability. Optimisation is performed using the Adam optimiser, and early stopping is implemented based on the validation loss.

Various improvements, such as dropout regularisation, handling missing images, and batch normalisation, are adopted to improve robustness. For multimodal layers, dropout is applied as:

where is the keep probability and denotes element-wise multiplication.

If an image is unavailable, the visual embedding is replaced by either:

or stochastically masked during training:

To stabilise training across heterogeneous modalities, batch normalisation is applied to fused representations:

where , are batch statistics and , are learnable scale and shift parameters.

The model is trained following the multimodal architecture illustrated in Figure X. It uses an early-fusion strategy, where outputs from the text, image, and temporal encoders are concatenated and optimised together within a single network. Training is carried out end-to-end using backpropagation, allowing the gradients to update all encoders simultaneously. For optimisation, we use binary cross-entropy loss:

where denotes the Depression label.

This configuration enables the model to learn how different modalities interact in a common feature space while collecting information specific to each modality. Early integration of these representations enhances the system's ability to detect subtle patterns between language, images, and posting behaviour. The early-fusion approach is selected because it offers a good balance between interpretability and computational efficiency, making it well-suited for real-world mental health analysis tasks.

The results show that using multiple types of input, such as text, pictures, and time, helps the model identify signs of depression more effectively than using only one type. Table 3 lists how each version of the model operates, including those that utilise only one type of data and those that combine them. When trained under three different random seeds, the multimodal configurations achieved higher accuracy and showed more consistent performance across runs.

Looking at the unimodal baselines, the image-only model performed the strongest, reaching an average accuracy of and an F1-score of . The text-only model followed, with an accuracy of and an F1-score of . Meanwhile, the temporal-only variant achieved an accuracy of and an F1-score of . Once modalities were fused, performance improved notably. The text + image model reached the highest accuracy of and the best F1-score of . The complete multimodal model, which combines text, image, and temporal signals, achieved a balanced accuracy of and an F1 score of . Across all fusion settings, AUC values ranged between and , showing an evident ability to discriminate depressive from non-depressive content rather than performing at a chance level. These numbers lead to three concrete observations. First, images carry strong predictive cues in this dataset. The image-only baseline is approximately as robust as the whole model in some metrics, and substantially better than text-only. Second, textual signals are informative but noisier in our collection; they improve overall performance when combined with images, but do not match image-only performance on their own. Third, temporal features consistently add modest gains in stability and discrimination when combined with other modalities, even though they are not highly predictive alone. Interpreting these observations requires care. The strong performance of the image-only model reflects a combination of meaningful visual cues and dataset-specific regularities. In many depressive posts in our corpus, images tend to exhibit darker tones, lower contrast, or solitary scenes, visual traits that a convolutional encoder can pick up reliably. This makes image features valuable on our dataset, but it also raises a caution as part of the model's success may come from learning visual style differences that correlate with labels, rather than learning deeper, content-level indicators of depression. Due to this, we treat the image-only result as an empirical outcome that is valid for our dataset while acknowledging the need for external validation to confirm generality.

The weaker performance of the text-only model can be traced to the linguistic diversity of the dataset. The corpus comprises posts written in English and Hindi, using both the scripts, often within the same sentence. Frequent code-switching, irregular grammar, and nonstandard spellings make tokenisation difficult and reduce the effectiveness of pretrained multilingual encoders. This variability explains the higher standard deviation observed for the text-only configuration. Even so, text remains indispensable for understanding emotional meaning. It directly expresses feelings such as hopelessness, guilt, or fatigue signals neither images nor timestamps can reliably convey. This complementary nature of modalities is reflected in the strong performance of the text+image model (), which effectively resolves ambiguity and improves precision and recall compared to unimodal models.

Temporal information plays a more subtle but helpful role. Posting behaviour has weak but not negligible predictive power, according to the time-only model (). These temporal embeddings enhance stability and marginally increase AUC scores when combined with visual or textual signals. Although the gain is modest, it consistently demonstrates that behavioural rhythm adds context beyond content alone.

Two further observations emerge from the variance analysis. First, the fusion-based models exhibit remarkably low variability across random seeds; for example, the whole model's F1-score varies by only . This low variance indicates that the multimodal combination improves consistency, as well as performance. Second, several confidence intervals overlap between fusion variants, suggesting that although additional or temporal modalities provide stability, their incremental accuracy gains are negligible. In other words, multimodal training contributes more robustness than dramatic jumps in headline metrics. The results also highlight certain limitations. The visual encoder could depend on surface-level patterns instead of real emotional indicators because of dataset-specific biases, such as the prevalence of darker or sadder images in depressed posts. Likewise, the lower and more variable text performance emphasises the need for more effective preprocessing and modelling strategies for code-switched and Romanised Hindi. Because the numeric differences among fusion variants are sometimes narrow, statistical significance testing remains crucial to confirm that observed trends are not due to random variation. All things considered, these results support the use of a multimodal strategy. Despite the drawbacks of each modality alone, when combined, they produce a more stable and accurate classifier than any unimodal baseline. Three possibilities for future study and practical implementation are monitoring prediction by validating the model on external datasets, examining potential visual and annotation biases, and enhancing linguistic preprocessing to support translation and code-mixing more effectively. Together, these steps would help turn the promising empirical gains observed here into robust, generalisable advances for automated depression detection on social media.