Hybrid Ensemble Fusion for Emotion-Linked Mental Health Screening: A Proof-of-Concept Pilot Study using Facial Expression and Textual Data

Abstract

Mental health disorders remain a major global health burden, with the World Health Organization estimating that roughly 970 million people, or one in eight, were living with a mental health condition in 2019, a figure worsened by the 25% surge in anxiety and depression during the COVID-19 pandemic. Traditional diagnostic approaches rely heavily on subjective clinical interviews, which can delay treatment and increase the risk of misdiagnosis. This paper reports a proof-of-concept pilot study investigating whether a hybrid ensemble fusion of facial-expression and textual features is architecturally viable as an early step toward an objective mental-health screening aid. Facial affect is captured using a ResNet50 convolutional network trained on a nine-class facial-expression corpus, while short textual descriptors, heuristically derived from each image's own emotion label as a proxy for candidate mental-state indicators, are encoded using BERT embeddings. The two modality-specific feature sets are combined through a stacking ensemble with Multinomial Naive Bayes and XGBoost as base learners and an XGBoost meta-learner. The unimodal facial branch achieves 88.6% test accuracy and an F1-score of 88.75%, while the fused stacking ensemble reaches 75% accuracy on a small held-out pilot set, with class-wise precision and recall values that suggest balanced performance across most categories at this scale. Synthetic Minority Oversampling is used to counter class imbalance in under-represented affective classes such as contempt and sleepy/fatigue. These results demonstrate the technical feasibility of the proposed fusion architecture but should be read as an early-stage proof of concept rather than a validated diagnostic result: the textual descriptors are derived from the same labels as the visual branch rather than from an independent modality, and the fused evaluation set is too small for statistically robust claims. We discuss these limitations explicitly and outline the clinically grounded data, independent-modality design, and larger-scale validation required before this pipeline could support real screening use.