MEXA: A Multi-Modal Emotion eXtraction and Analysis Framework with Cross-Attention Fusion and Contrastive Sarcasm Detection

Abstract

In digital communication, emotion analysis needs to be multimodal, that is, the integration of heterogeneous modalities such as language, facial expression and temporal visual dynamics. Most of the existing methods, however, are mainly unimodal and fail to consider the inconsistencies that are common between modalities which are crucial for sarcasm recognition. To overcome this drawback, this paper proposes MEXA (Multimodal Emotion eXtraction and Analysis), which is a three-branch deep learning framework for a unified categorical emotion recognition, emotion-intensity regression on a [0, 3] scale, and binary sarcasm detection. MEXA generates textual representations, visual features and temporal visual representations, by applying a fine-tuned 12-layer BERT model, a ResNet-50 backbone pretrained on AffectNet and a 3D CNN-LSTM model, respectively. A cross-attention Transformer fusion module is then used to fuse the information from the three modalities, enabling each to attend to complementary representations from the other modalities. To boost the discrimination of multimodal representations, MEXA also adds a supervised InfoNCE-based contrastive learning objective for sarcasm detection. The framework is tested and assessed on three metrics: MUStARD++ (Sarcasm Detection), CMU-MOSEI (Emotion-Intensity Estimation), and MELD (Emotion Recognition in Conversational Environments). With a weighted F1-score of 75.8%, MEXA is 1.3 percentage points ahead of the best reported results on MUStARD++. It has an average MAE of 0.759 on CMU-MOSEI for emotion-intensity regression, and a weighted F1-score of 65.4% on MELD for emotion recognition in seven classes. Due to substantial class imbalance in MELD (approx. 47% of samples are in the Neutral class, while Fear and Disgust make up about 2% of samples each), weighted F1 is used for the evaluation throughout.