TG-FSS: An Adaptive Taxonomy-Guided Framework for Selecting Optimal Fusion Strategies in Multimodal Architectures

Abstract

There have been two streams of research in multimodal fusion: taxonomy-aware topic modeling systems, which determine the structural nature of a fusion strategy (early, late, hybrid, attention-based, or graph-based), and fusion architectures, which perform multimodal fusion for a downstream task without being dependent on the taxonomy of the fusion strategy. This paper introduces the Taxonomy-Guided Fusion Strategy Selector (TG-FSS), a bridging module that takes a taxonomy-alignment system (TACTM++) and a tri-stage self-supervised sentiment-fusion pipeline (TSSP-MSA), and maps the semantic values from the taxonomy to an explicit, differentiable prior distribution over fusion strategies. The strategy-attention scoring network projects the embedding of the topic (produced by Topic Embedding), the taxonomy label (produced by TaxoAlignNet), and the alignment confidence (produced by TaxoAlignNet) along with the epistemic uncertainty of the sentiment branch (per modality), to yield a distribution over 5 canonical fusion strategies. It is trained end-to-end using a three-phase joint optimization schedule, and conditions a novel taxonomy-conditioned uncertainty-aware expert mixture fusion (TC-UEMF) rule, which conditions on the prior derived from the taxonomy and the uncertainty (derived from the experts) with a learned mixing coefficient. When evaluated under the same protocol as its constituent systems on CMU-MOSEI, CMU-MOSI, and the Multimodal Fusion Strategy Corpus, TG-FSS achieves higher sentiment classification accuracy (85.4% vs. 83.6%), F1 score (0.83 vs. 0.81), and correlation with human judgment (0.81 vs. 0.78) compared to the highest-scoring prior system, while also achieving higher taxonomy alignment accuracy (90.3% vs. 87.2%). The gap to the baseline is found to increase with the level of modality occlusion, as demonstrated in robustness analysis; and the ablation experiments reveal that the improvements are driven specifically by taxonomy conditioning, and not by an increase in model capacity. Additionally, strategy-selection analysis reveals that TG-FSS learns an interpretable, domain-specific routing behaviour, without any explicit supervision at the segment level. These findings show that the taxonomy discovery and the execution of the fusion can be considered as two separate problems, but they are mutually reinforcing and complementary if they are optimized in combination; and that the resulting architecture can be performance competitive with the state-of-the-art across resource dimensions with relatively small computational overhead of 6-18%.