MODALITY-AWARE OOD SUPPRESSION USING FEATURE DISCREPANCY FOR MULTI-MODAL EMOTION RECOGNITION

While conventional multi-modal emotion recognition (MER) focuses only on model learning for modality fusion, we are interested in tuning multi-modal data fed to MER model in the testing phase. Tuning the influence of each input may cause MER performance significantly due to the nature of MER datasets consisting of heterogeneous modalities. Thus, we propose a novel approach to detect and suppress a modality that is less useful for emotion prediction based on statistical differences between modality distributions. For the MER dataset annotated with discrete or continuous emotion labels, we experimentally find that the OoD modality adversely affects the prediction. Then, we show that when the proposed suppression method is attached to the backbone techniques in an ad-hoc manner, it can achieve outstanding MER performance improvement of 21%.