Repositorio Digital de la UPF
Publication
Artículo científico (JournalArticle)
Audio-visual gated-sequenced neural networks for affect recognition
The interest in automatic emotion recognition and the larger field of Affective Computing has recently gained momentum. The current emergence of large, video-based affect datasets offering rich multi-modal inputs facilitates the development of deep learning-based models for automatic affect analysis that currently holds the state of the art. However, recent approaches to process these modalities cannot fully exploit them due to the use of oversimplified fusion schemes. Furthermore, the efficient use of temporal information inherent to these huge data are also largely unexplored hindering their potential progress. In this work, we propose a multi-modal, sequence-based neural network with gating mechanisms for Valence and Arousal based affect recognition. Our model consists of three major networks: Firstly, a latent-feature generator that extracts compact representations from both modalities that have been artificially degraded to add robustness. Secondly, a multi-task discriminator that estimates both input identity and a first step emotion quadrant estimation. Thirdly, a sequence-based predictor with attention and gating mechanisms that effectively merges both modalities and uses this information through sequence modelling. In our experiments on the SEMAINE and SEWA affect datasets, we observe the impact of both proposed methods with progressive increase in accuracy. We further show in our ablation studies how the internal attention weight and gating coefficient impact our models’ estimates quality. Finally, we demonstrate state of the art accuracy through comparisons with current alternatives on both datasets., This work is partly supported by the Spanish Ministry of Science
and Innovation under project grant PID2020-114083GB-I00, and
the donation bahi2018-19 to the CMTech at UPF. Further funding
has been received from the European Union’s Horizon 2020
research and innovation programme under grant agreement No.
826506 (sustAGE) along with the UDeco project by the German
BMBF-KMU Innovativ program.
Project:
EC / H2020 826506
,
ES / 2PE PID2020-114083GB-I00