DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación
Contribución de congreso (ConferenceOutput)
Seeing and hearing egocentric actions: How much can we learn?
Trabajo presentado en la International Conference on Computer Vision Workshop (ICCVW), celebrada en Seúl (Corea del Sur), los días 27 y 28 de octubre de 2019, Our interaction with the world is an inherently multi-modal experience. However, the understanding of human-to-object interactions has historically been addressed focusing on a single modality. In particular, a limited number of works have considered to integrate the visual and audio modalities for this purpose. In this work, we propose a multimodal approach for egocentric action recognition in a kitchen environment that relies on audio and visual information. Our model combines a sparse temporal sampling strategy with a late fusion of audio, spatial,and temporal streams. Experimental results on the EPIC-Kitchens dataset show that multimodal integration leads to better performance than unimodal approaches. In particular, we achieved a5.18%improvement over the state of the art on verb classification., This work was partially funded by TIN2018-095232-B-C21, 2017 SGR 1742, Nestore, Validithi, 20141510 (La MaratoTV3) and CERCA Programme/Generalitat de Catalunya. A.C. is supported by a doctoral fellowship from the Mexican Council of Science
and Technology (CONACYT) (grant-no. 366596). We acknowledge
the support of NVIDIA Corporation with the donation of Titan Xp
GPUs