RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication
Contribución de congreso (ConferenceOutput)
Interactive-predictive neural multimodal systems
[EN] Despite the advances achieved by neural models in sequence
to sequence learning, exploited in a variety of tasks, they still make errors.
In many use cases, these are corrected by a human expert in a posterior
revision process. The interactive-predictive framework aims to minimize
the human effort spent on this process by considering partial corrections
for iteratively refining the hypothesis. In this work, we generalize the
interactive-predictive approach, typically applied in to machine translation field, to tackle other multimodal problems namely, image and video
captioning. We study the application of this framework to multimodal
neural sequence to sequence models. We show that, following this framework, we approximately halve the effort spent for correcting the outputs
generated by the automatic systems. Moreover, we deploy our systems
in a publicly accessible demonstration, that allows to better understand
the behavior of the interactive-predictive framework., The research leading to these results has received funding from MINECO under grant
IDIFEDER/2018/025 Sistemas de fabricacion inteligentes para la industria 4.0,
action co-funded by the European Regional Development Fund 2014-2020 (FEDER),
and from the European Commission under grant H2020, reference 825111 (DeepHealth). We also acknowledge NVIDIA Corporation for the donation of GPUs used
in this work.