Skip to main content

LECTURA DE LABIOS EN ESPAÑOL EN ESCENARIOS REALISTAS

PID2021-124719OB-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprogram Subprograma Estatal de Generación de Conocimiento
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2021
State Plan 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITÈCNICA DE VALÈNCIA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 6
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Continuous lipreading based on acoustic temporal alignments

  • Gimeno-Gómez, David|||0000-0002-7375-9515
  • Martínez-Hinarejos, Carlos-D.|||0000-0002-6139-2891
[EN] Visual speech recognition (VSR) is a challenging task that has received increasing interest during the last few decades. Current state of the art employs powerful end-to-end architectures based on deep learning which depend on large amounts of data and high computational resources for their estimation. We address the task of VSR for data scarcity scenarios with limited computational resources by using traditional approaches based on hidden Markov models. We present a novel learning strategy that employs information obtained from previous acoustic temporal alignments to improve the visual system performance. Furthermore, we studied multiple visual speech representations and how image resolution or frame rate affect its performance. All these experiments were conducted on the limited data VLRF corpus, a database which offers an audio-visual support to address continuous speech recognition in Spanish. The results show that our approach significantly outperforms the best results achieved on the task to date., This work was partially supported by Grant CIACIF/2021/295 funded by Generalitat Valenciana and by Grant PID2021-124719OB-I00 under project LLEER (PID2021-124719OB-100) funded by MCIN/AEI/10.13039/501100011033/ and by ERDF, EU A way of making Europe.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson&apos, s Diagnosis

  • Gimeno-Gómez, David|||0000-0002-7375-9515
  • Martínez-Hinarejos, Carlos-D.|||0000-0002-6139-2891
  • Botelho, Catarina
  • Pompili, Anna
  • Abad, Alberto
[EN] Recent works in pathological speech analysis have increasingly relied on powerful self-supervised speech representations, leading to promising results. However, the complex, black-box nature of these embeddings and the limited research on their interpretability significantly restrict their adoption for clinical diagnosis. To address this gap, we propose a novel, interpretable framework specifically designed to support Parkinson's Disease (PD) diagnosis. Through the design of simple yet effective cross-attention mechanisms for both embedding- and temporal-level analysis, the proposed framework offers interpretability from two distinct but complementary perspectives. Experimental findings across five well-established speech benchmarks for PD detection demonstrate the framework's capability to identify meaningful speech patterns within self-supervised representations for a wide range of assessment tasks. Fine-grained temporal analyses further underscore its potential to enhance the interpretability of deep-learning pathological speech models, paving the way for the development of more transparent, trustworthy, and clinically applicable computer-assisted diagnosis systems in this domain. Moreover, in terms of classification accuracy, our method achieves results competitive with state-of-the-art approaches, while also demonstrating robustness in cross-lingual scenarios when applied to spontaneous speech production., The work of David Gimeno-Gómez and Carlos-D. Martínez-Hinarejos was supported in part by GVA under Grant CIACIF/2021/295 and Grant CIBEFP/2023/167, and in part by MCIN/AEI and ERDF, EU A way of making Europe through Project LLEER under Grant PID2021-124719OB-I00. The work of Catarina Botelho, Anna Pompili, and Albetro Abad was supported in part by Portuguese National funds through Fundação para a Ciência e a Tecnologia, with reference DOI: 10.54499/UIDB/50021/2020, and in part by Portuguese Recovery and Resilience Plan and NextGenerationEU European Union Funds under Project C644865762-00000008 (Accelerat.AI). This work was supported by CRUE-Universitat Politècnica de València.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions

  • Gimeno-Gómez, David|||0000-0002-7375-9515
  • Martínez-Hinarejos, Carlos-D.|||0000-0002-6139-2891
[EN] Visual speech recognition remains an open research problem where different challenges must be considered by dispensing with the auditory sense, such as visual ambiguities, the inter-personal variability among speakers, and the complex modeling of silence. Nonetheless, recent remarkable results have been achieved in the field thanks to the availability of large-scale databases and the use of powerful attention mechanisms. Besides, multiple languages apart from English are nowadays a focus of interest. This paper presents noticeable advances in automatic continuous lipreading for Spanish. First, an end-to-end system based on the hybrid CTC/Attention architecture is presented. Experiments are conducted on two corpora of disparate nature, reaching state-of-the-art results that significantly improve the best performance obtained to date for both databases. In addition, a thorough ablation study is carried out, where it is studied how the different components that form the architecture influence the quality of speech recognition. Then, a rigorous error analysis is carried out to investigate the different factors that could affect the learning of the automatic system. Finally, a new Spanish lipreading benchmark is consolidated., This work was partially supported by Grant CIACIF/2021/295 funded by Generalitat Valenciana and by Grant PID2021-124719OB-I00 under project LLEER (PID2021-124719OB-100) funded by MCIN/AEI/10.13039/501100011033/ and by ERDF, EU A way of making Europe.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Comparing Speaker Adaptation Methods for Visual Speech Recognition for Continuous Spanish

  • Gimeno-Gómez, David|||0000-0002-7375-9515
  • Martínez-Hinarejos, Carlos-D.|||0000-0002-6139-2891
[EN] Visual speech recognition (VSR) is a challenging task that aims to interpret speech based solely on lip movements. However, although remarkable results have recently been reached in the field, this task remains an open research problem due to different challenges, such as visual ambiguities, the intra-personal variability among speakers, and the complex modeling of silence. Nonetheless, these challenges can be alleviated when the task is approached from a speaker-dependent perspective. Our work focuses on the adaptation of end-to-end VSR systems to a specific speaker. Hence, we propose two different adaptation methods based on the conventional fine-tuning technique or the so-called Adapters. We conduct a comparative study in terms of performance while considering different deployment aspects such as training time and storage cost. Results on the Spanish LIP-RTVE database show that both methods are able to obtain recognition rates comparable to the state of the art, even when only a limited amount of training data is available. Although it incurs a deterioration in performance, the Adapters-based method presents a more scalable and efficient solution, significantly reducing the training time and storage cost by up to 80%., This work was partially supported by the Grant CIACIF/2021/295 funded by Generalitat Valenciana and by the Grant PID2021-124719OB-I00 under the LLEER (PID2021-124719OB-100) project funded by MCIN/AEI/10.13039/501100011033/ and by ERDF EU, A way of making Europe .
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Tailored design of Audio-Visual Speech Recognition models using Branchformers

  • Gimeno-Gómez, David|||0000-0002-7375-9515
  • Martínez-Hinarejos, Carlos-D.|||0000-0002-6139-2891
[EN] Recent advances in Audio¿Visual Speech Recognition (AVSR) have led to unprecedented achievements in the field, improving the robustness of this type of system in adverse, noisy environments. In most cases, this task has been addressed through the design of models composed of two independent encoders, each dedicated to a specific modality. However, while recent works have explored unified audio¿visual encoders, determining the optimal cross-modal architecture remains an ongoing challenge. Furthermore, such approaches often rely on models comprising vast amounts of parameters and high computational cost training processes. In this paper, we aim to bridge this research gap by introducing a novel audio¿visual framework. Our proposed method constitutes, to the best of our knowledge, the first attempt to harness the flexibility and interpretability offered by encoder architectures, such as the Branchformer, in the design of parameter-efficient AVSR systems. To be more precise, the proposed framework consists of two steps: first, estimating audio- and video-only systems, and then designing a tailored audio¿visual unified encoder based on the layer-level branch scores provided by the modality-specific models. Extensive experiments on English and Spanish AVSR benchmarks covering multiple data conditions and scenarios demonstrated the effectiveness of our proposed method. Even when trained on a moderate scale of data, our models achieve competitive word error rates (WER) of approximately 2.5% for English and surpass existing approaches for Spanish, establishing a new benchmark with an average WER of around 9.1%. These results reflect how our tailored AVSR system is able to reach state-of-the-art recognition rates while significantly reducing the model complexity w.r.t. the prevalent approach in the field., This work was partially supported by Generalitat Valenciana, Spain through Grant CIACIF/2021/295 and Conselleria de Educacion, Universidades y Empleo under project LightVED, funded by the PROMETEO 2024 program (CIPROM/2023/17) . It was also supported by Grant PID2021-124719OB-I00 under project LLEER, funded by MCIN/AEI and ERDF, EU "A way of making Europe".
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Using LoRA and Reinforcement Learning in Interactive Machine Translation

  • Navarro-Martínez, Ángel|||0000-0002-5957-8152
  • Casacuberta Nolla, Francisco|||0000-0002-8497-5598
[EN] The use of large language models (LLMs) is rapidly expanding due to their impressive performance across various tasks . However, as newer versions continue to improve results, their increasing size poses challenges for maintaining multiple domain-specialized versions. The Low-Rank Adaptation (LoRA) method offers a solution to this limitation by enabling fine-tuning modifications to be stored in a file of just a few megabytes, significantly reducing storage requirements. In Machine Translation (MT), models are often specialized for specific domains or language pairs. In our case, we apply these models within Interactive Machine Translation (IMT), where generating high-quality translations and adapting effectively to user modifications are crucial. We have integrated Reinforcement Learning (RL) techniques, optimizing the model using various evaluation metrics. Our results demonstrate that these methods effectively improve translation quality; however, in some cases, this improvement comes at the cost of a slight reduction in generalization capability., This paper is part of the project I+D+i PID2023-148259NB-I00 (LeMUR), funded by Ministerio de Ciencia e Innovación. The second author is supported by grant ACIF/2021/356 from the Programa I+D+i de la Generalitat Valenciana .