Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 24
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

From Simultaneous to Streaming Machine Translation by Leveraging Streaming History

  • Iranzo-Sánchez, Javier
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] Simultaneous Machine Translation is the task of incrementally translating an input sentence before it is fully available. Currently, simultaneous translation is carried out by translating each sentence independently of the previously translated text. More generally, Streaming MT can be understood as an extension of Simultaneous MT to the incremental translation of a continuous input text stream. In this work, a state-of-the-art simultaneous sentencelevel MT system is extended to the streaming setup by leveraging the streaming history. Extensive empirical results are reported on IWSLT Translation Tasks, showing that leveraging the streaming history leads to significant quality gains. In particular, the proposed system proves to compare favorably to the best performing systems., The research leading to these results has received funding from the European Union¿s Horizon 2020 research and innovation programme under grant agreements no. 761758 (X5Gon) and 952215 (TAILOR), and Erasmus+ Education programme under grant agreement no. 20-226-093604-SCH (EXPERT); the Government of Spain¿s grant RTI2018-094879-B-I00 (Multisub) funded by MCIN/AEI/10.13039/501100011033 & ¿ERDF A way of making Europe¿, and FPU scholarships FPU18/04135; and the Generalitat Valenciana¿s research project Classroom Activity Recognition (ref. PROMETEO/2019/111). The authors gratefully acknowledge the computer resources at Artemisa, funded by the European Union ERDF and Comunitat Valenciana as well as the technical support provided by the Instituto de Física Corpuscular, IFIC (CSIC-UV).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

LSTM-Based One-Pass Decoder for Low-Latency Streaming

  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
  • Iranzo-Sánchez, Javier
[EN] Current state-of-the-art models based on Long-Short Term Memory (LSTM) networks have been extensively used in ASR to improve performance. However, using LSTMs under a streaming setup is not straightforward due to real-time constraints. In this paper we present a novel streaming decoder that includes a bidirectional LSTM acoustic model as well as an unidirectional LSTM language model to perform the decoding efficiently while keeping the performance comparable to that of an off-line setup. We perform a one-pass decoding using a sliding window scheme for a bidirectional LSTM acoustic model and an LSTM language model. This has been implemented and assessed under a pure streaming setup, and deployed into our production systems. We report WER and latency figures for the well-known LibriSpeech and TED-LIUM tasks, obtaining competitive WER results with low-latency responses., The research leading to these results has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon);MCIU/AEI/FEDER,UE under the Multisub (RTI2018-094879-B-I00) research project and the Government of Spains FPU scholarship FPU18/04135.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

The MLLP-UPV Supervised Machine Translation Systems for WMT19 News Translation Task

  • Iranzo-Sánchez, Javier
  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] This paper describes the participation of the MLLP research group of the Universitat Politècnica de València in the WMT 2019 News Translation Shared Task. In this edition, we have submitted systems for the German ¿ English and German ¿ French language pairs, participating in both directions of each pair. Our submitted systems, based on the Transformer architecture, make ample use of data filtering, synthetic data and domain adaptation through fine-tuning., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon); the Government of Spain¿s research project Multisub, ref. RTI2018-094879-B-I00 (MCIU/AEI/FEDER, EU); and the Universitat Politecnica de València's PAID-01-17 R&D support programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Tesis doctoral (DoctoralThesis)

Streaming Neural Speech Translation

  • Iranzo Sánchez, Javier
[ES] Gracias a avances significativos en aprendizaje profundo, la traducción del habla (ST) se ha convertido en un campo consolidado, lo que permite la utilización de la tecnología ST en soluciones para entornos de producción. Como consecuencia del aumento constante del número de horas de contenido audiovisual generado cada año, así como una mayor sensibilización sobre la importancia de la accesibilidad, la ST está preparada para convertirse en un elemento clave para la producción de contenidos audiovisuales, tanto de ocio como educativos. A pesar de que se ha progresado significativamente en ST, la mayor parte de la investigación se ha centrado en el escenario en diferido (offline), en el cual todo el audio de entrada está disponible. En cambio, la ST en directo (online) es una temática en la que falta mucho por investigar. En concreto, existe un caso de traducción en directo, la traducción continua (streaming), que traduce un flujo continuo de palabras en tiempo real y bajo unas estrictas condiciones de latencia. Este es un problema mucho más realista, que es necesario resolver para que sea posible aplicar la ST a una variedad de tareas de la vida real. Esta tesis está centrada en investigar y desarrollar las técnicas claves que son necesarias para una solución de ST continua. En primer lugar, de cara a permitir el desarrollo y la evaluación de sistemas de ST, se ha recopilado un nuevo conjunto de datos para ST multilingüe, que expande significativamente el número de horas disponibles para ST. A continuación se ha desarrollado un segmentador preparado para la condición continua, que se utiliza para segmentar las transcripciones intermedias de nuestra solución por etapas, que consiste en un sistema de reconocimiento automático del habla (ASR), seguido de un sistema de traducción automática (MT) encargado de traducir las transcripciones intermedias al idioma de destino elegido. Diversas investigaciones han concluido que la calidad de la segmentación es un factor muy influyente es la calidad del sistema MT, por lo que el desarrollo de un segmentador efectivo es un paso fundamental en el proceso de ST continua. Este segmentador se ha integrado en la solución por etapas, y estas se optimizan de manera conjunta para alcanzar el equilibrio óptimo entre calidad y latencia. La ST continua tiene unas restricciones de latencia mucho más estrictas que la ST en directo, ya que el nivel deseado de latencia tiene que mantenerse durante todo el proceso de traducción. Por tanto, es crucial ser capaz de medir de manera precisa esta latencia, pero las métricas estándar de ST en directo no se adaptan bien a esta tarea. Como consecuencia de esto, se proponen nuevos métodos para la evaluación de ST continua, que garantizan unos resultados precisos a la vez que interpretables. Por último, se presenta un nuevo método para mejorar la calidad de la traducción continua mediante el uso de información contextual. Mientras que los sistemas tradicionales de ST en directo traducen audios de manera aislada, existe abundante información contextual que está disponible para mejorar los sistemas de ST continua. Nuestra propuesta introduce el concepto de historia continua, que consiste en el almacenamiento de la información más reciente del proceso de traducción, que se utiliza más adelante por el modelo para mejorar la calidad de la traducción., [CA] Gràcies a avanços significatius en aprenentatge profund, la traducció de la parla (ST) s'ha convertit en un camp consolidat, la qual cosa permet la utilització de la tecnologia ST en solucions per a entorns de producció. A conseqüència de l'augment constant del nombre d'hores de contingut audiovisual generat cada any, així com una major sensibilització sobre la importància de l'accessibilitat, la ST està preparada per a convertir-se en un element clau per a la producció de continguts audiovisuals, tant d'oci com educatius. A pesar que s'ha progressat significativament en ST, la major part de la recerca s'ha centrat en l'escenari en diferit, en el qual tot l'àudio d'entrada està disponible. En canvi, la ST en directe és una temàtica en la qual falta molt per investigar. En concret, existeix un cas de traducció en directe, la traducció contínua, que tradueix un flux continu de paraules en temps real i sota unes estrictes condicions de latència. Aquest és un problema molt més realista, que és necessari resoldre perquè sigui possible aplicar la ST a una varietat de tasques de la vida real. Aquesta tesi està centrada en investigar i desenvolupar les tècniques claus que són necessàries per a una solució de ST contínua. En primer lloc, de cara a permetre el desenvolupament i l'avaluació de sistemes de ST, s'ha recopilat un nou conjunt de dades per a ST multilingüe, que expandeix significativament la quantitat de dades disponibles per a ST. A continuació s'ha desenvolupat un segmentador preparat per a la condició contínua, que s'utilitza per a segmentar les transcripcions intermèdies de la nostra solució per etapes, que consisteix en un sistema de reconeixement automàtic de la parla (ASR), seguit d'un sistema de traducció automàtica (MT) encarregat de traduir les transcripcions intermèdies a l'idioma de destí triat. Diveros treballs de recerca han conclòs que la qualitat de la segmentació és un factor molt important en la qualitat del sistema MT, per la qual cosa el desenvolupament d'un segmentador efectiu és un pas fonamental en el procés de ST contínua. Aquest segmentador s'ha integrat en la solució per etapes, i aquestes s'optimitzen de manera conjunta per a aconseguir l'equilibri òptim entre qualitat i latència. La ST contínua té unes restriccions de latència molt més estrictes que la ST en directe, ja que el nivell desitjat de latència ha de mantindre's durant tot el procés de traducció. Per tant, és crucial ser capaç de mesurar de manera precisa aquesta latència, però les mètriques estàndard de ST en directe no s'adapten bé a aquesta tasca. A conseqüència d'això, es proposen nous mètodes per a l'avaluació de ST contínua, que garanteixen uns resultats precisos alhora que interpretables. Finalment, es presenta un nou mètode per a millorar la qualitat de la traducció contínua mitjançant l'ús d'informació contextual. Mentre que els sistemes tradicionals de ST en directe tradueixen àudios de manera aïllada, existeix abundant informació contextual que està disponible per a millorar els sistemes de ST contínua. La nostra proposta introdueix el concepte d'història contínua, que consisteix en l'emmagatzematge de la informació més recent del procés de traducció, que s'utilitza més endavant pel model per a millorar la qualitat de la traducció., [EN] Thanks to significant advances in Deep Learning, Speech Translation (ST) has become a mature field that enables the use of ST technology in production-ready solutions. Due to the ever-increasing hours of audio-visual content produced each year, as well as higher awareness of the importance of media accessibility, ST is poised to become a key element for the production of entertainment and educational media. Although significant advances have been made in ST, most research has focused on the offline scenario, where the entire input audio is available. In contrast, online ST remains an under-researched topic. A special case of online ST, streaming ST, translates an unbounded input stream in a real-time fashion under strict latency constraints. This is a much more realistic problem that needs to be solved in order to apply ST to a variety of real-life tasks. The focus of this thesis is on researching and developing key techniques necessary for a successful streaming ST solution. First, in order to enable ST system development and evaluation, a new multilingual ST dataset is collected, which significantly expands the amount of hours available for ST. Then, a streaming-ready segmenter component is developed to segment the intermediate transcriptions of our proposed cascade solution, which consists in an Automatic Speech Recognition (ASR) system that transcribes the audio, followed by a Machine Translation (MT) system that translates the intermediate transcriptions into the desired language. Research has shown that segmentation quality plays a significant role in downstream MT performance, so the development of an effective streaming segmenter is a critical step in the streaming ST process. This segmenter is then integrated and the components of the cascade are jointly optimized to achieve an appropriate quality-latency trade-off. Streaming ST has much more strict latency constraints than standard online ST, as the desired latency level must be maintained during the whole translation process. Therefore, it is crucial to be able to accurately measure this latency, but the standard online ST metrics are not well suited for this task. As a consequence, new evaluation methods are proposed for streaming ST evaluation, which ensure realistic, yet interpretable results. Lastly, a novel method is presented for improving translation quality through the use of contextual information. Whereas standard online ST systems translate audios in isolation, there is a wealth of contextual information available for improving streaming ST systems. Our approach introduces the concept of streaming history by storing the most recent information of the translation process, which is then used by the model in order to improve translation quality., The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements no. 761758 (X5Gon) and 952215 (TAILOR), and Erasmus+ Educa- tion programme under grant agreement no. 20-226-093604-SCH (EXPERT); the Government of Spain’s grant RTI2018-094879-B-I00 (Multisub) funded by MCIN/AEI/10.13039/501100011033 & “ERDF A way of making Europe”, and FPU scholarships FPU18/04135; and the Generalitat Valenciana’s research project Classroom Activity Recognition (ref. PROMETEO/2019/111) and predoctoral research scholarship ACIF/2017/055.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Europarl-ST: A Multilingual Corpus for Speech Translation of Parliamentary Debates

  • Iranzo-Sánchez, Javier
  • Roselló, Nahuel
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] Current research into spoken language translation (SLT), or speech-to-text translation, is often hampered by the lack of specific data resources for this task, as currently available SLT datasets are restricted to a limited set of language pairs. In this paper we present Europarl-ST, a novel multilingual SLT corpus containing paired audio-text samples for SLT from and into 6 European languages, for a total of 30 different translation directions. This corpus has been compiled using the debates held in the European Parliament in the period between 2008 and 2012. This paper describes the corpus creation process and presents a series of automatic speech recognition, machine translation and spoken language translation experiments that highlight the potential of this new resource. The corpus is released under a Creative Commons license and is freely accessible and downloadable., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon); MCIU/AEI/FEDER,UE under the Multisub (RTI2018-094879-B-I00) research project and the Government of Spain's FPU scholarship FPU18/04135.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

MLLP-VRAIN Spanish ASR Systems for the Albayzín-RTVE 2020 Speech-to-Text Challenge: Extension

  • Baquero-Arnal, Pau
  • Iranzo-Sánchez, Javier
  • Pérez-González de Martos, Alejandro Manuel
  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
[EN] This paper describes the automatic speech recognition (ASR) systems built by the MLLP-VRAIN research group of Universitat Politècnica de València for the Albayzín-RTVE 2020 Speech-to-Text Challenge, and includes an extension of the work consisting of building and evaluating equivalent systems under the closed data conditions from the 2018 challenge. The primary system (p-streaming_1500ms_nlt) was a hybrid ASR system using streaming one-pass decoding with a context window of 1.5 seconds. This system achieved 16.0% WER on the test-2020 set. We also submitted three contrastive systems. From these, we highlight the system c2-streaming_600ms_t which, following a similar configuration as the primary system with a smaller context window of 0.6 s, scored 16.9% WER points on the same test set, with a measured empirical latency of 0.81 ± 0.09 s (mean ± stdev). That is, we obtained state-of-the-art latencies for high-quality automatic live captioning with a small WER degradation of 6% relative. As an extension, the equivalent closed-condition systems obtained 23.3% WER and 23.5% WER, respectively. When evaluated with an unconstrained language model, we obtained 19.9% WER and 20.4% WER; i.e., not far behind the top-performing systems with only 5% of the full acoustic data and with the extra ability of being streaming-capable. Indeed, all of these streaming systems could be put into production environments for automatic captioning of live media streams., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreements no. 761758 (X5Gon) and 952215 (TAILOR), and Erasmus+ Education programme under grant agreement no. 20-226-093604-SCH (EXPERT); the Government of Spain's grant RTI2018-094879-B-I00 (Multisub) funded by MCIN/AEI/10.13039/501100011033 & "ERDF A way of making Europe", and FPU scholarships FPU14/03981 and FPU18/04135; the Generalitat Valenciana's research project Classroom Activity Recognition (ref. PROMETEO/2019/111), and predoctoral research scholarship ACIF/2017/055; and the Universitat Politecnica de Valencia's PAID-01-17 R&D support programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Real-Time One-Pass Decoder for Speech Recognition Using LSTM Language Models

  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
  • Iranzo-Sánchez, Javier
Recurrent Neural Networks, in particular Long-Short TermMemory (LSTM) networks, are widely used in Automatic Speech Recognition for language modelling during decoding,usually as a mechanism for rescoring hypothesis. This paperproposes a new architecture to perform real-time one-pass de-coding using LSTM language models. To make decoding ef-ficient, the estimation of look-ahead scores was accelerated byprecomputing static look-ahead tables. These static tables wereprecomputed from a prunedn-gram model, reducing drasti-cally the computational cost during decoding. Additionally,the LSTM language model evaluation was efficiently performedusing Variance Regularization along with a strategy of lazyevaluation. The proposed one-pass decoder architecture wasevaluated on the well-known LibriSpeech and TED-LIUMv3datasets. Results showed that the proposed algorithm obtainsvery competitive WERs with 0.6 RTFs. Finally, our one-passdecoder is compared with a decoupled two-pass decoder., This research has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon) and the TIN2015-68326-R and RTI2018-094879-B-I00 (MINECO/MCIU/AEI/FEDER,UE) research projects, and the grant FPU14/03981 from the Spanish Ministry of Education, Culture and Sport.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

MLLP-VRAIN UPV systems for the IWSLT 2022 Simultaneous Speech Translation and Speech-to-Speech Translation tasks

  • Iranzo-Sánchez, Javier
  • Pérez-González de Martos, Alejandro
  • Baquero-Arnal, Pau
  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
[EN] This work describes the participation of the MLLP-VRAIN research group in the two shared tasks of the IWSLT 2022 conference: Simultaneous Speech Translation and Speech-to-Speech Translation. We present our streaming-ready ASR, MT and TTS systems for Speech Translation and Synthesis from English into German. Our submission combines these systems by means of a cascade approach paying special attention to data preparation and decoding for streaming inference., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreements no. 761758 (X5Gon) and 952215 (TAILOR), and Erasmus+ Education programme under grant agreement no. 20- 226-093604-SCH (EXPERT); the Government of Spain¿s grant RTI2018-094879-B-I00 (Multisub) funded by MCIN/AEI/10.13039/501100011033 & ERDF A way of making Europe, and FPU scholarships FPU18/04135; and the Generalitat Valenciana¿s research project Classroom Activity Recognition (ref. PROMETEO/2019/111).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

The MLLP-UPV Spanish-Portuguese and Portuguese-Spanish Machine Translation Systems for WMT19 Similar Language Translation Task

  • Baquero-Arnal, Pau
  • Iranzo-Sánchez, Javier
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] This paper describes the participation of the MLLP research group of the Universitat Politècnica de València in the WMT 2019 Similar Language Translation Shared Task. We have submitted systems for the Portuguese ↔ Spanish language pair, in both directions. We have submitted systems based on the Transformer architecture as well as an in development novel architecture which we have called 2D alternating RNN. We have carried out domain adaptation through fine-tuning., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon); the Government of Spain's research project Multisub, ref. RTI2018-094879-B-I00 (MCIU/AEI/FEDER, EU) and the Generalitat Valenciana's predoctoral research scholarship ACIF/2017/055.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Streaming cascade-based speech translation leveraged by a direct segmentation model

  • Iranzo-Sánchez, Javier
  • Baquero-Arnal, Pau
  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Giménez Pastor, Adrián
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
[EN] The cascade approach to Speech Translation (ST) is based on a pipeline that concatenates an Automatic Speech Recognition (ASR) system followed by a Machine Translation (MT) system. Nowadays, state-of-the-art ST systems are populated with deep neural networks that are conceived to work in an offline setup in which the audio input to be translated is fully available in advance. However, a streaming setup defines a completely different picture, in which an unbounded audio input gradually becomes available and at the same time the translation needs to be generated under real-time constraints. In this work, we present a state-of-the-art streaming ST system in which neural-based models integrated in the ASR and MT components are carefully adapted in terms of their training and decoding procedures in order to run under a streaming setup. In addition, a direct segmentation model that adapts the continuous ASR output to the capacity of simultaneous MT systems trained at the sentence level is introduced to guarantee low latency while preserving the translation quality of the complete ST system. The resulting ST system is thoroughly evaluated on the real-life streaming Europarl-ST benchmark to gauge the trade-off between quality and latency for each component individually as well as for the complete ST system., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement no. 761758 (X5Gon) and 952215 (TAILOR); the Government of Spain's research project Multisub, ref. RTI2018-094879-B-I00 (MCIU/AEI/FEDER,EU) and FPU scholarships FPU14/03981 and FPU18/04135; and the Generalitat Valenciana's research project Classroom Activity Recognition, ref. PROMETEO/2019/111 and predoctoral research scholarship ACIF/2017/055.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

The MLLP-UPV German-English Machine Translation System for WMT18

  • Iranzo-Sánchez, Javier
  • Baquero-Arnal, Pau
  • Martínez-Villaronga, Adrià
  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] This paper describes the statistical machine translation system built by the MLLP research group of Universitat Politècnica de València for the German¿English news translation shared task of the EMNLP 2018 Third Conference on Machine Translation (WMT18). We used an ensemble of Transformer architecture¿based neural machine translation systems. To train our system under ¿constrained¿ conditions, we filtered the provided parallel data with a scoring technique using character-based language models, and we added parallel data based on synthetic source sentences generated from the provided monolingual corpora., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon); the Spanish government's TIN2015-68326-R (MINECO/FEDER) research project MORE, university collaboration grant programme 2017-2018, and faculty training scholarship FPU13/06241; the Generalitat Valenciana's predoctoral research scholarship ACIF/2017/055; as well as the Universitat Politecnica de València's PAID-01-17 R&D support programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Europarl-ASR: A Large Corpus of Parliamentary Debates for Streaming ASR Benchmarking and Speech Data Filtering/Verbatimization

  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
  • Iranzo-Sánchez, Javier
  • Baquero-Arnal, Pau
  • Roselló, Nahuel
  • Pérez-González de Martos, Alejandro Manuel
[EN] We introduce Europarl-ASR, a large speech and text corpus of parliamentary debates including 1300 hours of transcribed speeches and 70 million tokens of text in English extracted from European Parliament sessions. The training set is labelled with the Parliament¿s non-fully-verbatim official transcripts, time-aligned. As verbatimness is critical for acoustic model training, we also provide automatically noise-filtered and automatically verbatimized transcripts of all speeches based on speech data filtering and verbatimization techniques. Additionally, 18 hours of transcribed speeches were manually verbatimized to build reliable speaker-dependent and speaker-independent development/test sets for streaming ASR benchmarking. The availability of manual non-verbatim and verbatim transcripts for dev/test speeches makes this corpus useful for the assessment of automatic filtering and verbatimization techniques. This paper describes the corpus and its creation, and provides off-line and streaming ASR baselines for both the speaker-dependent and speaker-independent tasks using the three training transcription sets. The corpus is publicly released under an open licence., [CA] "Europarl-ASR: Un extens corpus parlamentari de referència per a reconeixement de la parla i filtratge/literalització de transcripcions": Presentem Europarl-ASR, un extens corpus de veu i text de debats parlamentaris amb 1300 hores d'intervencions transcrites i 70 milions de paraules de text en anglés extrets de sessions del Parlament Europeu. Les transcripcions oficials del Parlament Europeu, no literals, s'han sincronitzat per a tot el conjunt d'entrenament. Com que l'entrenament de models acústics requereix transcripcions com més literals millor, també s'han inclòs transcripcions filtrades i transcripcions literalitzades de totes les intervencions, basades en tècniques de filtratge i literalització automàtics. A més, s'han inclòs 18 hores de transcripcions literals revisades manualment per definir dos conjunts de validació i avaluació de referència per a reconeixement automàtic de la parla en temps real, amb oradors coneguts i amb oradors desconeguts. Pel fet de disposar de transcripcions literals i no literals, aquest corpus és també ideal per a l'anàlisi de tècniques de filtratge i de literalització. En aquest article, es descriu la creació del corpus i es proporcionen mesures de referència de reconeixement automàtic de la parla en temps real i en diferit, amb oradors coneguts i amb oradors desconeguts, usant els tres conjunts de transcripcions d'entrenament. El corpus es fa públic amb una llicència oberta., This work has received funding from the EU's H2020 research and innovation programme under grant agreements 761758 (X5gon) and 952215 (TAILOR); the Government of Spain's research project Multisub (RTI2018-094879-B-I00, MCIU/AEI/FEDER,EU) and FPU scholarships FPU14/03981 and FPU18/04135; the Generalitat Valenciana's research project Classroom Activity Recognition (PROMETEO/2019/111) and predoctoral research scholarship ACIF/2017/055; and the Universitat Politecnica de València's ` PAID-01-17 R&D support programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Towards cross-lingual voice cloning in higher education

  • Pérez-González de Martos, Alejandro Manuel
  • Jiménez, Manuel
  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
  • Giménez Pastor, Adrián
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Turró Ribalta, Carlos|||0000-0002-3840-9405
  • Juan, Alfons|||0000-0002-9984-4072
[EN] The rapid progress of modern AI tools for automatic speech recognition and machine translation is leading to a progressive cost reduction to produce publishable subtitles for educational videos in multiple languages. Similarly, text-to-speech technology is experiencing large improvements in terms of quality, flexibility and capabilities. In particular, state-of-the-art systems are now capable of seamlessly dealing with multiple languages and speakers in an integrated manner, thus enabling lecturer¿s voice cloning in languages she/he might not even speak. This work is to report the experience gained on using such systems at the Universitat Politècnica de València (UPV), mainly as a guidance for other educational organizations willing to conduct similar studies. It builds on previous work on the UPV¿s main repository of educational videos, MediaUPV, to produce multilingual subtitles at scale and low cost. Here, a detailed account is given on how this work has been extended to also allow for massive machine dubbing of MediaUPV. This includes collecting 59 h of clean speech data from UPV¿s academic staff, and extending our production pipeline of subtitles with a state-of-the-art multilingual and multi-speaker text-to-speech system trained from the collected data. Our main result comes from an extensive, subjective evaluation of this system by lecturers contributing to data collection. In brief, it is shown that text-to-speech technology is not only mature enough for its application to MediaUPV, but also needed as soon as possible by students to improve its accessibility and bridge language barriers., We wish first to thank all UPV lecturers who made this study possi-ble. We are also very grateful for the funding support received by the European Union's Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon) , the Spanish government under grant RTI2018-094879-B-I00 (Multisub, MCIU/AEI/FEDER) , and the Universitat Politecnica de Valencia's, Spain PAID-01-17 R&D sup-port programme. Funding for open access charge: CRUE-Universitat Politecnica de Valencia
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Improved Hybrid Streaming ASR with Transformer Language Models

  • Baquero-Arnal, Pau
  • Iranzo-Sánchez, Javier
  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] Streaming ASR is gaining momentum due to its wide applicability, though it is still unclear how best to come close to the accuracy of state-of-the-art off-line ASR systems when the output must come within a short delay after the incoming audio stream. Following our previous work on streaming one-pass decoding with hybrid ASR systems and LSTM language models, in this work we report further improvements by replacing LSTMs with Transformer models. First, two key ideas are discussed so as to run these models fast during inference. Then, empirical results on LibriSpeech and TED-LIUM are provided showing that Transformer language models lead to improved recognition rates on both tasks. ASR systems obtained in this work can be seamlessly transfered to a streaming setup with minimal quality losses. Indeed, to the best of our knowledge, no better results have been reported on these tasks when assessed under a streaming setup., The research leading to these results has received funding from the European Union s Horizon 2020 research and innovation program under grant agreement no. 761758 (X5Gon); the Government of Spain s research project Multisub, ref. RTI2018-094879-B-I00 (MCIU/AEI/FEDER,EU); and the Generalitat Valenciana predoctoral research scholarship ACIF/2017/055
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Stream-level Latency Evaluation for Simultaneous Machine Translation

  • Iranzo-Sánchez, Javier
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between translation quality and response time, and with this purpose multiple latency measures have been proposed. However, latency evaluations for simultaneous translation are estimated at the sentence level, not taking into account the sequential nature of a streaming scenario. Indeed, these sentence-level latency measures are not well suited for continuous stream translation, resulting in figures that are not coherent with the simultaneous translation policy of the system being assessed. This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation, that is successfully evaluated on streaming conditions for a reference IWSLT task., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement no. 761758 (X5Gon) and 952215 (TAILOR) and Erasmus+ Education program under grant agreement no. 20-226-093604-SCH; the Government of Spain's research project Multisub, ref. RTI2018-094879-B-I00 (MCIU/AEI/FEDER,EU) and FPU scholarships FPU18/04135; and the Generalitat Valenciana's research project Classroom Activity Recognition, ref. PROMETEO/2019/111.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Towards Simultaneous Machine Interpretation

  • Pérez-González de Martos, Alejandro Manuel
  • Iranzo-Sánchez, Javier
  • Giménez Pastor, Adrián
  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
[EN] Automatic speech-to-speech translation (S2S) is one of the most challenging speech and language processing tasks, especially when considering its application to real-time settings. Recent advances on streaming Automatic Speech Recognition (ASR), simultaneous Machine Translation (MT) and incremental neural Text-To-Speech (TTS) make it possible to develop real-time cascade S2S systems with greatly improved accuracy. On the way to simultaneous machine interpretation, a state-of-the-art cascade streaming S2S system is described and empirically assessed in the simultaneous interpretation of European Parliament debates. We pay particular attention to the TTS component, particularly in terms of speech naturalness under a variety of response-time settings, as well as in terms of speaker similarity for its cross-lingual voice cloning capabilities., Work supported by the EU s H2020 research and innovation programme under grant agreement no. 761758 (X5gon), the Spanish government under grant RTI2018-094879-B-I00 (Multisub, MCIU/AEI/FEDER), and the Universitat Politecnica de Valencia s PAID-01-17 R&D support programme
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Direct Segmentation Models for Streaming Speech Translation

  • Iranzo-Sánchez, Javier
  • Baquero-Arnal, Pau
  • Giménez Pastor, Adrián
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] The cascade approach to Speech Translation (ST) is based on a pipeline that concatenates an Automatic Speech Recognition (ASR) system followed by a Machine Translation (MT) system. These systems are usually connected by a segmenter that splits the ASR output into, hopefully, semantically self-contained chunks to be fed into the MT system. This is specially challenging in the case of streaming ST, where latency requirements must also be taken into account. This work proposes novel segmentation models for streaming ST that incorporate not only textual, but also acoustic information to decide when the ASR output is split into a chunk. An extensive and thorough experimental setup is carried out on the Europarl-ST dataset to prove the contribution of acoustic information to the performance of the segmentation model in terms of BLEU score in a streaming ST scenario. Finally, comparative results with previous work also show the superiority of the segmentation models proposed in this work., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement no. 761758 (X5Gon); the Government of Spain's research project Multisub, ref. RTI2018- 094879-B-I00 (MCIU/AEI/FEDER,EU), the Generalitat Valenciana's research project Classroom Activity Recognition, ref. PROMETEO/2019/111., FPU scholarship FPU18/04135; and the Generalitat Valencianas predoctoral research scholarship ACIF/2017/055. The authors wish to thank the anonymous reviewers for their criticisms and suggestions.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Live Streaming Speech Recognition Using Deep Bidirectional LSTM Acoustic Models and Interpolated Language Models

  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
[EN] Although Long-Short Term Memory (LSTM) networks and deep Transformers are now extensively used in offline ASR, it is unclear how best offline systems can be adapted to work with them under the streaming setup. After gaining considerable experience on this regard in recent years, in this paper we show how an optimized, low-latency streaming decoder can be built in which bidirectional LSTM acoustic models, together with general interpolated language models, can be nicely integrated with minimal performance degradation. In brief, our streaming decoder consists of a one-pass, real-time search engine relying on a limited-duration window sliding over time and a number of ad hoc acoustic and language model pruning techniques. Extensive empirical assessment is provided on truly streaming tasks derived from the well-known LibriSpeech and TED talks datasets, as well as from TV shows on a main Spanish broadcasting station., This work was supported in part by European Union's Horizon 2020 Research and Innovation Programme under Grant 761758 (X5gon), and 952215 (TAILOR) and Erasmus+ Education Program under Grant Agreement 20-226-093604-SCH, in part by MCIN/AEI/10.13039/501100011033 ERDF A way of making Europe under Grant RTI2018-094879-B-I00, and in part by Generalitat Valenciana's Research Project Classroom Activity Recognition under Grant PROMETEO/2019/111. Funding for open access charge: CRUE-Universitat Politecnica de Valencia. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Lei Xie.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Tesis de maestría (MasterThesis)

Machine Translation of Open Educational Resources: Evaluating Translation Quality and the Transition to Neural Machine Translation, Traducción automática de recursos educativos abiertos: Evaluación de la calidad de la traducción y la transición hacia la traducción automáticaneuronal

  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
[EN] Open Education has become a revolutionary approach towards the future of education, enabling worldwide free access to a huge volume of Open Educational Resources (OER). The rapid growth of OER and MOOCs has not gone unnoticed by governments and international organizations, as is demonstrated by the UNESCO-sponsored 2012 Paris OER Declaration and the 2017 Ljubljana OER Action Plan, the latter addressing ve strategic actions to support the mainstreaming of OER around UN Sustainable Development Goal 4 (SDG4) on Quality Education . In the EU, the European Commission's 2013 Opening up Education agenda recognized that the EU lacks a critical mass of good quality educational content in multiple languages. Although there is a clear need for multilingual services in Open Education, current OER platforms and MOOCs do not o er multilingual communication and seldom o er multilingual content. Based on this evidence, this master's thesis aims to foster Open Education with contributions on machine translation for the provision of multilingual access to OER and MOOC platforms. With the trans-disciplinary tools of automatic speech recognition (ASR), machine translation (MT), text-to-speech synthesis (TTS), and dialogue, multilingual access to OER will be possible for everyone regardless of their mother tongue or learning abilities. Firstly, we present work on the evaluation of MT, including intelligent interaction approaches to post-editing, as carried out in the framework of EU project transLectures. Evaluating MT is still an open question, and so exploring di erent ways to address this e ectively and applying them in real scenarios has a clear interest. The results obtained con rm that the intelligent interaction approach can make postediting automatic transcriptions and translations even more cost-e ective. Secondly, we present work on developing state-of-the-art neural machine translation (NMT) systems to transition from the previous phrase-based MT models. The new NMT paradigms provide signi cant improvements in MT quality, and so at this point it is key to move towards them, research on how to improve them, and, again, apply them in real scenarios to verify their usefulness. Our work resulted in a rstrank classi cation in an international evaluation campaign on MT, and we show the impact that these new NMT systems have in real OER scenarios. A structured comparison shows that our results are on par with the high quality of recent Google Translate results, and also shows that it is possible to go beyond Google Translate's quality through domain adaptation of MT systems., [ES] La Educación Abierta es cada vez más un modelo revolucionario de cara al futuro de la educación, permitiendo el acceso libre a nivel global a un enorme volumen de Recursos Educativos Abiertos (REA). El rápido crecimiento de los REA y de los cursos MOOC no ha pasado desapercibido para los gobiernos y las organizaciones internacionales, como se comprueba por la Declaración de París sobre los REA de 2012 y el Plan de Acción de Liubliana sobre los REA de 2017. Este último propone cinco acciones estratégicas para la implantación de los REA en torno al 4.o Objetivo de Desarrollo Sostenible de la ONU (ODS4) sobre Educación de Calidad . En la UE, la agenda Apertura de la educación de la Comisión Europea (2013) reconoce que la UE carece de una masa crítica de contenidos educativos de alta calidad en múltiples lenguas. A pesar de la clara necesidad de servicios multilingües para la Educación Abierta, las plataformas actuales de REA y MOOC no ofrecen comunicación multilingüe y raramente ofrecen contenidos multilingües. En base a esta evidencia, este Trabajo de Fin de Máster propone fomentar la Educación Abierta con contribuciones sobre la traducción automática (TA) para la provisión de acceso multilingüe a plataformas de REA y MOOC. Las herramientas interdisciplinares del reconocimiento automático del habla, la traducción automática, la síntesis de texto a voz y las tecnologías del diálogo harán posible el acceso multilingüe a los REA para todos, independientemente de la lengua materna o de las capacidades de aprendizaje de cada uno. En primer lugar, presentamos un trabajo sobre la evaluación de la TA, incluyendo métodos de interacción inteligente para la postedición, en el marco del proyecto europeo transLectures. La evaluación de la TA es aún una cuestión abierta, de forma que explorar diferentes vías para abordarla de forma efectiva y aplicarlas en escenarios reales tiene un claro interés. Mostraremos resultados que con rman que la interacción inteligente reduce el esfuerzo necesario para la postedición de transcripciones y traducciones automáticas. En segundo lugar, presentamos un esfuerzo de desarrollo de sistemas de traducción automática neuronal (TAN) punteros para superar los anteriores modelos de TA basada en frases. Los nuevos paradigmas de la TAN aportan mejoras signi cativas en la calidad de la TA, y es fundamental adoptarlos, investigar sobre cómo mejorarlos, y, de nuevo, aplicarlos en escenarios reales para veri car su utilidad. El sistema de TAN desarrollado se ha clasi cado entre los mejores en una competición internacional de TA, y aquí mostramos la repercusión que tienen estos nuevos sistemas de TAN en escenarios reales de REA. Nos hemos comparado y hemos comprobado que nuestros resultados compiten con la alta calidad de los resultados más recientes de Google Translate, y que es posible ir más allá de esta calidad con la adaptación al dominio de los sistemas de TA., [CA] L'Educació Oberta ha esdevingut un model revolucionari de cara al futur de l'educació, permetent l'accés lliure a nivell global a un enorme volum de Recursos Educatius Oberts (REO). El ràpid creixement dels REO i dels cursos MOOC no ha passat desapercebut per als governs i les organitzacions internacionals, com es comprova per la Declaració de París sobre els REO de 2012 i el Pla d'Acció de Ljubljana sobre els REO de 2017. Aquest últim proposa cinc accions estratègiques per a la implantació dels REO entorn del 4t Objectiu de Desenvolupament Sostenible de l'ONU (ODS4) sobre Educació de Qualitat . A l'UE, l'agenda Obertura de l'educació de la Comissió Europea (2013) reconeix que a l'UE manca una massa crítica de continguts educatius d'alta qualitat en múltiples llengües. Malgrat la clara necessitat de serveis multilingües per a l'Educació Oberta, les plataformes actuals de REO i MOOC no ofereixen comunicació multilingüe i rarament ofereixen continguts multilingües. Sobre la base d'aquesta evidència, aquest Treball de Fi de Màster es proposa fomentar l'Educació Oberta amb contribucions sobre la traducció automàtica (TA) per a la provisió d'accés multilingüe a plataformes de REO i MOOC. Les eines interdisciplinàries del reconeixement automàtic de la parla, la traducció automàtica, la síntesi de text a veu i les tecnologies del diàleg faran possible l'accés multilingüe als REO per a tots, independentment de la llengua materna o de les destreses d'aprenentatge. En primer lloc, presentem un treball sobre l'avaluació de la TA, incloent-hi mètodes d'interacció intel·ligent per a la postedició, en el marc del projecte europeu transLectures. L'avaluació de la TA és encara una qüestió oberta, de manera que explorar diferents vies per a abordar-la de forma efectiva i aplicar-les en escenaris reals té un interés clar. Els resultats obtinguts con rmen que la interacció intel·ligent redueix l'esforç necessari per a la postedició de transcripcions i traduccions automàtiques. En segon lloc, presentem el desenvolupament de sistemes de traducció automàtica neuronal (TAN) punters per a superar els anteriors models de TA basada en frases. Els nous paradigmes de la TAN aporten millores signi catives en la qualitat de la TA, i és fonamental abraçar-los, investigar sobre com millorar-los, i, novament, aplicarlos en escenaris reals per a veri car-ne la utilitat. El sistema de TAN desenvolupat s'ha classi cat entre els millors en una competició internacional de TA, i ací mostrem la repercussió que tenen aquests nous sistemes de TAN en escenaris reals de REO. Ens hem comparat i hem comprovat que els nostres resultats competeixen amb l'alta qualitat dels resultats més recents de Google Translate, i que és possible anar més enllà d'aquesta qualitat amb l'adaptació al domini dels sistemes de TA., The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 287755 (transLectures), Competitiveness and Innovation Framework Programme (CIP) under grant agreement no. 621030 (EMMA) and Horizon 2020 research and innovation programme under grant agreement no. 761758 (X5gon); from the Government of Spain's research projects iTrans2 (ref. TIN2009-14511, MICINN/FEDER EU) and MORE (ref. TIN2015-68326-R, MINECO/FEDER EU); and from the Universitat Politècnica de València's PAID-01-17 R&D support programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Speaker-Adapted Confidence Measures for ASR using Deep Bidirectional Recurrent Neural Networks

  • Del Agua Teba, Miguel Angel
  • Giménez Pastor, Adrián
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Juan, Alfons|||0000-0002-9984-4072
[EN] In the last years, Deep Bidirectional Recurrent Neural Networks (DBRNN) and DBRNN with Long Short-Term Memory cells (DBLSTM) have outperformed the most accurate classifiers for confidence estimation in automatic speech recognition. At the same time, we have recently shown that speaker adaptation of confidence measures using DBLSTM yields significant improvements over non-adapted confidence measures. In accordance with these two recent contributions to the state of the art in confidence estimation, this paper presents a comprehensive study of speaker-adapted confidence measures using DBRNN and DBLSTM models. Firstly, we present new empirical evidences of the superiority of RNN-based confidence classifiers evaluated over a large speech corpus consisting of the English LibriSpeech and the Spanish poliMedia tasks. Secondly, we show new results on speaker-adapted confidence measures considering a multi-task framework in which RNN-based confidence classifiers trained with LibriSpeech are adapted to speakers of the TED-LIUM corpus. These experiments confirm that speaker-adapted confidence measures outperform their non-adapted counterparts. Lastly, we describe an unsupervised adaptation method of the acoustic DBLSTM model based on confidence measures which results in better automatic speech recognition performance., This work was supported in part by the European Union's Horizon 2020 research and innovation programme under Grant 761758 (X5gon), in part by the Seventh Framework Programme (FP7/2007-2013) under Grant 287755 (transLectures), in part by the ICT Policy Support Programme (ICT PSP/2007-2013) as part of the Competitiveness and Innovation Framework Programme under Grant 621030 (EMMA), and in part by the Spanish Government's TIN2015-68326-R (MINECO/FEDER) research project MORE.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Tesis doctoral (DoctoralThesis)

Automatic Speech Recognition and Machine Translation with Deep Neural Networks for Open Educational Resources, Parliamentary Contents and Broadcast Media

  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
[ES] En la última década, el reconocimiento automático del habla (RAH) y la traducción automática (TA) han mejorado enormemente mediante el uso de modelos de redes neuronales profundas (RNP) en constante evolución. Si a principios de los 2010 los sistemas de RAH y TA previos a las RNP llegaron a afrontar con éxito algunas aplicaciones reales como la transcripción y traducción de vídeos docentes pregrabados, ahora en los 2020 son abordables aplicaciones que suponen un reto mucho mayor, como la subtitulación de retransmisiones audiovisuales en directo. En este mismo período, se están invirtiendo cada vez mayores esfuerzos en la accesibilidad a los medios audiovisuales para todos, incluidas las personas sordas. El RAH y la TA, en su estado actual, son grandes herramientas para aumentar la disponibilidad de medidas de accesibilidad como subtítulos, transcripciones y traducciones, y también para proporcionar acceso multilingüe a todo tipo de contenidos. En esta tesis doctoral presentamos resultados de investigación sobre RAH y TA basadas en RNP en tres campos muy activos: los recursos educativos abiertos, los contenidos parlamentarios y los medios audiovisuales. En el área de los recursos educativos abiertos (REA), presentamos primeramente trabajos sobre la evaluación y postedición de RAH y TA con métodos de interacción inteligente, en el marco del proyecto de investigación europeo "transLectures: Transcripción y Traducción de Vídeos Docentes". Los resultados obtenidos confirman que la interacción inteligente puede reducir aún más el esfuerzo de postedición de transcripciones y traducciones automáticas. Seguidamente, en el contexto del posterior proyecto europeo X5gon, presentamos una investigación sobre el desarrollo de sistemas de TA neuronal basados en RNP, y sobre sacar el máximo partido de corpus de TA masivos mediante filtrado automático de datos. Este trabajo dio como resultado sistemas de TA neuronal clasificados entre los mejores en una competición internacional de TA, y mostramos cómo estos nuevos sistemas mejoraron la calidad de los subtítulos multilingües en casos reales de REA. En el ámbito también en crecimiento de las tecnologías del lenguaje para contenidos parlamentarios, describimos una investigación sobre técnicas de filtrado de datos de habla para el RAH en tiempo real en el contexto de debates del Parlamento Europeo. Esta investigación permitió la publicación de Europarl-ASR, un nuevo y amplio corpus de habla para entrenamiento y evaluación de sistemas de RAH en continuo, así como para la evaluación comparativa de técnicas de filtrado de datos de habla. Finalmente, presentamos un trabajo en un ámbito en la vanguardia tecnológica del RAH y de la TA: la subtitulación de retransmisiones audiovisuales en directo, en el marco del Convenio de colaboración I+D+i 2020-2023 entre la radiotelevisión pública valenciana À Punt y la Universitat Politècnica de València para la subtitulación asistida por ordenador de contenidos audiovisuales en tiempo real. Esta investigación ha resultado en la implantación de sistemas de RAH en tiempo real, de alta precisión y baja latencia, para una lengua no mayoritaria en el mundo (el catalán) y una de las lenguas más habladas del mundo (el castellano) en un medio audiovisual real., [CA] En l'última dècada, el reconeixement automàtic de la parla (RAP) i la traducció automàtica (TA) han millorat enormement mitjançant l'ús de models de xarxes neuronals profundes (XNP) en constant evolució. Si a principis dels 2010 els sistemes de RAP i TA previs a les XNP van arribar a afrontar amb èxit algunes aplicacions reals com la transcripció i traducció de vídeos docents pregravats, ara en els 2020 són abordables aplicacions que suposen un repte molt major, com la subtitulació de retransmissions audiovisuals en directe. En aquest mateix període, s'estan invertint cada vegada majors esforços en l'accessibilitat als mitjans audiovisuals per a tots, incloses les persones sordes. El RAP i la TA, en el seu estat actual, són grans eines per a incrementar la disponibilitat de mesures d'accessibilitat com subtítols, transcripcions i traduccions, també com una manera de proporcionar accés multilingüe a tota classe de continguts. En aquesta tesi doctoral presentem resultats d'investigació sobre RAP i TA basades en XNP en tres camps molt actius: els recursos educatius oberts, els continguts parlamentaris i els mitjans audiovisuals. En l'àrea dels recursos educatius oberts (REO), presentem primerament treballs sobre l'avaluació i postedició de RAP i TA amb mètodes d'interacció intel·ligent, en el marc del projecte d'investigació europeu "transLectures: Transcripció i traducció de vídeos docents". Els resultats obtinguts confirmen que la interacció intel·ligent pot reduir encara més l'esforç de postedició de transcripcions i traduccions automàtiques. Seguidament, en el context del posterior projecte europeu X5gon, presentem una investigació sobre el desenvolupament de sistemes de TA neuronal basats en XNP, i sobre traure el màxim partit de corpus de TA massius mitjançant filtratge automàtic de dades. Aquest treball va donar com a resultat sistemes de TA neuronal classificats entre els millors en una competició internacional de TA, i mostrem com aquests nous sistemes milloren la qualitat dels subtítols multilingües en casos reals de REO. En l'àmbit també en creixement de les tecnologies del llenguatge per a continguts parlamentaris, descrivim una investigació sobre tècniques de filtratge de dades de parla per al RAP en temps real en el context de debats del Parlament Europeu. Aquesta investigació va permetre la publicació d'Europarl-ASR, un corpus de parla nou i ampli per a l'entrenament i l'avaluació de sistemes de RAP en continu, així com per a l'avaluació comparativa de tècniques de filtratge de dades de parla. Finalment, presentem un treball en un àmbit en l'avantguarda tecnològica del RAP i de la TA: la subtitulació de retransmissions audiovisuals en directe, en el context del Conveni de col·laboració R+D+i 2020-2023 entre la radiotelevisió pública valenciana À Punt i la Universitat Politècnica de València per a la subtitulació assistida per ordinador de continguts audiovisuals en temps real. Aquesta investigació ha donat com a resultat la implantació de sistemes de RAP en temps real, amb alta precisió i baixa latència, per a una llengua no majoritària en el món (el català) i una de les llengües més parlades del món (el castellà) en un mitjà audiovisual real., [EN] In the last decade, automatic speech recognition (ASR) and machine translation (MT) have improved enormously through the use of constantly evolving deep neural network (DNN) models. If at the beginning of the 2010s the then pre-DNN ASR and MT systems were ready to tackle with success some real-life applications such as offline video lecture transcription and translation, now in the 2020s much more challenging applications are within grasp, such as live broadcast media subtitling. At the same time in this period, media accessibility for everyone, including deaf and hard-of-hearing people, is being given more and more importance. ASR and MT, in their current state, are powerful tools to increase the coverage of accessibility measures such as subtitles, transcriptions and translations, also as a way of providing multilingual access to all types of content. In this PhD thesis, we present research results on automatic speech recognition and machine translation based on deep neural networks in three very active domains: open educational resources, parliamentary contents and broadcast media. Regarding open educational resources (OER), we first present work on the evaluation and post-editing of ASR and MT with intelligent interaction approaches, as carried out in the framework of EU project transLectures: Transcription and Translation of Video Lectures. The results obtained confirm that the intelligent interaction approach can make post-editing automatic transcriptions and translations even more cost-effective. Then, in the context of subsequent EU project X5gon, we present research on developing DNN-based neural MT systems, and making the most of larger MT corpora through automatic data filtering. This work resulted in a first-rank classification in an international evaluation campaign on MT, and we show how these new NMT systems improved the quality of multilingual subtitles in real OER scenarios. In the also growing domain of language technologies for parliamentary contents, we describe research on speech data curation techniques for streaming ASR in the context of European Parliament debates. This research resulted in the release of Europarl-ASR, a new, large speech corpus for streaming ASR system training and evaluation, as well as for the benchmarking of speech data curation techniques. Finally, we present work in a domain on the edge of the state of the art for ASR and MT: the live subtitling of broadcast media, in the context of the 2020-2023 R&D collaboration agreement between the Valencian public broadcaster À Punt and the Universitat Politècnica de València for real-time computer assisted subtitling of media contents. This research has resulted in the deployment of high-quality, low-latency, real-time streaming ASR systems for a less-spoken language (Catalan) and a widely spoken language (Spanish) in a real broadcast use case., The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 287755 (transLectures), Competitiveness and Innovation Framework Programme (CIP) under grant agreement no. 621030 (EMMA), Horizon 2020 research and innovation programme under grant agreements no. 761758 (X5gon) and no. 952215 (TAILOR), and EU4Health Programme 2021–2027 as part of Europe’s Beating Cancer Plan under grant agreements no. 101056995 (INTERACT-EUROPE) and no. 101129375 (INTERACT-EUROPE 100); from the Government of Spain’s research projects iTrans2 (ref. TIN2009-14511, MICINN/ERDF EU), MORE (ref. TIN2015-68326-R,MINECO/ERDF EU), Multisub (ref. RTI2018-094879-B-I00, MCIN/AEI/10.13039/501100011033 ERDF “A way of making Europe”), and XLinDub (ref. PID2021-122443OB-I00, MCIN/AEI/10.13039/501100011033 ERDF “A way of making Europe”); from the Generalitat Valenciana’s “R&D collaboration agreement between the Corporació Valenciana de Mitjans de Comunicació (À Punt Mèdia) and the Universitat Politècnica de València (UPV) for real-time computer assisted subtitling of audiovisual contents based on artificial intelligence”, and research project Classroom Activity Recognition (PROMETEO/2019/111); and from the Universitat Politècnica de València’s PAID-01-17 R&D support programme. This work uses data from the RTVE 2018 and 2020 Databases. This set of data has been provided by RTVE Corporation to help develop Spanish-language speech technologies.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

MLLP-VRAIN Spanish ASR Systems for the Albayzin-RTVE 2020 Speech-To-Text Challenge

  • Jorge-Cano, Javier|||0000-0002-9279-6768
  • Giménez Pastor, Adrián
  • Garcés Díaz-Munío, Gonçal|||0000-0002-2594-5858
  • Silvestre Cerdà, Joan Albert|||0000-0003-2291-8296
  • Civera Saiz, Jorge|||0000-0002-0963-0143
  • Sanchis Navarro, José Alberto|||0000-0002-2943-0990
  • Juan, Alfons|||0000-0002-9984-4072
  • Baquero-Arnal, Pau
  • Iranzo-Sánchez, Javier
  • Pérez-González de Martos, Alejandro Manuel
[EN] This paper describes the automatic speech recognition (ASR) systems built by the MLLP-VRAIN research group of Universitat Politecnica de València for the Albayzin-RTVE 2020 Speech-to-Text Challenge. The primary system (p-streaming_1500ms_nlt) was a hybrid BLSTM-HMM ASR system using streaming one-pass decoding with a context window of 1.5 seconds and a linear combination of an n-gram, a LSTM, and a Transformer language model (LM). The acoustic model was trained on nearly 4,000 hours of speech data from different sources, using the MLLP's transLectures-UPV toolkit (TLK) and TensorFlow; whilst LMs were trained using SRILM (n-gram), CUED-RNNLM (LSTM) and Fairseq (Transformer), with up to 102G tokens. This system achieved 11.6% and 16.0% WER on the test-2018 and test-2020 sets, respectively. As it is streaming-enabled, it could be put into production environments for automatic captioning of live media streams, with a theoretical delay of 1.5 seconds. Along with the primary system, we also submitted three contrastive systems. From these, we highlight the system c2-streaming_600ms_t that, following the same configuration of the primary one, but using a smaller context window of 0.6 seconds and a Transformer LM, scored 12.3% and 16.9% WER points respectively on the same test sets, with a measured empirical latency of 0.81+-0.09 seconds (mean+-stdev). This is, we obtained state-of-the-art latencies for high-quality automatic live captioning with a small WER degradation of 6% relative., [CA] En aquest article, es descriuen els sistemes de reconeixement automàtic de la parla (RAP) creats pel grup d'investigació MLLP-VRAIN de la Universitat Politecnica de València per a la competició Albayzin-RTVE 2020 Speech-to-Text Challenge. El sistema primari (p-streaming_1500ms_nlt) és un sistema de RAP híbrid BLSTM-HMM amb descodificació en temps real en una passada amb una finestra de context d'1,5 segons i una combinació lineal de models de llenguatge (ML) d'n-grames, LSTM i Transformer. El model acústic s'ha entrenat amb vora 4000 hores de parla transcrita de diferents fonts, usant el transLectures-UPV toolkit (TLK) del grup MLLP i TensorFlow; mentre que els ML s'han entrenat amb SRILM (n-grames), CUED-RNNLM (LSTM) i Fairseq (Transformer), amb 102G paraules (tokens). Aquest sistema ha obtingut 11,6 % i 16,0 % de WER en els conjunts test-2018 i test-2020, respectivament. És un sistema amb capacitat de temps real, que pot desplegar-se en producció per a subtitulació automàtica de fluxos audiovisuals en directe, amb un retard teòric d'1,5 segons. A banda del sistema primari, s'han presentat tres sistemes contrastius. D'aquests, destaquem el sistema c2-streaming_600ms_t que, amb la mateixa configuració que el sistema primari, però amb una finestra de context més reduïda de 0,6 segons i un ML Transformer, ha obtingut 12,3 % i 16,9 % de WER, respectivament, sobre els mateixos conjunts, amb una latència empírica mesurada de 0,81+-0,09 segons (mitjana+-desv). És a dir, s'han obtingut latències punteres per a subtitulació automàtica en directe d'alta qualitat amb una degradació del WER petita, del 6 % relatiu., The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement no. 761758 (X5Gon); the Government of Spain¿s research project Multisub (ref. RTI2018-094879-B-I00, MCIU/AEI/FEDER,EU) and FPU scholarships FPU14/03981 and FPU18/04135; and the Generalitat Valenciana¿s research project Classroom Activity Recognition (ref. PROMETEO/2019/111) and predoctoral research scholarship ACIF/2017/055
Dipòsit Digital de la UB
Publicación Artículo científico (JournalArticle)

A Conceptual Probabilistic Framework for Annotation Aggregation of Citizen Science Data

  • Cerquides Bueno, Jesús
  • Mülâyim, Mehmet Oguz
  • Hernández-González, Jerónimo
  • Shankar, Amudha Ravi
  • Fernández Márquez, Jose Luis
Over the last decade, hundreds of thousands of volunteers have contributed to science by collecting or analyzing data. This public participation in science, also known as citizen science, has contributed to significant discoveries and led to publications in major scientific journals. However, little attention has been paid to data quality issues. In this work we argue that being able to determine the accuracy of data obtained by crowdsourcing is a fundamental question and we point out that, for many real-life scenarios, mathematical tools and processes for the evaluation of data quality are missing. We propose a probabilistic methodology for the evaluation of the accuracy of labeling data obtained by crowdsourcing in citizen science. The methodology builds on an abstract probabilistic graphical model formalism, which is shown to generalize some already existing label aggregation models. We show how to make practical use of the methodology through a comparison of data obtained from different citizen science communities analyzing the earthquake that took place in Albania in 2019.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A Conceptual Probabilistic Framework for Annotation Aggregation of Citizen Science Data

  • Cerquides Bueno, Jesús
  • Mülâyim, Mehmet Oguz
  • Hernández-González, Jerónimo
  • Shankar, Amudha Ravi
  • Fernández Márquez, Jose Luis
Over the last decade, hundreds of thousands of volunteers have contributed to science by collecting or analyzing data. This public participation in science, also known as citizen science, has contributed to significant discoveries and led to publications in major scientific journals. However, little attention has been paid to data quality issues. In this work we argue that being able to determine the accuracy of data obtained by crowdsourcing is a fundamental question and we point out that, for many real-life scenarios, mathematical tools and processes for the evaluation of data quality are missing. We propose a probabilistic methodology for the evaluation of the accuracy of labeling data obtained by crowdsourcing in citizen science. The methodology builds on an abstract probabilistic graphical model formalism, which is shown to generalize some already existing label aggregation models. We show how to make practical use of the methodology through a comparison of data obtained from different citizen science communities analyzing the earthquake that took place in Albania in 2019.