Skip to main content

ARQUITECTURAS AVANZADAS DE APRENDIZAJE PROFUNDO APLICADAS AL PROCESADO DE VOZ, AUDIO Y LENGUAJE

PID2019-107579RB-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2019
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITECNICA DE CATALUNYA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 54
Encontrada(s) 1 página(s)

UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Multiformer: a head-configurable transformer-based model for direct speech translation

  • Sant Muniesa, Gerard
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Alastruey Lasheras, Belen
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as long sequence lengths and redundancy between adjacent tokens. Therefore, we believe that regular self-attention mechanism might not be well suited for it. Different approaches have been proposed to overcome these problems, such as the use of efficient attention mechanisms. However, the use of these methods usually comes with a cost, which is a performance reduction caused by information loss. In this study, we present the Multiformer, a Transformer-based model which allows the use of different attention mechanisms on each head. By doing this, the model is able to bias the self-attention towards the extraction of more diverse token interactions, and the information loss is reduced. Finally, we perform an analysis of the head contributions, and we observe that those architectures where all heads relevance is uniformly distributed obtain better results. Our results show that mixing attention patterns along the different heads and layers outperforms our baseline by up to 0.7 BLEU., This work was partially funded by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Syntax-driven iterative expansion language models for controllable text generation

  • Casas Manzanares, Noé
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations., This work is partially supported by Lucy Software / United Language Group (ULG) and the Catalan Agency for Management of University and Research Grants (AGAUR) through an Industrial Ph.D. Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund through the postdoctoral senior grant Ramón y Cajal and by the Agencia Estatal de Investigación through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Sign language translation from instructional videos

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Cardoso Duarte, Amanda
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Giró Nieto, Xavier|||0000-0002-9935-5332
The advances in automatic sign language translation (SLT) to spoken languages have been mostly benchmarked with datasets of limited size and restricted domains. Our work advances the state of the art by providing the first baseline results on How2Sign, a large and broad dataset. We train a Transformer over I3D video features, using the reduced BLEU as a reference metric for validation, instead of the widely used BLEU score. We report a result of 8.03 on the BLEU score, and publish the first open-source implementation of its kind to promote further advances., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 / AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21 and by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

SegAugment: maximizing the utility of speech translation data with segmentation-based augmentations

  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
  • Ruiz Costa-jussà, Marta
End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment obtains state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time., Work at UPC was supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Analysis of acoustic information in end-to-end spoken language translation

  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
End-to-End Transformer-based models are the most popular approach for Spoken Language Translation (SLT). While obtaining state-of-the-art results, we are still far from understanding how these models extract acoustic information from the data and how they are transformed into semantic representations. In this paper, we seek to provide a better understanding of the flow of acoustic information along speech-to-text translation models. By means of the Speaker Classification and Spectrogram Reconstruction tasks, this study (i) interprets the main role of the encoder with respect to the acoustic features, (ii) highlights the importance of the acoustic information throughout the model and its transfer between encoder and decoder, and (iii) reveals the significant effect of downsampling convolutional layers for learning acoustic features. (iv) Finally, we also observe the existence of a strong correlation between the semantic domain and the speakers’ labels in MuST-C., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RB-I00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Speech translation with foundation models and optimal transport: UPC at IWSLT23

  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Fonollosa, José A. R.
  • Ruiz Costa-jussà, Marta
This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023., The work done by Ioannis Tsiamas and Gerard I. Gállego was supported by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention multimodal system for Odyssey 2024 Speech Emotion Recognition Challenge

  • Costa, Federico|||0000-0002-1389-3595
  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
As computer-based applications are becoming more integrated into our daily lives, the importance of Speech Emotion Recognition (SER) has increased significantly. Promoting research with innovative approaches in SER, the Odyssey 2024 Speech Emotion Recognition Challenge was organized as part of the Odyssey 2024 Speaker and Language Recognition Workshop. In this paper we describe the Double Multi-Head Attention Multimodal System developed for this challenge. Pre-trained self-supervised models were used to extract informative acoustic and text features. An early fusion strategy was adopted, where a Multi-Head Attention layer transforms these mixed features into complementary contextualized representations. A second attention mechanism is then applied to pool these representations into an utterance-level vector. Our proposed system achieved the third position in the categorical task ranking with a 34.41% Macro-F1 score, where 31 teams participated in total., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The first author is supported by a FI grant from the Catalan government., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

The UPC RDF-to-Text System at WebNLG Challenge 2020

  • Bergés Lladó, David
  • Cantenys Sabà, Roser
  • Creus Castanyer, Roger
  • Domingo Roig, Oriol
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
This work describes the end-to-end system architecture presented at WebNLG Challenge 2020. The system follows the traditional Machine Translation (MT) pipeline, based on the Transformer model, applied in most text-to-text problems. Our solution is enriched by means of a Back Translation step over the original corpus. Thus, the system directly relies on lexicalise format since the synthetic data limits the use of delexicalisation., We want to thank anonymous reviewers for their comments on the paper. This work was supported by the project ADAVOICE, PID2019-107579RBI00 / AEI / 10.13039/501100011033.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Automatic adaption of the step size in gradient descent training

  • Nogueiras Rodríguez, Albino|||0000-0002-3159-1718
  • Nogueiras Marco, Ignasi
Selecting an appropriate step size is critical in Gradient Descent algorithms used to train Neural Networks for Deep Learning tasks. A small value of the step size leads to slow convergence, while a large value can lead to instability. In this paper, we present ACME, a gradient descent algorithm that automatically updates the step size to its optimum value. When combined with Adaptive Moment Estimation (ADAM) and/or Adaptive Sharpness Aware Minimization (ASAM), ACME achieves performances similar to them, but without the burden of carefully adjusting the learning rate or designing intricate learning rate scheduling policies. The code of ACME, ready to be used by itself or in combination with ADAM and/or ASAM, is available at https://github.com/ignasi00/acme.git., The contribution of Albino Nogueiras-Rodríguez to this paper was supported by the Spanish Government project PID2019-107579RB-I00 / AEI / 10.13039/501100011033. Ignasi Nogueiras-Marco acknowledges funding from “la Caixa” Foundation (ID 100,010,434) under agreement HR-18-0036, and from the Spanish Government through Grant PID2021-122893NB-C21, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

BSC-UPC at EmoSPeech-IberLEF2024: attention pooling for emotion recognition

  • Casals i Salvador, Marc
  • Costa, Federico|||0000-0002-1389-3595
  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
The domain of speech emotion recognition (SER) has persistently been a frontier within the landscape of machine learning. It is an active field that has been revolutionized in the last few decades and whose implementations are remarkable in multiple applications that could affect daily life. Consequently, the Iberian Languages Evaluation Forum (IberLEF) of 2024 held a competitive challenge to leverage the SER results with a Spanish corpus. This paper presents the approach followed with the goal of participating in this competition. The main architecture consists of different pre-trained speech and text models to extract features from both modalities, utilizing an attention pooling mechanism. The proposed system has achieved the first position in the challenge with an 86.69% in Macro F1-Score., This work has been promoted and financed by the Government of Catalonia through the Aina project, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Evaluating gender bias in speech translation

  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Basta, Christine Raouf Saad
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
The scientific community is increasingly aware of the necessity to embrace pluralism and consistently represent major and minor social groups. Currently, there are no standard evaluation techniques for different types of biases. Accordingly, there is an urgent need to provide evaluation sets and protocols to measure existing biases in our automatic systems. Evaluating the biases should be an essential step towards mitigating them in the systems. This paper introduces WinoST, a new freely available challenge set for evaluating gender bias in speech translation. WinoST is the speech version of WinoMT which is a MT challenge set and both follow an evaluation protocol to measure gender accuracy. Using a state-of-the-art end-to-end speech translation system, we report the gender bias evaluation on four language pairs and we show that gender accuracy in speech translation is more than 23% lower than in MT., This work is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657), the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship, Universitat Politècnica de Catalunya with the collaboration of Banco de Santander and the Spanish Ministerio de Ciencia e Innovación through the project PID2019- 107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

On the locality of attention in direct speech translation

  • Alastruey Lasheras, Belén
  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
Transformers have achieved state-of-the-art results across multiple NLP tasks. However, the self-attention mechanism complexity scales quadratically with the sequence length, creating an obstacle for tasks involving long sequences, like in the speech domain. In this paper, we discuss the usefulness of self-attention for Direct Speech Translation. First, we analyze the layer-wise token contributions in the self-attention of the encoder, unveiling local diagonal patterns. To prove that some attention weights are avoidable, we propose to substitute the standard self-attention with a local efficient one, setting the amount of context used based on the results of the analysis. With this approach, our model matches the baseline performance, and improves the efficiency by skipping the computation of those weights that standard attention discards., This work was partially funded by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Measuring the mixing of contextual information in the transformer

  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block --multi-head attention, residual connection, and layer normalization-- and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods., Javier Ferrando and Gerard I. Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

End-to-end speech translation with pre-trained models and adapters: UPC at IWSLT 2021

  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuSTC test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation., This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention for speaker verification

  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Safari, Pooyan
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
Most state-of-the-art Deep Learning systems for text-independent speaker verification are based on speaker embedding extractors. These architectures are commonly composed of a feature extractor front-end together with a pooling layer to encode variable-length utterances into fixed-length speaker vectors. In this paper we present Double Multi-Head Attention (MHA) pooling, which extends our previous approach based on Self MHA. An additional self attention layer is added to the pooling layer that summarizes the context vectors produced by MHA into a unique speaker representation. This method enhances the pooling mechanism by giving weights to the information captured for each head and it results in creating more discriminative speaker embeddings. We have evaluated our approach with the VoxCeleb2 dataset. Our results show 6.09% and 5.23% relative improvement in terms of EER compared to Self Attention pooling and Self MHA, respectively. According to the obtained results, Double MHA has shown to be an excellent approach to efficiently select the most relevant features captured by the CNN-based front-ends from the speech signal., This work was supported by the Spanish project PID2019-107579RBI00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

High frequent in-domain word segmentation and forward translation for the WMT21 Biomedical task

  • Rafieian, Bardia
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
This paper reports the optimization of using the out-of-domain data in the Biomedical translation task. We firstly optimized our parallel training dataset using the BabelNet in-domain terminology words. Afterward, to increase the training set, we studied the effects of the out-of-domain data on biomedical translation tasks, and we created a mixture of in-domain and out-of-domain training sets and added more in-domain data using forward translation in the English-Spanish task. Finally, with a simple bpe optimization method, we increased the number of in-domain subwords in our mixed training set and trained the Transformer model on the generated data. Results show improvements using our proposed method. © 2021 Association for Computational Linguistics, This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI 256 / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Tackling low-resourced sign language translation: UPC at WMT-SLT 22

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Giró Nieto, Xavier|||0000-0002-9935-5332
  • Torres Viñals, Jordi|||0000-0003-1963-7418
This paper describes the system developed at the Universitat Politècnica de Catalunya for the Workshop on Machine Translation 2022 Sign Language Translation Task, in particular, for the sign-to-text direction. We use a Transformer model implemented with the Fairseq modeling toolkit. We have experimented with the vocabulary size, data augmentation techniques and pretraining the model with the PHOENIX-14T dataset. Our system obtains 0.50 BLEU score for the test set, improving the organizers’ baseline by 0.38 BLEU. We remark the poor results for both the baseline and our system, and thus, the unreliability of our findings., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 /AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

TALP-UPC at ProbSum 2023: fine-tuning and data augmentation strategies for NER

  • Torrero Salas, Neil
  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
This paper describes the submission of the TALP-UPC team to the Problem List Summarization task from the BioNLP 2023 workshop. This task consists of automatically extracting a list of health issues from the e-health medical record of a given patient. Our submission combines additional steps of data annotationwith finetuning of BERT pre-trained language models. Our experiments focus on the impact of finetuning on different datasets as well as the addition of data augmentation techniques to delay overfitting., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RBI00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Mass-editing memory with attention in transformers: a cross-lingual exploration of knowledge

  • Tamayo Mela, Daniel
  • Gonzalez Aguirre, Aitor
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
  • Villegas, Marta
Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT., This work has been promoted and financed by the Government of Catalonia through the Aina project, by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia Funded by EU– NextGenerationEU within the framework of the project ILENIA with reference 2022/TL22/00215337, 2022/TL22/00215336, 2022/TL22/00215335, 2022/TL22/00215334, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Self-supervised deep learning approaches to speaker recognition: A Ph.D. Thesis overview

  • Khan, Umair
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
Recent advances in Deep Learning (DL) for speaker recognition have improved the performance but are constrained to the need of labels for the background data, which is difficult in prac- tice. In i-vector based speaker recognition, cosine (unsuper- vised) and PLDA (supervised) are the basic scoring techniques, with a big performance gap between the two. In this thesis we tried to fill this gap without using speaker labels in several ways. We applied Restricted Boltzmann Machine (RBM) vectors for the tasks of speaker clustering and tracking in TV broadcast shows. The experiments on AGORA database show that us- ing this approach we gain a relative improvement of 12% and 11% for speaker clustering and tracking tasks, respectively. We also applied DL techniques in order to increase the discrimina- tive power of i-vectors in speaker verification task, for which we have proposed the use of autoencoder in several ways, i.e., (1) as a pre-training for a Deep Neural Network (DNN), (2) as a near- est neighbor autoencoder for i-vectors, (3) as an average pooled nearest neighbor autoencoder. The experiments on VoxCeleb database show that we gain a relative improvement of 21%, 42% and 53%, using the three systems respectively. Finally we also proposed a self-supervised end-to-end speaker verification system. The architecture is based on a Convolutional Neural Network (CNN), trained as a siamese network with multiple branches. From the results we can see that our system shows comparable performance to a supervised baseline, This work was supported by the project PID2019-107579RBI00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Towards opening the black box of neural machine translation: source and target interpretations of the transformer

  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Alastruey Lasheras, Belen
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Ruiz Costa-jussà, Marta
In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step). However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions. Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions. In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts. Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models. We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour., Javier Ferrando and Gerard I.Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Explaining how transformers use context to build predictions

  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Ruiz Costa-jussà, Marta
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions., Javier Ferrando, Gerard I.Gállego and Ioannis Tsiamas are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Toxicity in multilingual machine translation at scale

  • Ruiz Costa-jussà, Marta
  • Smith, Eric
  • Ropers, Christophe
  • Licht, Daniel
  • Maillard, Jean
  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users. In this paper we focus on one type of critical error: added toxicity. We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages. An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%. The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability. We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity. We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity. Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied. Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations. WARNING: this paper contains examples of toxicity that may be offensive or upsetting in nature., Javier Ferrando and Carlos Escolano are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033 and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

On the use of audio to improve dialogue policies

  • Roncel Diaz, Daniel
  • Costa, Federico|||0000-0002-1389-3595
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
With the significant progress of speech technologies, spoken goal-oriented dialogue systems are becoming increasingly popular. One of the main modules of a dialogue system is typically the dialogue policy, which is responsible for determining system actions. This component usually relies only on audio transcriptions, being strongly dependent on their quality and ignoring very important extralinguistic information embedded in the user’s speech. In this paper, we propose new architectures to add audio information by combining speech and text embeddings using a Double Multi-Head Attention component. Our experiments show that audio embedding-aware dialogue policies outperform text-based ones, particularly in noisy transcription scenarios, and that how text and audio embeddings are combined is crucial to improve performance. We obtained a 9.8% relative improvement in the User Request Score compared to an only-text-based dialogue system on the DSTC2 dataset., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The second author is supported by a FI grant from the Catalan government., Award-winning
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Multilingual neural machine translation: case-study for Catalan, Spanish and Portuguese romance languages

  • Vergés Boncompte, Pere
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
In this paper, we describe the TALP-UPC participation in the WMT Similar Language Translation task between Catalan, Spanish, and Portuguese, all of them, Romance languages. We made use of different techniques to improve the translation between these languages. The multilingual shared encoder/decoder has been used for all of them. Additionally, we applied back-translation to take advantage of the monolingual data. Finally, we have applied fine-tuning to improve the in-domain data. Each of these techniques brings improvements over the previous one. In the official evaluation, our system was ranked 1st in the Portuguese-to-Spanish direction, 2nd in the opposite direction, and 3rd in the Catalan-Spanish pair., This work is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

The TALP-UPC system description for WMT20 news translation task: multilingual adaptation for low resource MT

  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
In this article, we describe the TALP-UPC participation in the WMT20 news translation shared task for Tamil-English. Given the low amount of parallel training data, we resort to adapt the task to a multilingual system to benefit from the positive transfer from high resource languages. We use iterative backtranslation to fine-tune the system and benefit from the monolingual data available. In order to measure the effectivity of such methods, we compare our results to a bilingual baseline system., This work is supported in part by the Google Faculty Research Award 2019, the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019- 107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Findings of the first shared task on lifelong learning machine translation

  • Barrault, Loïc
  • Biesialska, Magdalena Marta|||0000-0001-7890-3523
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Bougares, Fethi
  • Galibert, Olivier
A lifelong learning system can adapt to new data without forgetting previously acquired knowledge. In this paper, we introduce the first benchmark for lifelong learning machine translation. For this purpose, we provide training, lifelong and test data sets for two language pairs: English-German and English-French. Additionally, we report the results of our baseline systems, which we make available to the public. The goal of this shared task is to encourage research on the emerging topic of lifelong learning machine translation., This work is is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

TALP-UPC at ProbSum 2023: fine-tuning and data augmentation strategies for NER

  • Torrero Salas, Neil
  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos
This paper describes the submission of the TALP-UPC team to the Problem List Summarization task from the BioNLP 2023 workshop. This task consists of automatically extracting a list of health issues from the e-health medical record of a given patient. Our submission combines additional steps of data annotationwith finetuning of BERT pre-trained language models. Our experiments focus on the impact of finetuning on different datasets as well as the addition of data augmentation techniques to delay overfitting., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RBI00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Sign language translation from instructional videos

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion
  • Cardoso Duarte, Amanda
  • Torres Viñals, Jordi
  • Giró Nieto, Xavier
The advances in automatic sign language translation (SLT) to spoken languages have been mostly benchmarked with datasets of limited size and restricted domains. Our work advances the state of the art by providing the first baseline results on How2Sign, a large and broad dataset. We train a Transformer over I3D video features, using the reduced BLEU as a reference metric for validation, instead of the widely used BLEU score. We report a result of 8.03 on the BLEU score, and publish the first open-source implementation of its kind to promote further advances., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 / AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21 and by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Multiformer: a head-configurable transformer-based model for direct speech translation

  • Sant Muniesa, Gerard
  • Gallego Olsina, Gerard Ion
  • Alastruey Lasheras, Belen
  • Ruiz Costa-Jussà, Marta
Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as long sequence lengths and redundancy between adjacent tokens. Therefore, we believe that regular self-attention mechanism might not be well suited for it. Different approaches have been proposed to overcome these problems, such as the use of efficient attention mechanisms. However, the use of these methods usually comes with a cost, which is a performance reduction caused by information loss. In this study, we present the Multiformer, a Transformer-based model which allows the use of different attention mechanisms on each head. By doing this, the model is able to bias the self-attention towards the extraction of more diverse token interactions, and the information loss is reduced. Finally, we perform an analysis of the head contributions, and we observe that those architectures where all heads relevance is uniformly distributed obtain better results. Our results show that mixing attention patterns along the different heads and layers outperforms our baseline by up to 0.7 BLEU., This work was partially funded by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention for speaker verification

  • India Massana, Miquel Àngel
  • Safari, Pooyan
  • Hernando Pericás, Francisco Javier
Most state-of-the-art Deep Learning systems for text-independent speaker verification are based on speaker embedding extractors. These architectures are commonly composed of a feature extractor front-end together with a pooling layer to encode variable-length utterances into fixed-length speaker vectors. In this paper we present Double Multi-Head Attention (MHA) pooling, which extends our previous approach based on Self MHA. An additional self attention layer is added to the pooling layer that summarizes the context vectors produced by MHA into a unique speaker representation. This method enhances the pooling mechanism by giving weights to the information captured for each head and it results in creating more discriminative speaker embeddings. We have evaluated our approach with the VoxCeleb2 dataset. Our results show 6.09% and 5.23% relative improvement in terms of EER compared to Self Attention pooling and Self MHA, respectively. According to the obtained results, Double MHA has shown to be an excellent approach to efficiently select the most relevant features captured by the CNN-based front-ends from the speech signal., This work was supported by the Spanish project PID2019-107579RBI00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The UPC RDF-to-Text System at WebNLG Challenge 2020

  • Bergés Lladó, David
  • Cantenys Sabà, Roser
  • Creus Castanyer, Roger
  • Domingo Roig, Oriol
  • Rodríguez Fonollosa, José Adrián
This work describes the end-to-end system architecture presented at WebNLG Challenge 2020. The system follows the traditional Machine Translation (MT) pipeline, based on the Transformer model, applied in most text-to-text problems. Our solution is enriched by means of a Back Translation step over the original corpus. Thus, the system directly relies on lexicalise format since the synthetic data limits the use of delexicalisation., We want to thank anonymous reviewers for their comments on the paper. This work was supported by the project ADAVOICE, PID2019-107579RBI00 / AEI / 10.13039/501100011033., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Analysis of acoustic information in end-to-end spoken language translation

  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos
End-to-End Transformer-based models are the most popular approach for Spoken Language Translation (SLT). While obtaining state-of-the-art results, we are still far from understanding how these models extract acoustic information from the data and how they are transformed into semantic representations. In this paper, we seek to provide a better understanding of the flow of acoustic information along speech-to-text translation models. By means of the Speaker Classification and Spectrogram Reconstruction tasks, this study (i) interprets the main role of the encoder with respect to the acoustic features, (ii) highlights the importance of the acoustic information throughout the model and its transfer between encoder and decoder, and (iii) reveals the significant effect of downsampling convolutional layers for learning acoustic features. (iv) Finally, we also observe the existence of a strong correlation between the semantic domain and the speakers’ labels in MuST-C., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RB-I00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention multimodal system for Odyssey 2024 Speech Emotion Recognition Challenge

  • Costa, Federico
  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
As computer-based applications are becoming more integrated into our daily lives, the importance of Speech Emotion Recognition (SER) has increased significantly. Promoting research with innovative approaches in SER, the Odyssey 2024 Speech Emotion Recognition Challenge was organized as part of the Odyssey 2024 Speaker and Language Recognition Workshop. In this paper we describe the Double Multi-Head Attention Multimodal System developed for this challenge. Pre-trained self-supervised models were used to extract informative acoustic and text features. An early fusion strategy was adopted, where a Multi-Head Attention layer transforms these mixed features into complementary contextualized representations. A second attention mechanism is then applied to pool these representations into an utterance-level vector. Our proposed system achieved the third position in the categorical task ranking with a 34.41% Macro-F1 score, where 31 teams participated in total., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The first author is supported by a FI grant from the Catalan government., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

High frequent in-domain word segmentation and forward translation for the WMT21 Biomedical task

  • Rafieian, Bardia
  • Ruiz Costa-Jussà, Marta
This paper reports the optimization of using the out-of-domain data in the Biomedical translation task. We firstly optimized our parallel training dataset using the BabelNet in-domain terminology words. Afterward, to increase the training set, we studied the effects of the out-of-domain data on biomedical translation tasks, and we created a mixture of in-domain and out-of-domain training sets and added more in-domain data using forward translation in the English-Spanish task. Finally, with a simple bpe optimization method, we increased the number of in-domain subwords in our mixed training set and trained the Transformer model on the generated data. Results show improvements using our proposed method. © 2021 Association for Computational Linguistics, This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI 256 / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

BSC-UPC at EmoSPeech-IberLEF2024: attention pooling for emotion recognition

  • Casals i Salvador, Marc
  • Costa, Federico
  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
The domain of speech emotion recognition (SER) has persistently been a frontier within the landscape of machine learning. It is an active field that has been revolutionized in the last few decades and whose implementations are remarkable in multiple applications that could affect daily life. Consequently, the Iberian Languages Evaluation Forum (IberLEF) of 2024 held a competitive challenge to leverage the SER results with a Spanish corpus. This paper presents the approach followed with the goal of participating in this competition. The main architecture consists of different pre-trained speech and text models to extract features from both modalities, utilizing an attention pooling mechanism. The proposed system has achieved the first position in the challenge with an 86.69% in Macro F1-Score., This work has been promoted and financed by the Government of Catalonia through the Aina project, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Toxicity in multilingual machine translation at scale

  • Ruiz Costa-jussà, Marta
  • Smith, Eric
  • Ropers, Christophe
  • Licht, Daniel
  • Maillard, Jean
  • Ferrando Monsonís, Javier
  • Escolano Peinado, Carlos
Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users. In this paper we focus on one type of critical error: added toxicity. We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages. An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%. The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability. We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity. We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity. Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied. Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations. WARNING: this paper contains examples of toxicity that may be offensive or upsetting in nature., Javier Ferrando and Carlos Escolano are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033 and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Towards opening the black box of neural machine translation: source and target interpretations of the transformer

  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Alastruey Lasheras, Belen
  • Escolano Peinado, Carlos
  • Ruiz Costa-jussà, Marta
In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step). However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions. Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions. In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts. Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models. We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour., Javier Ferrando and Gerard I.Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Measuring the mixing of contextual information in the transformer

  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Ruiz Costa-Jussà, Marta
The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block --multi-head attention, residual connection, and layer normalization-- and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods., Javier Ferrando and Gerard I. Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Findings of the first shared task on lifelong learning machine translation

  • Barrault, Loïc
  • Biesialska, Magdalena Marta
  • Ruiz Costa-Jussà, Marta
  • Bougares, Fethi
  • Galibert, Olivier
A lifelong learning system can adapt to new data without forgetting previously acquired knowledge. In this paper, we introduce the first benchmark for lifelong learning machine translation. For this purpose, we provide training, lifelong and test data sets for two language pairs: English-German and English-French. Additionally, we report the results of our baseline systems, which we make available to the public. The goal of this shared task is to encourage research on the emerging topic of lifelong learning machine translation., This work is is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Multilingual neural machine translation: case-study for Catalan, Spanish and Portuguese romance languages

  • Vergés Boncompte, Pere
  • Ruiz Costa-Jussà, Marta
In this paper, we describe the TALP-UPC participation in the WMT Similar Language Translation task between Catalan, Spanish, and Portuguese, all of them, Romance languages. We made use of different techniques to improve the translation between these languages. The multilingual shared encoder/decoder has been used for all of them. Additionally, we applied back-translation to take advantage of the monolingual data. Finally, we have applied fine-tuning to improve the in-domain data. Each of these techniques brings improvements over the previous one. In the official evaluation, our system was ranked 1st in the Portuguese-to-Spanish direction, 2nd in the opposite direction, and 3rd in the Catalan-Spanish pair., This work is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Evaluating gender bias in speech translation

  • Ruiz Costa-Jussà, Marta
  • Basta, Christine Raouf Saad
  • Gallego Olsina, Gerard Ion
The scientific community is increasingly aware of the necessity to embrace pluralism and consistently represent major and minor social groups. Currently, there are no standard evaluation techniques for different types of biases. Accordingly, there is an urgent need to provide evaluation sets and protocols to measure existing biases in our automatic systems. Evaluating the biases should be an essential step towards mitigating them in the systems. This paper introduces WinoST, a new freely available challenge set for evaluating gender bias in speech translation. WinoST is the speech version of WinoMT which is a MT challenge set and both follow an evaluation protocol to measure gender accuracy. Using a state-of-the-art end-to-end speech translation system, we report the gender bias evaluation on four language pairs and we show that gender accuracy in speech translation is more than 23% lower than in MT., This work is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657), the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship, Universitat Politècnica de Catalunya with the collaboration of Banco de Santander and the Spanish Ministerio de Ciencia e Innovación through the project PID2019- 107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Tackling low-resourced sign language translation: UPC at WMT-SLT 22

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion
  • Giró Nieto, Xavier
  • Torres Viñals, Jordi
This paper describes the system developed at the Universitat Politècnica de Catalunya for the Workshop on Machine Translation 2022 Sign Language Translation Task, in particular, for the sign-to-text direction. We use a Transformer model implemented with the Fairseq modeling toolkit. We have experimented with the vocabulary size, data augmentation techniques and pretraining the model with the PHOENIX-14T dataset. Our system obtains 0.50 BLEU score for the test set, improving the organizers’ baseline by 0.38 BLEU. We remark the poor results for both the baseline and our system, and thus, the unreliability of our findings., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 /AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

On the locality of attention in direct speech translation

  • Alastruey Lasheras, Belén
  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Ruiz Costa-Jussà, Marta
Transformers have achieved state-of-the-art results across multiple NLP tasks. However, the self-attention mechanism complexity scales quadratically with the sequence length, creating an obstacle for tasks involving long sequences, like in the speech domain. In this paper, we discuss the usefulness of self-attention for Direct Speech Translation. First, we analyze the layer-wise token contributions in the self-attention of the encoder, unveiling local diagonal patterns. To prove that some attention weights are avoidable, we propose to substitute the standard self-attention with a local efficient one, setting the amount of context used based on the results of the analysis. With this approach, our model matches the baseline performance, and improves the efficiency by skipping the computation of those weights that standard attention discards., This work was partially funded by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Explaining how transformers use context to build predictions

  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Tsiamas, Ioannis
  • Ruiz Costa-jussà, Marta
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions., Javier Ferrando, Gerard I.Gállego and Ioannis Tsiamas are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Speech translation with foundation models and optimal transport: UPC at IWSLT23

  • Tsiamas, Ioannis
  • Gallego Olsina, Gerard Ion
  • Fonollosa, José A. R.
  • Ruiz Costa-jussà, Marta
This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023., The work done by Ioannis Tsiamas and Gerard I. Gállego was supported by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Self-supervised deep learning approaches to speaker recognition: A Ph.D. Thesis overview

  • Khan, Umair
  • Hernando Pericás, Francisco Javier
Recent advances in Deep Learning (DL) for speaker recognition have improved the performance but are constrained to the need of labels for the background data, which is difficult in prac- tice. In i-vector based speaker recognition, cosine (unsuper- vised) and PLDA (supervised) are the basic scoring techniques, with a big performance gap between the two. In this thesis we tried to fill this gap without using speaker labels in several ways. We applied Restricted Boltzmann Machine (RBM) vectors for the tasks of speaker clustering and tracking in TV broadcast shows. The experiments on AGORA database show that us- ing this approach we gain a relative improvement of 12% and 11% for speaker clustering and tracking tasks, respectively. We also applied DL techniques in order to increase the discrimina- tive power of i-vectors in speaker verification task, for which we have proposed the use of autoencoder in several ways, i.e., (1) as a pre-training for a Deep Neural Network (DNN), (2) as a near- est neighbor autoencoder for i-vectors, (3) as an average pooled nearest neighbor autoencoder. The experiments on VoxCeleb database show that we gain a relative improvement of 21%, 42% and 53%, using the three systems respectively. Finally we also proposed a self-supervised end-to-end speaker verification system. The architecture is based on a Convolutional Neural Network (CNN), trained as a siamese network with multiple branches. From the results we can see that our system shows comparable performance to a supervised baseline, This work was supported by the project PID2019-107579RBI00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Syntax-driven iterative expansion language models for controllable text generation

  • Casas Manzanares, Noé
  • Rodríguez Fonollosa, José Adrián
  • Ruiz Costa-Jussà, Marta
The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations., This work is partially supported by Lucy Software / United Language Group (ULG) and the Catalan Agency for Management of University and Research Grants (AGAUR) through an Industrial Ph.D. Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund through the postdoctoral senior grant Ramón y Cajal and by the Agencia Estatal de Investigación through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Mass-editing memory with attention in transformers: a cross-lingual exploration of knowledge

  • Tamayo Mela, Daniel
  • Gonzalez Aguirre, Aitor
  • Hernando Pericás, Francisco Javier
  • Villegas, Marta
Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT., This work has been promoted and financed by the Government of Catalonia through the Aina project, by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia Funded by EU– NextGenerationEU within the framework of the project ILENIA with reference 2022/TL22/00215337, 2022/TL22/00215336, 2022/TL22/00215335, 2022/TL22/00215334, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

End-to-end speech translation with pre-trained models and adapters: UPC at IWSLT 2021

  • Gallego Olsina, Gerard Ion
  • Tsiamas, Ioannis
  • Escolano Peinado, Carlos
  • Rodríguez Fonollosa, José Adrián
  • Ruiz Costa-Jussà, Marta
This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuSTC test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation., This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Automatic adaption of the step size in gradient descent training

  • Nogueiras Rodríguez, Albino
  • Nogueiras Marco, Ignasi
Selecting an appropriate step size is critical in Gradient Descent algorithms used to train Neural Networks for Deep Learning tasks. A small value of the step size leads to slow convergence, while a large value can lead to instability. In this paper, we present ACME, a gradient descent algorithm that automatically updates the step size to its optimum value. When combined with Adaptive Moment Estimation (ADAM) and/or Adaptive Sharpness Aware Minimization (ASAM), ACME achieves performances similar to them, but without the burden of carefully adjusting the learning rate or designing intricate learning rate scheduling policies. The code of ACME, ready to be used by itself or in combination with ADAM and/or ASAM, is available at https://github.com/ignasi00/acme.git., The contribution of Albino Nogueiras-Rodríguez to this paper was supported by the Spanish Government project PID2019-107579RB-I00 / AEI / 10.13039/501100011033. Ignasi Nogueiras-Marco acknowledges funding from “la Caixa” Foundation (ID 100,010,434) under agreement HR-18-0036, and from the Spanish Government through Grant PID2021-122893NB-C21, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The TALP-UPC system description for WMT20 news translation task: multilingual adaptation for low resource MT

  • Escolano Peinado, Carlos
  • Ruiz Costa-Jussà, Marta
  • Rodríguez Fonollosa, José Adrián
In this article, we describe the TALP-UPC participation in the WMT20 news translation shared task for Tamil-English. Given the low amount of parallel training data, we resort to adapt the task to a multilingual system to benefit from the positive transfer from high resource languages. We use iterative backtranslation to fine-tune the system and benefit from the monolingual data available. In order to measure the effectivity of such methods, we compare our results to a bilingual baseline system., This work is supported in part by the Google Faculty Research Award 2019, the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019- 107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

On the use of audio to improve dialogue policies

  • Roncel Diaz, Daniel
  • Costa, Federico
  • Hernando Pericás, Francisco Javier
With the significant progress of speech technologies, spoken goal-oriented dialogue systems are becoming increasingly popular. One of the main modules of a dialogue system is typically the dialogue policy, which is responsible for determining system actions. This component usually relies only on audio transcriptions, being strongly dependent on their quality and ignoring very important extralinguistic information embedded in the user’s speech. In this paper, we propose new architectures to add audio information by combining speech and text embeddings using a Double Multi-Head Attention component. Our experiments show that audio embedding-aware dialogue policies outperform text-based ones, particularly in noisy transcription scenarios, and that how text and audio embeddings are combined is crucial to improve performance. We obtained a 9.8% relative improvement in the User Request Score compared to an only-text-based dialogue system on the DSTC2 dataset., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The second author is supported by a FI grant from the Catalan government., Award-winning, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

SegAugment: maximizing the utility of speech translation data with segmentation-based augmentations

  • Tsiamas, Ioannis
  • Rodríguez Fonollosa, José Adrián
  • Ruiz Costa-jussà, Marta
End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment obtains state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time., Work at UPC was supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)