Skip to main content

ARQUITECTURAS AVANZADAS DE APRENDIZAJE PROFUNDO APLICADAS AL PROCESADO DE VOZ, AUDIO Y LENGUAJE

PID2019-107579RB-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2019
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITECNICA DE CATALUNYA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 84
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Real-time interpolation of global ionospheric maps by means of sparse representation

  • Yang, Heng
  • Monte-Moreno, Enric
  • Hernández-Pajares, Manuel
  • Roma-Dollase, David
In this paper, we propose a method for the generation of real-time global ionospheric map (RT-GIM) of vertical total electron content (VTEC) from GNSS measurements. The need for interpolation arises from the fact that the ionospheric pierce point (IPP) measurements from satellites to stations are not distributed uniformly over the ionosphere, leaving unfilled gaps at oceans or poles. The method we propose is based on using a high-quality historical database of post-processed GIMs that comprises more than two solar cycles, calculates the GIM by weighted superposition on a subset of the database with the compatible solar condition. The linear combination of GIMs in the database was obtained by minimizing a ℓ distance between VTEC measurements at the IPPs and the VTECs from the database, adding a ℓ penalization on the weights to assure a sparse solution. The process uses a Sun-fixed geomagnetic reference frame. This method uses the atomic decomposition/least absolute shrinkage and selection operator (LASSO), which will be denoted as atomic decomposition interpolator of GIMs (ADIGIM). As the computation is done in milliseconds, the interpolation is performed in real time. In this work, two products were developed, denoted as UADG and UARG, the UADG in real time and UARG with a latency of 24 h to benefit from the availability of a greater number of stations. The altimeter JASON3 VTEC measurements were used as reference. The quality of interpolated RT-GIMs from day 258 of 2019 to 155 of the year 2020 is compared with other RT/non-RT GIM products such as those from International GNSS Service (IGS), Centre National d’Etudes Spatiales (CNES), Chinese Academy of Sciences (CAS), Polytechnic University of Catalonia (UPC) and others. The RT ADIGIM performance proved to be better, nearly as good as the rapid or final GIMs computed retrospectively with delays of hours to days. Besides, the non-RT ADIGIM quality is as good or better than most GIM products. The oceanic regions have been included in the assessment which showed that ADIGIM interpolation gives the best estimation (referred to JASON3). The developed method, UADG, will constitute the next-generation UPC RT-GIM, and also UARG will improve the current product UQRG (the current UPC rapid GIM product computed retrospectively) due to its complementary information., This work has been partially supported by the Project PID2019-107579RB-I00 (MICINN). Also was partially supported by the 2017 SGR-0851 Grant of the Generalitat de Catalunya and by the EU Project 101007599 - PITHIA-NRF.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Forecast of the Global TEC by Nearest Neighbour Technique

  • Monte-Moreno, Enric
  • Yang, Heng
  • Hernández-Pajares, Manuel
We propose a method for Global Ionospheric Maps of Total Electron Content forecasting using the Nearest Neighbour method. The assumption is that in a database of global ionosphere maps spanning more than two solar cycles, one can select a set of past observations that have similar geomagnetic conditions to those of the current map. The assumption is that the current ionospheric condition can be expressed by a linear combination of conditions seen in the past. The average of these maps leads to common geomagnetic components being preserved and those not shared by several maps being reduced. The method is based on searching the historical database for the dates of the maps closest to the current map and using as a prediction the maps in the database that correspond to time shifts on the prediction horizons. In contrast to other methods of machine learning, the implementation only requires a distance computation and does not need a previous step of model training and adjustment for each prediction horizon. It also provides confidence intervals for the forecast. The method has been analyzed for two full years (2015 and 2018), for selected days of 2015 and 2018, i.e., two storm days and two non-storm days and the performance of the system has been compared with CODE (24-and 48-h forecast horizons)., This work was been partially supported by the project PID2019-107579RB-I00 (MICINN) and done in the context of PITHIA NRF EU project. The work of Heng Yang was also partially supported by Natural Science Foundation of Chongqing, China (No. cstc2021jcyj-msxmX0191), by the Science and Technology Research Program of Chongqing Municipal Education Commission of China (Grant No. KJQN202101414), and by the Cooperative Projects between Undergraduate Universities in Chongqing and Institutes affiliated with Chinese Academy of Sciences (No.HZ2021014).
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Multiformer: a head-configurable transformer-based model for direct speech translation

  • Sant Muniesa, Gerard
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Alastruey Lasheras, Belen
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as long sequence lengths and redundancy between adjacent tokens. Therefore, we believe that regular self-attention mechanism might not be well suited for it. Different approaches have been proposed to overcome these problems, such as the use of efficient attention mechanisms. However, the use of these methods usually comes with a cost, which is a performance reduction caused by information loss. In this study, we present the Multiformer, a Transformer-based model which allows the use of different attention mechanisms on each head. By doing this, the model is able to bias the self-attention towards the extraction of more diverse token interactions, and the information loss is reduced. Finally, we perform an analysis of the head contributions, and we observe that those architectures where all heads relevance is uniformly distributed obtain better results. Our results show that mixing attention patterns along the different heads and layers outperforms our baseline by up to 0.7 BLEU., This work was partially funded by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Syntax-driven iterative expansion language models for controllable text generation

  • Casas Manzanares, Noé
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations., This work is partially supported by Lucy Software / United Language Group (ULG) and the Catalan Agency for Management of University and Research Grants (AGAUR) through an Industrial Ph.D. Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund through the postdoctoral senior grant Ramón y Cajal and by the Agencia Estatal de Investigación through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Sign language translation from instructional videos

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Cardoso Duarte, Amanda
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Giró Nieto, Xavier|||0000-0002-9935-5332
The advances in automatic sign language translation (SLT) to spoken languages have been mostly benchmarked with datasets of limited size and restricted domains. Our work advances the state of the art by providing the first baseline results on How2Sign, a large and broad dataset. We train a Transformer over I3D video features, using the reduced BLEU as a reference metric for validation, instead of the widely used BLEU score. We report a result of 8.03 on the BLEU score, and publish the first open-source implementation of its kind to promote further advances., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 / AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21 and by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

SegAugment: maximizing the utility of speech translation data with segmentation-based augmentations

  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
  • Ruiz Costa-jussà, Marta
End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment obtains state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time., Work at UPC was supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Analysis of acoustic information in end-to-end spoken language translation

  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
End-to-End Transformer-based models are the most popular approach for Spoken Language Translation (SLT). While obtaining state-of-the-art results, we are still far from understanding how these models extract acoustic information from the data and how they are transformed into semantic representations. In this paper, we seek to provide a better understanding of the flow of acoustic information along speech-to-text translation models. By means of the Speaker Classification and Spectrogram Reconstruction tasks, this study (i) interprets the main role of the encoder with respect to the acoustic features, (ii) highlights the importance of the acoustic information throughout the model and its transfer between encoder and decoder, and (iii) reveals the significant effect of downsampling convolutional layers for learning acoustic features. (iv) Finally, we also observe the existence of a strong correlation between the semantic domain and the speakers’ labels in MuST-C., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RB-I00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Forecast of the global TEC by nearest neighbour technique

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Yang, Heng
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
We propose a method for Global Ionospheric Maps of Total Electron Content forecasting using the Nearest Neighbour method. The assumption is that in a database of global ionosphere maps spanning more than two solar cycles, one can select a set of past observations that have similar geomagnetic conditions to those of the current map. The assumption is that the current ionospheric condition can be expressed by a linear combination of conditions seen in the past. The average of these maps leads to common geomagnetic components being preserved and those not shared by several maps being reduced. The method is based on searching the historical database for the dates of the maps closest to the current map and using as a prediction the maps in the database that correspond to time shifts on the prediction horizons. In contrast to other methods of machine learning, the implementation only requires a distance computation and does not need a previous step of model training and adjustment for each prediction horizon. It also provides confidence intervals for the forecast. The method has been analyzed for two full years (2015 and 2018), for selected days of 2015 and 2018, i.e., two storm days and two non-storm days and the performance of the system has been compared with CODE (24- and 48-h forecast horizons)., This work was been partially supported by the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and done in the context of PITHIA NRF EU project. The work of Heng Yang was also partially supported by Natural Science Foundation of Chongqing, China (No. cstc2021jcyj-msxmX0191) and by the Science and Technology Research Program of Chongqing Municipal Education Commission of China (Grant No. KJ202101414)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Promoting generalized cross-lingual question answering in few-resource scenarios via self-knowledge distillation, Fomentando la respuesta a preguntas cross-lingüística generalizada en escenarios con pocos recursos mediante auto-destilación del conocimiento

  • Carrino, Casimiro Pio
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Gasco Sánchez, Luis
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
We address the challenge of Generalized Cross-Lingual Transfer (G-XLT) in extractive Question Answering, where question and context languages differ, a problem particularly difficult for low-resource languages. Working with only a thousand parallel QA samples, we combine cross-lingual sampling with self-knowledge distillation to regularize cross-lingual fine-tuning. We introduce the novel mean Average Precision at k (mAP@k) coefficient, which mitigates the negative impact of incorrect predictions during training and serves as a diagnostic tool providing early training guidance and reliable indicators of model learning. Evaluations on MLQA, XQuAD, and TyDiQA-GoldP datasets demonstrate that our approach consistently outperforms standard cross-entropy fine-tuning of the mBERT multilingual model. Our method represents a promising alternative to machine translation-based approaches, particularly valuable for low-resource languages where translation quality is poor, offering an efficient solution for cross-lingual transfer in data-scarce settings., This work was supported by the project PID2019-107579RB-I00 (MICINN)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Global teleconnections of rapid ionospheric fluctuations as observed via GNSS measurements

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Yang, Heng
The rapid fluctuations in the electron content of the ionosphere can significantly affect the performance and reliability of communication and GNSS systems. These fluctuations are closely linked to space weather and can result in simultaneous changes in the total electron content of the ionosphere across multiple regions globally. We show the existence of statistically significant synchronous connections between geographical locations when observing fluctuations in electron content. The two findings concerning rapid fluctuations in the electron content of the ionosphere are: a) Evidence of synchronous teleconnections in rapid local electron content variations across a wide range of geographical regions of the globe, and b) The influence of the solar cycle on the geographic distribution of teleconnections is relatively minor, primarily impacting the frequency of teleconnections per unit of time. We characterize teleconnection patterns for different space weather variables throughout the 2016-2022 study period, encompassing the descending and ascending phases of solar cycles 24 and 25. Previous research has mainly focused on the impact of ionospheric irregularities in limited regions, weather conditions, and short time periods, while our findings indicate the occurrence of this phenomenon globally and during both high and low space weather activity., The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191, and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423. This work was supported by the project PID2019-107579RB-I00 (MICINN). The work has been performed coinciding with the execution of the PITHIA-NRF H2020 project (H2020-INFRAIA-2018-2020101007599)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Speech translation with foundation models and optimal transport: UPC at IWSLT23

  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Fonollosa, José A. R.
  • Ruiz Costa-jussà, Marta
This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023., The work done by Ioannis Tsiamas and Gerard I. Gállego was supported by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Language modelling for speaker diarization in telephonic interviews

  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
The aim of this paper is to investigate the benefit of combining both language and acoustic modelling for speaker diarization. Although conventional systems only use acoustic features, in some scenarios linguistic data contain high discriminative speaker information, even more reliable than the acoustic ones. In this study we analyze how an appropriate fusion of both kind of features is able to obtain good results in these cases. The proposed system is based on an iterative algorithm where a LSTM network is used as a speaker classifier. The network is fed with character-level word embeddings and a GMM based acoustic score created with the output labels from previous iterations. The presented algorithm has been evaluated in a Call-Center database, which is composed of telephone interview audios. The combination of acoustic features and linguistic content shows a 84.29% improvement in terms of a word-level DER as compared to a HMM/VB baseline system. The results of this study confirms that linguistic content can be efficiently used for some speaker recognition tasks., This work was partially supported by the Spanish Project DeepVoice (TEC2015-69266-P) and by the project PID2019-107579RBI00/ AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention multimodal system for Odyssey 2024 Speech Emotion Recognition Challenge

  • Costa, Federico|||0000-0002-1389-3595
  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
As computer-based applications are becoming more integrated into our daily lives, the importance of Speech Emotion Recognition (SER) has increased significantly. Promoting research with innovative approaches in SER, the Odyssey 2024 Speech Emotion Recognition Challenge was organized as part of the Odyssey 2024 Speaker and Language Recognition Workshop. In this paper we describe the Double Multi-Head Attention Multimodal System developed for this challenge. Pre-trained self-supervised models were used to extract informative acoustic and text features. An early fusion strategy was adopted, where a Multi-Head Attention layer transforms these mixed features into complementary contextualized representations. A second attention mechanism is then applied to pool these representations into an utterance-level vector. Our proposed system achieved the third position in the categorical task ranking with a 34.41% Macro-F1 score, where 31 teams participated in total., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The first author is supported by a FI grant from the Catalan government., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

The UPC RDF-to-Text System at WebNLG Challenge 2020

  • Bergés Lladó, David
  • Cantenys Sabà, Roser
  • Creus Castanyer, Roger
  • Domingo Roig, Oriol
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
This work describes the end-to-end system architecture presented at WebNLG Challenge 2020. The system follows the traditional Machine Translation (MT) pipeline, based on the Transformer model, applied in most text-to-text problems. Our solution is enriched by means of a Back Translation step over the original corpus. Thus, the system directly relies on lexicalise format since the synthetic data limits the use of delexicalisation., We want to thank anonymous reviewers for their comments on the paper. This work was supported by the project ADAVOICE, PID2019-107579RBI00 / AEI / 10.13039/501100011033.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Automatic adaption of the step size in gradient descent training

  • Nogueiras Rodríguez, Albino|||0000-0002-3159-1718
  • Nogueiras Marco, Ignasi
Selecting an appropriate step size is critical in Gradient Descent algorithms used to train Neural Networks for Deep Learning tasks. A small value of the step size leads to slow convergence, while a large value can lead to instability. In this paper, we present ACME, a gradient descent algorithm that automatically updates the step size to its optimum value. When combined with Adaptive Moment Estimation (ADAM) and/or Adaptive Sharpness Aware Minimization (ASAM), ACME achieves performances similar to them, but without the burden of carefully adjusting the learning rate or designing intricate learning rate scheduling policies. The code of ACME, ready to be used by itself or in combination with ADAM and/or ASAM, is available at https://github.com/ignasi00/acme.git., The contribution of Albino Nogueiras-Rodríguez to this paper was supported by the Spanish Government project PID2019-107579RB-I00 / AEI / 10.13039/501100011033. Ignasi Nogueiras-Marco acknowledges funding from “la Caixa” Foundation (ID 100,010,434) under agreement HR-18-0036, and from the Spanish Government through Grant PID2021-122893NB-C21, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

BSC-UPC at EmoSPeech-IberLEF2024: attention pooling for emotion recognition

  • Casals i Salvador, Marc
  • Costa, Federico|||0000-0002-1389-3595
  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
The domain of speech emotion recognition (SER) has persistently been a frontier within the landscape of machine learning. It is an active field that has been revolutionized in the last few decades and whose implementations are remarkable in multiple applications that could affect daily life. Consequently, the Iberian Languages Evaluation Forum (IberLEF) of 2024 held a competitive challenge to leverage the SER results with a Spanish corpus. This paper presents the approach followed with the goal of participating in this competition. The main architecture consists of different pre-trained speech and text models to extract features from both modalities, utilizing an attention pooling mechanism. The proposed system has achieved the first position in the challenge with an 86.69% in Macro F1-Score., This work has been promoted and financed by the Government of Catalonia through the Aina project, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Evaluating gender bias in speech translation

  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Basta, Christine Raouf Saad
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
The scientific community is increasingly aware of the necessity to embrace pluralism and consistently represent major and minor social groups. Currently, there are no standard evaluation techniques for different types of biases. Accordingly, there is an urgent need to provide evaluation sets and protocols to measure existing biases in our automatic systems. Evaluating the biases should be an essential step towards mitigating them in the systems. This paper introduces WinoST, a new freely available challenge set for evaluating gender bias in speech translation. WinoST is the speech version of WinoMT which is a MT challenge set and both follow an evaluation protocol to measure gender accuracy. Using a state-of-the-art end-to-end speech translation system, we report the gender bias evaluation on four language pairs and we show that gender accuracy in speech translation is more than 23% lower than in MT., This work is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657), the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship, Universitat Politècnica de Catalunya with the collaboration of Banco de Santander and the Spanish Ministerio de Ciencia e Innovación through the project PID2019- 107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Self attention networks in speaker recognition

  • Safari, Pooyan
  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
Recently, there has been a significant surge of interest in Self-Attention Networks (SANs) based on the Transformer architecture. This can be attributed to their notable ability for parallelization and their impressive performance across various Natural Language Processing applications. On the other hand, the utilization of large-scale, multi-purpose language models trained through self-supervision is progressively more prevalent, for tasks like speech recognition. In this context, the pre-trained model, which has been trained on extensive speech data, can be fine-tuned for particular downstream tasks like speaker verification. These massive models typically rely on SANs as their foundational architecture. Therefore, studying the potential capabilities and training challenges of such models is of utmost importance for the future generation of speaker verification systems. In this direction, we propose a speaker embedding extractor based on SANs to obtain a discriminative speaker representation given non-fixed length speech utterances. With the advancements suggested in this work, we could achieve up to 41% relative performance improvement in terms of EER compared to the naive SAN which was proposed in our previous work. Moreover, we empirically show the training instability in such architectures in terms of rank collapse and further investigate the potential solutions to alleviate this shortcoming., This work was supported by the Project PID2019-107579RB-I00/AEI/10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Extensive study on the underlying gender bias in contextualized word embeddings

  • Basta, Christine Raouf Saad
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Casas Manzanares, Noé
Gender bias is affecting many natural language processing applications. While we are still far from proposing debiasing methods that will solve the problem, we are making progress analyzing the impact of this bias in current algorithms. This paper provides an extensive study of the underlying gender bias in popular contextualized word embeddings. Our study provides an insightful analysis of evaluation measures applied to several English data domains and the layers of the contextualized word embeddings. It is also adapted and extended to the Spanish language. Our study points out the advantages and limitations of the various evaluation measures that we are using and aims to standardize the evaluation of gender bias in contextualized word embeddings., This work is supported in part by the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship and the Industrial PhD Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the Projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

A genetic programming approach for estimating economic sentiment in the Baltic countries and the European Union

  • Claveria González, Oscar
  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Torra Porras, Salvador
In this study, we introduce a sentiment construction method based on the evolution of survey-based indicators. We make use of genetic algorithms to evolve qualitative expectations in order to generate country-specific empirical economic sentiment indicators in the three Baltic republics and the European Union. First, for each country we search for the non-linear combination of firms’ and households’ expectations that minimises a fitness function. Second, we compute the frequency with which each survey expectation appears in the evolved indicators and examine the lag structure per variable selected by the algorithm. The industry survey indicator with the highest predictive performance are production expectations, while in the case of the consumer survey the distribution between variables is multi-modal. Third, we evaluate the out-of-sample predictive performance of the generated indicators, obtaining more accurate estimates of year-on-year GDP growth rates than with the scaled industrial and consumer confidence indicators. Finally, we use non-linear constrained optimisation to combine the evolved expectations of firms and consumers and generate aggregate expectations of of year-on-year GDP growth. We find that, in most cases, aggregate expectations outperform recursive autoregressive predictions of economic growth., This work was supported by the Spanish Ministry of Science and Innovation under Grant PID2019-107579RB-I00., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

On the locality of attention in direct speech translation

  • Alastruey Lasheras, Belén
  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
Transformers have achieved state-of-the-art results across multiple NLP tasks. However, the self-attention mechanism complexity scales quadratically with the sequence length, creating an obstacle for tasks involving long sequences, like in the speech domain. In this paper, we discuss the usefulness of self-attention for Direct Speech Translation. First, we analyze the layer-wise token contributions in the self-attention of the encoder, unveiling local diagonal patterns. To prove that some attention weights are avoidable, we propose to substitute the standard self-attention with a local efficient one, setting the amount of context used based on the results of the analysis. With this approach, our model matches the baseline performance, and improves the efficiency by skipping the computation of those weights that standard attention discards., This work was partially funded by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Power-law distribution of rapid ionosphere electron content fluctuations via GNSS measurements

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Yang, Heng
The ionosphere is a dynamic region of the Earth’s upper atmosphere that exhibits various fluctuations in electron density. These fluctuations can be estimated using measurements from the global navigation satellite system (GNSS). In this article, our focus is on characterizing the probability distribution of total electron content (TEC) fluctuations and their temporal duration. The findings of this study, conducted over half a solar cycle and involving more than 100 stations distributed worldwide, demonstrate that the distribution of rapid ionosphere electron content fluctuations follows a power-law behavior, both in amplitude and duration. This explains the occurrence of occasional extreme values of amplitude and duration. In addition, despite significant differences in space weather conditions during the solar cycle, including geomagnetic storms, the power-law distributions of both the rate of change in TEC index amplitude and duration of bursts of activity, exhibit the same shape parameter ( α ), which is similar to the values observed during less active periods. Another finding is the statistical independence of amplitude and duration. The observed power-law distribution and its temporal and spatial dependencies provide practical insights for developing robust models and algorithms for ionospheric impact mitigation in GNSS positioning and navigation systems. Moreover, these findings have implications for space weather monitoring and forecasting, contributing to the assessment of ionospheric disturbances and for a wide range of technological systems, including telecommunication networks among GNSS., This work was supported in part by Project PID2019- 107579RB-I00/AEI/10.13039/501100011033 and in part by the PITHIA-NRF H2020 Project under Grant H2020-INFRAIA-2018-2020101007599. The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191; and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Measuring the mixing of contextual information in the transformer

  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block --multi-head attention, residual connection, and layer normalization-- and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods., Javier Ferrando and Gerard I. Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

End-to-end speech translation with pre-trained models and adapters: UPC at IWSLT 2021

  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuSTC test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation., This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention for speaker verification

  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Safari, Pooyan
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
Most state-of-the-art Deep Learning systems for text-independent speaker verification are based on speaker embedding extractors. These architectures are commonly composed of a feature extractor front-end together with a pooling layer to encode variable-length utterances into fixed-length speaker vectors. In this paper we present Double Multi-Head Attention (MHA) pooling, which extends our previous approach based on Self MHA. An additional self attention layer is added to the pooling layer that summarizes the context vectors produced by MHA into a unique speaker representation. This method enhances the pooling mechanism by giving weights to the information captured for each head and it results in creating more discriminative speaker embeddings. We have evaluated our approach with the VoxCeleb2 dataset. Our results show 6.09% and 5.23% relative improvement in terms of EER compared to Self Attention pooling and Self MHA, respectively. According to the obtained results, Double MHA has shown to be an excellent approach to efficiently select the most relevant features captured by the CNN-based front-ends from the speech signal., This work was supported by the Spanish project PID2019-107579RBI00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Real-time interpolation of global ionospheric maps by means of sparse representation

  • Yang, Heng
  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Roma Dollase, David|||0000-0003-4450-6073
In this paper, we propose a method for the generation of real-time global ionospheric map (RT-GIM) of vertical total electron content (VTEC) from GNSS measurements. The need for interpolation arises from the fact that the ionospheric pierce point (IPP) measurements from satellites to stations are not distributed uniformly over the ionosphere, leaving unfilled gaps at oceans or poles. The method we propose is based on using a high-quality historical database of post-processed GIMs that comprises more than two solar cycles, calculates the GIM by weighted superposition on a subset of the database with the compatible solar condition. The linear combination of GIMs in the database was obtained by minimizing a ℓ2 distance between VTEC measurements at the IPPs and the VTECs from the database, adding a ℓ1 penalization on the weights to assure a sparse solution. The process uses a Sun-fixed geomagnetic reference frame. This method uses the atomic decomposition/least absolute shrinkage and selection operator (LASSO), which will be denoted as atomic decomposition interpolator of GIMs (ADIGIM). As the computation is done in milliseconds, the interpolation is performed in real time. In this work, two products were developed, denoted as UADG and UARG, the UADG in real time and UARG with a latency of 24 h to benefit from the availability of a greater number of stations. The altimeter JASON3 VTEC measurements were used as reference. The quality of interpolated RT-GIMs from day 258 of 2019 to 155 of the year 2020 is compared with other RT/non-RT GIM products such as those from International GNSS Service (IGS), Centre National d’Etudes Spatiales (CNES), Chinese Academy of Sciences (CAS), Polytechnic University of Catalonia (UPC) and others. The RT ADIGIM performance proved to be better, nearly as good as the rapid or final GIMs computed retrospectively with delays of hours to days. Besides, the non-RT ADIGIM quality is as good or better than most GIM products. The oceanic regions have been included in the assessment which showed that ADIGIM interpolation gives the best estimation (referred to JASON3). The developed method, UADG, will constitute the next-generation UPC RT-GIM, and also UARG will improve the current product UQRG (the current UPC rapid GIM product computed retrospectively) due to its complementary information., This work has been partially supported by the Project PID2019-107579RB-I00 (MICINN). Also was partially supported by the 2017 SGR-0851 Grant of the Generalitat de Catalunya and by the EU Project 101007599 - PITHIA-NRF., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

High frequent in-domain word segmentation and forward translation for the WMT21 Biomedical task

  • Rafieian, Bardia
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
This paper reports the optimization of using the out-of-domain data in the Biomedical translation task. We firstly optimized our parallel training dataset using the BabelNet in-domain terminology words. Afterward, to increase the training set, we studied the effects of the out-of-domain data on biomedical translation tasks, and we created a mixture of in-domain and out-of-domain training sets and added more in-domain data using forward translation in the English-Spanish task. Finally, with a simple bpe optimization method, we increased the number of in-domain subwords in our mixed training set and trained the Transformer model on the generated data. Results show improvements using our proposed method. © 2021 Association for Computational Linguistics, This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI 256 / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Tackling low-resourced sign language translation: UPC at WMT-SLT 22

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Giró Nieto, Xavier|||0000-0002-9935-5332
  • Torres Viñals, Jordi|||0000-0003-1963-7418
This paper describes the system developed at the Universitat Politècnica de Catalunya for the Workshop on Machine Translation 2022 Sign Language Translation Task, in particular, for the sign-to-text direction. We use a Transformer model implemented with the Fairseq modeling toolkit. We have experimented with the vocabulary size, data augmentation techniques and pretraining the model with the PHOENIX-14T dataset. Our system obtains 0.50 BLEU score for the test set, improving the organizers’ baseline by 0.38 BLEU. We remark the poor results for both the baseline and our system, and thus, the unreliability of our findings., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 /AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

TALP-UPC at ProbSum 2023: fine-tuning and data augmentation strategies for NER

  • Torrero Salas, Neil
  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
This paper describes the submission of the TALP-UPC team to the Problem List Summarization task from the BioNLP 2023 workshop. This task consists of automatically extracting a list of health issues from the e-health medical record of a given patient. Our submission combines additional steps of data annotationwith finetuning of BERT pre-trained language models. Our experiments focus on the impact of finetuning on different datasets as well as the addition of data augmentation techniques to delay overfitting., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RBI00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Estimation of polar depletion regions by VTEC contrast and watershed enhancing

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Lyu, Haixia
  • Yang, Heng
  • Aragon-Angel, Angela
This article presents a method for determining near-Pole ionization depletion regions and troughs from global navigation satellite system (GNSS) vertical total electron content (VTEC) maps. To define the regions, we use Image processing tools, considering the VTEC maps as gray-level images. In this work, watershed segmentation is applied to determine the edges of regions, based on the flow of areas between the lowest ionization zones (<1%) and the medium ionization regions (>10%). In order to enhance the contrast between low-ionization regions and those with a higher relative ionization, the histogram of the maps is equalized, which increases the contrast of the VTEC maps, which is the distance between low-ionization regions and those with a higher relative ionization are increased. The use of percentile thresholds makes this method independent of hypotheses about the generation of depletions, their morphology, or the number of depletion regions that can occur on a map. We test the performance by comparing the depletion regions identified in eight events reported in the literature under different conditions of the ionosphere. The events used as reference were measured by satellites, radar, and GNSS. The method that we propose allows for real-time processing of GNSS VTEC maps. The morphology of the detected regions matches the shape that is reported in the literature., This work was supported by the project PID2019- 107579RB-I00/AEI/10.13039/501100011033, and ECfunded activity PolarGIMs2, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Sound-to-image translation through direct cross-modal connection using a convolutional–attention generative model

  • Areias Fanzeres, Leonardo|||0000-0003-1179-6596
  • Nadeu Camprubí, Climent|||0000-0002-5863-0983
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
Sound plays a fundamental role in human perception, conveying information about events, objects, and spatial dynamics that may not be visually accessible. However, current technologies such as Acoustic Event Detection typically reduce complex soundscapes to textual labels, often failing to preserve their semantic richness. This limitation motivates the exploration of sound-to-image (S2I) translation as an alternative connection between audio and visual modalities. Unlike multimodal approaches guided by intermediary constraints during the learning process, we investigate S2I translation without class supervision, cluster-based alignment, or textual mediation, a paradigm we refer to as direct S2I translation. To the best of our knowledge, apart from our previous work, no prior study addresses S2I translation under this fully direct setting. We propose a convolutional–attention generative framework composed of an audio encoder and a densely connected GAN integrating self-attention and cross-attention mechanisms. The attention-based model is systematically compared with a purely convolutional baseline. Results show that introducing attention at early stages of the generator significantly improves translation performance, increasing the likelihood of producing interpretable and semantically coherent visual representations of sound. These findings indicate that attention strengthens semantic correspondence between audio and vision while preserving the fully direct nature of the translation process., This research was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under the PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00/AEI/10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Mass-editing memory with attention in transformers: a cross-lingual exploration of knowledge

  • Tamayo Mela, Daniel
  • Gonzalez Aguirre, Aitor
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
  • Villegas, Marta
Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT., This work has been promoted and financed by the Government of Catalonia through the Aina project, by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia Funded by EU– NextGenerationEU within the framework of the project ILENIA with reference 2022/TL22/00215337, 2022/TL22/00215336, 2022/TL22/00215335, 2022/TL22/00215334, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Self-supervised deep learning approaches to speaker recognition: A Ph.D. Thesis overview

  • Khan, Umair
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
Recent advances in Deep Learning (DL) for speaker recognition have improved the performance but are constrained to the need of labels for the background data, which is difficult in prac- tice. In i-vector based speaker recognition, cosine (unsuper- vised) and PLDA (supervised) are the basic scoring techniques, with a big performance gap between the two. In this thesis we tried to fill this gap without using speaker labels in several ways. We applied Restricted Boltzmann Machine (RBM) vectors for the tasks of speaker clustering and tracking in TV broadcast shows. The experiments on AGORA database show that us- ing this approach we gain a relative improvement of 12% and 11% for speaker clustering and tracking tasks, respectively. We also applied DL techniques in order to increase the discrimina- tive power of i-vectors in speaker verification task, for which we have proposed the use of autoencoder in several ways, i.e., (1) as a pre-training for a Deep Neural Network (DNN), (2) as a near- est neighbor autoencoder for i-vectors, (3) as an average pooled nearest neighbor autoencoder. The experiments on VoxCeleb database show that we gain a relative improvement of 21%, 42% and 53%, using the three systems respectively. Finally we also proposed a self-supervised end-to-end speaker verification system. The architecture is based on a Convolutional Neural Network (CNN), trained as a siamese network with multiple branches. From the results we can see that our system shows comparable performance to a supervised baseline, This work was supported by the project PID2019-107579RBI00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Towards opening the black box of neural machine translation: source and target interpretations of the transformer

  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Alastruey Lasheras, Belen
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Ruiz Costa-jussà, Marta
In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step). However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions. Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions. In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts. Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models. We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour., Javier Ferrando and Gerard I.Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Explaining how transformers use context to build predictions

  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Ruiz Costa-jussà, Marta
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions., Javier Ferrando, Gerard I.Gállego and Ioannis Tsiamas are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Toxicity in multilingual machine translation at scale

  • Ruiz Costa-jussà, Marta
  • Smith, Eric
  • Ropers, Christophe
  • Licht, Daniel
  • Maillard, Jean
  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users. In this paper we focus on one type of critical error: added toxicity. We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages. An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%. The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability. We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity. We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity. Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied. Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations. WARNING: this paper contains examples of toxicity that may be offensive or upsetting in nature., Javier Ferrando and Carlos Escolano are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033 and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

On the use of audio to improve dialogue policies

  • Roncel Diaz, Daniel
  • Costa, Federico|||0000-0002-1389-3595
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
With the significant progress of speech technologies, spoken goal-oriented dialogue systems are becoming increasingly popular. One of the main modules of a dialogue system is typically the dialogue policy, which is responsible for determining system actions. This component usually relies only on audio transcriptions, being strongly dependent on their quality and ignoring very important extralinguistic information embedded in the user’s speech. In this paper, we propose new architectures to add audio information by combining speech and text embeddings using a Double Multi-Head Attention component. Our experiments show that audio embedding-aware dialogue policies outperform text-based ones, particularly in noisy transcription scenarios, and that how text and audio embeddings are combined is crucial to improve performance. We obtained a 9.8% relative improvement in the User Request Score compared to an only-text-based dialogue system on the DSTC2 dataset., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The second author is supported by a FI grant from the Catalan government., Award-winning
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Multilingual neural machine translation: case-study for Catalan, Spanish and Portuguese romance languages

  • Vergés Boncompte, Pere
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
In this paper, we describe the TALP-UPC participation in the WMT Similar Language Translation task between Catalan, Spanish, and Portuguese, all of them, Romance languages. We made use of different techniques to improve the translation between these languages. The multilingual shared encoder/decoder has been used for all of them. Additionally, we applied back-translation to take advantage of the monolingual data. Finally, we have applied fine-tuning to improve the in-domain data. Each of these techniques brings improvements over the previous one. In the official evaluation, our system was ranked 1st in the Portuguese-to-Spanish direction, 2nd in the opposite direction, and 3rd in the Catalan-Spanish pair., This work is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Estudio, informe, memoria (Report)

Explaining how transformers use context to build predictions

  • Ferrando Monsonís, Javier|||0000-0002-2637-0961
  • Gallego Olsina, Gerard Ion|||0000-0001-7466-3606
  • Tsiamas, Ioannis|||0000-0003-1049-2515
  • Ruiz Costa-jussà, Marta
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model's prediction, it is still unclear how prior words affect the model's decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions., Javier Ferrando, Gerard I. Gállego and Ioannis Tsiamas are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

The TALP-UPC system description for WMT20 news translation task: multilingual adaptation for low resource MT

  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
In this article, we describe the TALP-UPC participation in the WMT20 news translation shared task for Tamil-English. Given the low amount of parallel training data, we resort to adapt the task to a multilingual system to benefit from the positive transfer from high resource languages. We use iterative backtranslation to fine-tune the system and benefit from the monolingual data available. In order to measure the effectivity of such methods, we compare our results to a bilingual baseline system., This work is supported in part by the Google Faculty Research Award 2019, the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019- 107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

From bilingual to multilingual neural-based machine translation by incremental training

  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
A common intermediate language representation in neural machine translation can be used to extend bilingual systems by incremental training. We propose a new architecture based on introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we can train multiple encoders and decoders for each language, sharing among them a common intermediate representation. Translation results on the low-resource tasks (Turkish-English and Kazakh-English tasks) show a BLEU improvement of up to 2.8 points. However, results on a larger dataset (Russian-English and Kazakh-English) show BLEU losses of a similar amount. While our system provides improvements only for the low-resource tasks in terms of translation quality, our system is capable of quickly deploying new language pairs without the need to retrain the rest of the system, which may be a game changer in some situations. Specifically, what is most relevant regarding our architecture is that it is capable of: reducing the number of production systems, with respect to the number of languages, from quadratic to linear; incrementally adding a new language to the system without retraining the languages already there; and allowing for translations from the new language to all the others present in the system., This work also is supported in part by the Google Faculty Research Awards and, also by, the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the projects EUR2019-103819, PCIN-2017-079 andPID2019-107579RB-I00., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Sound-to-imagination: an exploratory study on cross-modal translation using diverse audiovisual data

  • Areias Fanzeres, Leonardo|||0000-0003-1179-6596
  • Nadeu Camprubí, Climent|||0000-0002-5863-0983
The motivation of our research is to explore the possibilities of automatic sound-to-image (S2I) translation for enabling a human receiver to visually infer occurrences of sound-related events. We expect the computer to ‘imagine’ scenes from captured sounds, generating original images that depict the sound-emitting sources. Previous studies on similar topics opted for simplified approaches using data with low content diversity and/or supervision/self-supervision for training. In contrast, our approach involves performing S2I translation using thousands of distinct and unknown scenes, using sound class annotations solely for data preparation, just enough to ensure aural–visual semantic coherence. To model the translator, we employ an audio encoder and a conditional generative adversarial network (GAN) with a deep densely connected generator. Furthermore, we present a solution using informativity classifiers for quantitatively evaluating the generated images. This allows us to analyze the influence of network-bottleneck variation on the translation process, highlighting a potential trade-off between informativity and pixel space convergence. Despite the complexity of the specified S2I translation task, we were able to generalize the model enough to obtain more than 14%, on average, of interpretable and semantically coherent images translated from unknown sounds., The present work was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI), project PID2019-107579RBI00/AEI/10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Findings of the first shared task on lifelong learning machine translation

  • Barrault, Loïc
  • Biesialska, Magdalena Marta|||0000-0001-7890-3523
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Bougares, Fethi
  • Galibert, Olivier
A lifelong learning system can adapt to new data without forgetting previously acquired knowledge. In this paper, we introduce the first benchmark for lifelong learning machine translation. For this purpose, we provide training, lifelong and test data sets for two language pairs: English-German and English-French. Additionally, we report the results of our baseline systems, which we make available to the public. The goal of this shared task is to encourage research on the emerging topic of lifelong learning machine translation., This work is is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

TALP-UPC at ProbSum 2023: fine-tuning and data augmentation strategies for NER

  • Torrero Salas, Neil
  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos
This paper describes the submission of the TALP-UPC team to the Problem List Summarization task from the BioNLP 2023 workshop. This task consists of automatically extracting a list of health issues from the e-health medical record of a given patient. Our submission combines additional steps of data annotationwith finetuning of BERT pre-trained language models. Our experiments focus on the impact of finetuning on different datasets as well as the addition of data augmentation techniques to delay overfitting., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RBI00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Self attention networks in speaker recognition

  • Safari, Pooyan
  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
Recently, there has been a significant surge of interest in Self-Attention Networks (SANs) based on the Transformer architecture. This can be attributed to their notable ability for parallelization and their impressive performance across various Natural Language Processing applications. On the other hand, the utilization of large-scale, multi-purpose language models trained through self-supervision is progressively more prevalent, for tasks like speech recognition. In this context, the pre-trained model, which has been trained on extensive speech data, can be fine-tuned for particular downstream tasks like speaker verification. These massive models typically rely on SANs as their foundational architecture. Therefore, studying the potential capabilities and training challenges of such models is of utmost importance for the future generation of speaker verification systems. In this direction, we propose a speaker embedding extractor based on SANs to obtain a discriminative speaker representation given non-fixed length speech utterances. With the advancements suggested in this work, we could achieve up to 41% relative performance improvement in terms of EER compared to the naive SAN which was proposed in our previous work. Moreover, we empirically show the training instability in such architectures in terms of rank collapse and further investigate the potential solutions to alleviate this shortcoming., This work was supported by the Project PID2019-107579RB-I00/AEI/10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication

Explaining how transformers use context to build predictions

  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Tsiamas, Ioannis
  • Ruiz Costa-jussà, Marta
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model's prediction, it is still unclear how prior words affect the model's decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions., Javier Ferrando, Gerard I. Gállego and Ioannis Tsiamas are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Preprint
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Sign language translation from instructional videos

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion
  • Cardoso Duarte, Amanda
  • Torres Viñals, Jordi
  • Giró Nieto, Xavier
The advances in automatic sign language translation (SLT) to spoken languages have been mostly benchmarked with datasets of limited size and restricted domains. Our work advances the state of the art by providing the first baseline results on How2Sign, a large and broad dataset. We train a Transformer over I3D video features, using the reduced BLEU as a reference metric for validation, instead of the widely used BLEU score. We report a result of 8.03 on the BLEU score, and publish the first open-source implementation of its kind to promote further advances., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 / AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21 and by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Multiformer: a head-configurable transformer-based model for direct speech translation

  • Sant Muniesa, Gerard
  • Gallego Olsina, Gerard Ion
  • Alastruey Lasheras, Belen
  • Ruiz Costa-Jussà, Marta
Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as long sequence lengths and redundancy between adjacent tokens. Therefore, we believe that regular self-attention mechanism might not be well suited for it. Different approaches have been proposed to overcome these problems, such as the use of efficient attention mechanisms. However, the use of these methods usually comes with a cost, which is a performance reduction caused by information loss. In this study, we present the Multiformer, a Transformer-based model which allows the use of different attention mechanisms on each head. By doing this, the model is able to bias the self-attention towards the extraction of more diverse token interactions, and the information loss is reduced. Finally, we perform an analysis of the head contributions, and we observe that those architectures where all heads relevance is uniformly distributed obtain better results. Our results show that mixing attention patterns along the different heads and layers outperforms our baseline by up to 0.7 BLEU., This work was partially funded by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention for speaker verification

  • India Massana, Miquel Àngel
  • Safari, Pooyan
  • Hernando Pericás, Francisco Javier
Most state-of-the-art Deep Learning systems for text-independent speaker verification are based on speaker embedding extractors. These architectures are commonly composed of a feature extractor front-end together with a pooling layer to encode variable-length utterances into fixed-length speaker vectors. In this paper we present Double Multi-Head Attention (MHA) pooling, which extends our previous approach based on Self MHA. An additional self attention layer is added to the pooling layer that summarizes the context vectors produced by MHA into a unique speaker representation. This method enhances the pooling mechanism by giving weights to the information captured for each head and it results in creating more discriminative speaker embeddings. We have evaluated our approach with the VoxCeleb2 dataset. Our results show 6.09% and 5.23% relative improvement in terms of EER compared to Self Attention pooling and Self MHA, respectively. According to the obtained results, Double MHA has shown to be an excellent approach to efficiently select the most relevant features captured by the CNN-based front-ends from the speech signal., This work was supported by the Spanish project PID2019-107579RBI00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The UPC RDF-to-Text System at WebNLG Challenge 2020

  • Bergés Lladó, David
  • Cantenys Sabà, Roser
  • Creus Castanyer, Roger
  • Domingo Roig, Oriol
  • Rodríguez Fonollosa, José Adrián
This work describes the end-to-end system architecture presented at WebNLG Challenge 2020. The system follows the traditional Machine Translation (MT) pipeline, based on the Transformer model, applied in most text-to-text problems. Our solution is enriched by means of a Back Translation step over the original corpus. Thus, the system directly relies on lexicalise format since the synthetic data limits the use of delexicalisation., We want to thank anonymous reviewers for their comments on the paper. This work was supported by the project ADAVOICE, PID2019-107579RBI00 / AEI / 10.13039/501100011033., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Estimation of polar depletion regions by VTEC contrast and watershed enhancing

  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Lyu, Haixia
  • Yang, Heng
  • Aragon-Angel, Angela
This article presents a method for determining near-Pole ionization depletion regions and troughs from global navigation satellite system (GNSS) vertical total electron content (VTEC) maps. To define the regions, we use Image processing tools, considering the VTEC maps as gray-level images. In this work, watershed segmentation is applied to determine the edges of regions, based on the flow of areas between the lowest ionization zones (<1%) and the medium ionization regions (>10%). In order to enhance the contrast between low-ionization regions and those with a higher relative ionization, the histogram of the maps is equalized, which increases the contrast of the VTEC maps, which is the distance between low-ionization regions and those with a higher relative ionization are increased. The use of percentile thresholds makes this method independent of hypotheses about the generation of depletions, their morphology, or the number of depletion regions that can occur on a map. We test the performance by comparing the depletion regions identified in eight events reported in the literature under different conditions of the ionosphere. The events used as reference were measured by satellites, radar, and GNSS. The method that we propose allows for real-time processing of GNSS VTEC maps. The morphology of the detected regions matches the shape that is reported in the literature., This work was supported by the project PID2019- 107579RB-I00/AEI/10.13039/501100011033, and ECfunded activity PolarGIMs2, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Promoting generalized cross-lingual question answering in few-resource scenarios via self-knowledge distillation, Fomentando la respuesta a preguntas cross-lingüística generalizada en escenarios con pocos recursos mediante auto-destilación del conocimiento

  • Carrino, Casimiro Pio
  • Escolano Peinado, Carlos
  • Gasco Sánchez, Luis
  • Rodríguez Fonollosa, José Adrián
We address the challenge of Generalized Cross-Lingual Transfer (G-XLT) in extractive Question Answering, where question and context languages differ, a problem particularly difficult for low-resource languages. Working with only a thousand parallel QA samples, we combine cross-lingual sampling with self-knowledge distillation to regularize cross-lingual fine-tuning. We introduce the novel mean Average Precision at k (mAP@k) coefficient, which mitigates the negative impact of incorrect predictions during training and serves as a diagnostic tool providing early training guidance and reliable indicators of model learning. Evaluations on MLQA, XQuAD, and TyDiQA-GoldP datasets demonstrate that our approach consistently outperforms standard cross-entropy fine-tuning of the mBERT multilingual model. Our method represents a promising alternative to machine translation-based approaches, particularly valuable for low-resource languages where translation quality is poor, offering an efficient solution for cross-lingual transfer in data-scarce settings., This work was supported by the project PID2019-107579RB-I00 (MICINN)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Analysis of acoustic information in end-to-end spoken language translation

  • Sant Muniesa, Gerard
  • Escolano Peinado, Carlos
End-to-End Transformer-based models are the most popular approach for Spoken Language Translation (SLT). While obtaining state-of-the-art results, we are still far from understanding how these models extract acoustic information from the data and how they are transformed into semantic representations. In this paper, we seek to provide a better understanding of the flow of acoustic information along speech-to-text translation models. By means of the Speaker Classification and Spectrogram Reconstruction tasks, this study (i) interprets the main role of the encoder with respect to the acoustic features, (ii) highlights the importance of the acoustic information throughout the model and its transfer between encoder and decoder, and (iii) reveals the significant effect of downsampling convolutional layers for learning acoustic features. (iv) Finally, we also observe the existence of a strong correlation between the semantic domain and the speakers’ labels in MuST-C., This work was funded by Spanish State Research Agency (AEI) project PID2019-107579RB-I00 (AEI/10.13039/501100011033) and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Double multi-head attention multimodal system for Odyssey 2024 Speech Emotion Recognition Challenge

  • Costa, Federico
  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
As computer-based applications are becoming more integrated into our daily lives, the importance of Speech Emotion Recognition (SER) has increased significantly. Promoting research with innovative approaches in SER, the Odyssey 2024 Speech Emotion Recognition Challenge was organized as part of the Odyssey 2024 Speaker and Language Recognition Workshop. In this paper we describe the Double Multi-Head Attention Multimodal System developed for this challenge. Pre-trained self-supervised models were used to extract informative acoustic and text features. An early fusion strategy was adopted, where a Multi-Head Attention layer transforms these mixed features into complementary contextualized representations. A second attention mechanism is then applied to pool these representations into an utterance-level vector. Our proposed system achieved the third position in the categorical task ranking with a 34.41% Macro-F1 score, where 31 teams participated in total., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The first author is supported by a FI grant from the Catalan government., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A genetic programming approach for estimating economic sentiment in the Baltic countries and the European Union

  • Claveria González, Oscar
  • Monte Moreno, Enrique
  • Torra Porras, Salvador
In this study, we introduce a sentiment construction method based on the evolution of survey-based indicators. We make use of genetic algorithms to evolve qualitative expectations in order to generate country-specific empirical economic sentiment indicators in the three Baltic republics and the European Union. First, for each country we search for the non-linear combination of firms’ and households’ expectations that minimises a fitness function. Second, we compute the frequency with which each survey expectation appears in the evolved indicators and examine the lag structure per variable selected by the algorithm. The industry survey indicator with the highest predictive performance are production expectations, while in the case of the consumer survey the distribution between variables is multi-modal. Third, we evaluate the out-of-sample predictive performance of the generated indicators, obtaining more accurate estimates of year-on-year GDP growth rates than with the scaled industrial and consumer confidence indicators. Finally, we use non-linear constrained optimisation to combine the evolved expectations of firms and consumers and generate aggregate expectations of of year-on-year GDP growth. We find that, in most cases, aggregate expectations outperform recursive autoregressive predictions of economic growth., This work was supported by the Spanish Ministry of Science and Innovation under Grant PID2019-107579RB-I00., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

High frequent in-domain word segmentation and forward translation for the WMT21 Biomedical task

  • Rafieian, Bardia
  • Ruiz Costa-Jussà, Marta
This paper reports the optimization of using the out-of-domain data in the Biomedical translation task. We firstly optimized our parallel training dataset using the BabelNet in-domain terminology words. Afterward, to increase the training set, we studied the effects of the out-of-domain data on biomedical translation tasks, and we created a mixture of in-domain and out-of-domain training sets and added more in-domain data using forward translation in the English-Spanish task. Finally, with a simple bpe optimization method, we increased the number of in-domain subwords in our mixed training set and trained the Transformer model on the generated data. Results show improvements using our proposed method. © 2021 Association for Computational Linguistics, This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI 256 / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

BSC-UPC at EmoSPeech-IberLEF2024: attention pooling for emotion recognition

  • Casals i Salvador, Marc
  • Costa, Federico
  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
The domain of speech emotion recognition (SER) has persistently been a frontier within the landscape of machine learning. It is an active field that has been revolutionized in the last few decades and whose implementations are remarkable in multiple applications that could affect daily life. Consequently, the Iberian Languages Evaluation Forum (IberLEF) of 2024 held a competitive challenge to leverage the SER results with a Spanish corpus. This paper presents the approach followed with the goal of participating in this competition. The main architecture consists of different pre-trained speech and text models to extract features from both modalities, utilizing an attention pooling mechanism. The proposed system has achieved the first position in the challenge with an 86.69% in Macro F1-Score., This work has been promoted and financed by the Government of Catalonia through the Aina project, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Real-time interpolation of global ionospheric maps by means of sparse representation

  • Yang, Heng
  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Roma Dollase, David
In this paper, we propose a method for the generation of real-time global ionospheric map (RT-GIM) of vertical total electron content (VTEC) from GNSS measurements. The need for interpolation arises from the fact that the ionospheric pierce point (IPP) measurements from satellites to stations are not distributed uniformly over the ionosphere, leaving unfilled gaps at oceans or poles. The method we propose is based on using a high-quality historical database of post-processed GIMs that comprises more than two solar cycles, calculates the GIM by weighted superposition on a subset of the database with the compatible solar condition. The linear combination of GIMs in the database was obtained by minimizing a ℓ2 distance between VTEC measurements at the IPPs and the VTECs from the database, adding a ℓ1 penalization on the weights to assure a sparse solution. The process uses a Sun-fixed geomagnetic reference frame. This method uses the atomic decomposition/least absolute shrinkage and selection operator (LASSO), which will be denoted as atomic decomposition interpolator of GIMs (ADIGIM). As the computation is done in milliseconds, the interpolation is performed in real time. In this work, two products were developed, denoted as UADG and UARG, the UADG in real time and UARG with a latency of 24 h to benefit from the availability of a greater number of stations. The altimeter JASON3 VTEC measurements were used as reference. The quality of interpolated RT-GIMs from day 258 of 2019 to 155 of the year 2020 is compared with other RT/non-RT GIM products such as those from International GNSS Service (IGS), Centre National d’Etudes Spatiales (CNES), Chinese Academy of Sciences (CAS), Polytechnic University of Catalonia (UPC) and others. The RT ADIGIM performance proved to be better, nearly as good as the rapid or final GIMs computed retrospectively with delays of hours to days. Besides, the non-RT ADIGIM quality is as good or better than most GIM products. The oceanic regions have been included in the assessment which showed that ADIGIM interpolation gives the best estimation (referred to JASON3). The developed method, UADG, will constitute the next-generation UPC RT-GIM, and also UARG will improve the current product UQRG (the current UPC rapid GIM product computed retrospectively) due to its complementary information., This work has been partially supported by the Project PID2019-107579RB-I00 (MICINN). Also was partially supported by the 2017 SGR-0851 Grant of the Generalitat de Catalunya and by the EU Project 101007599 - PITHIA-NRF., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Toxicity in multilingual machine translation at scale

  • Ruiz Costa-jussà, Marta
  • Smith, Eric
  • Ropers, Christophe
  • Licht, Daniel
  • Maillard, Jean
  • Ferrando Monsonís, Javier
  • Escolano Peinado, Carlos
Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users. In this paper we focus on one type of critical error: added toxicity. We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages. An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%. The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability. We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity. We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity. Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied. Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations. WARNING: this paper contains examples of toxicity that may be offensive or upsetting in nature., Javier Ferrando and Carlos Escolano are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033 and the “European Union NextGenerationEU/PRTR” under the project ROB-IN (PLEC2021-007859), Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Global teleconnections of rapid ionospheric fluctuations as observed via GNSS measurements

  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Yang, Heng
The rapid fluctuations in the electron content of the ionosphere can significantly affect the performance and reliability of communication and GNSS systems. These fluctuations are closely linked to space weather and can result in simultaneous changes in the total electron content of the ionosphere across multiple regions globally. We show the existence of statistically significant synchronous connections between geographical locations when observing fluctuations in electron content. The two findings concerning rapid fluctuations in the electron content of the ionosphere are: a) Evidence of synchronous teleconnections in rapid local electron content variations across a wide range of geographical regions of the globe, and b) The influence of the solar cycle on the geographic distribution of teleconnections is relatively minor, primarily impacting the frequency of teleconnections per unit of time. We characterize teleconnection patterns for different space weather variables throughout the 2016-2022 study period, encompassing the descending and ascending phases of solar cycles 24 and 25. Previous research has mainly focused on the impact of ionospheric irregularities in limited regions, weather conditions, and short time periods, while our findings indicate the occurrence of this phenomenon globally and during both high and low space weather activity., The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191, and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423. This work was supported by the project PID2019-107579RB-I00 (MICINN). The work has been performed coinciding with the execution of the PITHIA-NRF H2020 project (H2020-INFRAIA-2018-2020101007599)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Towards opening the black box of neural machine translation: source and target interpretations of the transformer

  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Alastruey Lasheras, Belen
  • Escolano Peinado, Carlos
  • Ruiz Costa-jussà, Marta
In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step). However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions. Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions. In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts. Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models. We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour., Javier Ferrando and Gerard I.Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Power-law distribution of rapid ionosphere electron content fluctuations via GNSS measurements

  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Yang, Heng
The ionosphere is a dynamic region of the Earth’s upper atmosphere that exhibits various fluctuations in electron density. These fluctuations can be estimated using measurements from the global navigation satellite system (GNSS). In this article, our focus is on characterizing the probability distribution of total electron content (TEC) fluctuations and their temporal duration. The findings of this study, conducted over half a solar cycle and involving more than 100 stations distributed worldwide, demonstrate that the distribution of rapid ionosphere electron content fluctuations follows a power-law behavior, both in amplitude and duration. This explains the occurrence of occasional extreme values of amplitude and duration. In addition, despite significant differences in space weather conditions during the solar cycle, including geomagnetic storms, the power-law distributions of both the rate of change in TEC index amplitude and duration of bursts of activity, exhibit the same shape parameter ( α ), which is similar to the values observed during less active periods. Another finding is the statistical independence of amplitude and duration. The observed power-law distribution and its temporal and spatial dependencies provide practical insights for developing robust models and algorithms for ionospheric impact mitigation in GNSS positioning and navigation systems. Moreover, these findings have implications for space weather monitoring and forecasting, contributing to the assessment of ionospheric disturbances and for a wide range of technological systems, including telecommunication networks among GNSS., This work was supported in part by Project PID2019- 107579RB-I00/AEI/10.13039/501100011033 and in part by the PITHIA-NRF H2020 Project under Grant H2020-INFRAIA-2018-2020101007599. The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191; and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Measuring the mixing of contextual information in the transformer

  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Ruiz Costa-Jussà, Marta
The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block --multi-head attention, residual connection, and layer normalization-- and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods., Javier Ferrando and Gerard I. Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Findings of the first shared task on lifelong learning machine translation

  • Barrault, Loïc
  • Biesialska, Magdalena Marta
  • Ruiz Costa-Jussà, Marta
  • Bougares, Fethi
  • Galibert, Olivier
A lifelong learning system can adapt to new data without forgetting previously acquired knowledge. In this paper, we introduce the first benchmark for lifelong learning machine translation. For this purpose, we provide training, lifelong and test data sets for two language pairs: English-German and English-French. Additionally, we report the results of our baseline systems, which we make available to the public. The goal of this shared task is to encourage research on the emerging topic of lifelong learning machine translation., This work is is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Multilingual neural machine translation: case-study for Catalan, Spanish and Portuguese romance languages

  • Vergés Boncompte, Pere
  • Ruiz Costa-Jussà, Marta
In this paper, we describe the TALP-UPC participation in the WMT Similar Language Translation task between Catalan, Spanish, and Portuguese, all of them, Romance languages. We made use of different techniques to improve the translation between these languages. The multilingual shared encoder/decoder has been used for all of them. Additionally, we applied back-translation to take advantage of the monolingual data. Finally, we have applied fine-tuning to improve the in-domain data. Each of these techniques brings improvements over the previous one. In the official evaluation, our system was ranked 1st in the Portuguese-to-Spanish direction, 2nd in the opposite direction, and 3rd in the Catalan-Spanish pair., This work is supported in part by the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Evaluating gender bias in speech translation

  • Ruiz Costa-Jussà, Marta
  • Basta, Christine Raouf Saad
  • Gallego Olsina, Gerard Ion
The scientific community is increasingly aware of the necessity to embrace pluralism and consistently represent major and minor social groups. Currently, there are no standard evaluation techniques for different types of biases. Accordingly, there is an urgent need to provide evaluation sets and protocols to measure existing biases in our automatic systems. Evaluating the biases should be an essential step towards mitigating them in the systems. This paper introduces WinoST, a new freely available challenge set for evaluating gender bias in speech translation. WinoST is the speech version of WinoMT which is a MT challenge set and both follow an evaluation protocol to measure gender accuracy. Using a state-of-the-art end-to-end speech translation system, we report the gender bias evaluation on four language pairs and we show that gender accuracy in speech translation is more than 23% lower than in MT., This work is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657), the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship, Universitat Politècnica de Catalunya with the collaboration of Banco de Santander and the Spanish Ministerio de Ciencia e Innovación through the project PID2019- 107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Sound-to-imagination: an exploratory study on cross-modal translation using diverse audiovisual data

  • Areias Fanzeres, Leonardo
  • Nadeu Camprubí, Climent
The motivation of our research is to explore the possibilities of automatic sound-to-image (S2I) translation for enabling a human receiver to visually infer occurrences of sound-related events. We expect the computer to ‘imagine’ scenes from captured sounds, generating original images that depict the sound-emitting sources. Previous studies on similar topics opted for simplified approaches using data with low content diversity and/or supervision/self-supervision for training. In contrast, our approach involves performing S2I translation using thousands of distinct and unknown scenes, using sound class annotations solely for data preparation, just enough to ensure aural–visual semantic coherence. To model the translator, we employ an audio encoder and a conditional generative adversarial network (GAN) with a deep densely connected generator. Furthermore, we present a solution using informativity classifiers for quantitatively evaluating the generated images. This allows us to analyze the influence of network-bottleneck variation on the translation process, highlighting a potential trade-off between informativity and pixel space convergence. Despite the complexity of the specified S2I translation task, we were able to generalize the model enough to obtain more than 14%, on average, of interpretable and semantically coherent images translated from unknown sounds., The present work was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI), project PID2019-107579RBI00/AEI/10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Tackling low-resourced sign language translation: UPC at WMT-SLT 22

  • Tarrés Benet, Laia
  • Gallego Olsina, Gerard Ion
  • Giró Nieto, Xavier
  • Torres Viñals, Jordi
This paper describes the system developed at the Universitat Politècnica de Catalunya for the Workshop on Machine Translation 2022 Sign Language Translation Task, in particular, for the sign-to-text direction. We use a Transformer model implemented with the Fairseq modeling toolkit. We have experimented with the vocabulary size, data augmentation techniques and pretraining the model with the PHOENIX-14T dataset. Our system obtains 0.50 BLEU score for the test set, improving the organizers’ baseline by 0.38 BLEU. We remark the poor results for both the baseline and our system, and thus, the unreliability of our findings., This research was partially supported by research grant Adavoice PID2019-107579RB-I00 /AEI / 10.13039/501100011033, research grants PRE2020-094223, PID2021-126248OB-I00 and PID2019-107255GB-C21., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Extensive study on the underlying gender bias in contextualized word embeddings

  • Basta, Christine Raouf Saad
  • Ruiz Costa-Jussà, Marta
  • Casas Manzanares, Noé
Gender bias is affecting many natural language processing applications. While we are still far from proposing debiasing methods that will solve the problem, we are making progress analyzing the impact of this bias in current algorithms. This paper provides an extensive study of the underlying gender bias in popular contextualized word embeddings. Our study provides an insightful analysis of evaluation measures applied to several English data domains and the layers of the contextualized word embeddings. It is also adapted and extended to the Spanish language. Our study points out the advantages and limitations of the various evaluation measures that we are using and aims to standardize the evaluation of gender bias in contextualized word embeddings., This work is supported in part by the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship and the Industrial PhD Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the Projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

On the locality of attention in direct speech translation

  • Alastruey Lasheras, Belén
  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Ruiz Costa-Jussà, Marta
Transformers have achieved state-of-the-art results across multiple NLP tasks. However, the self-attention mechanism complexity scales quadratically with the sequence length, creating an obstacle for tasks involving long sequences, like in the speech domain. In this paper, we discuss the usefulness of self-attention for Direct Speech Translation. First, we analyze the layer-wise token contributions in the self-attention of the encoder, unveiling local diagonal patterns. To prove that some attention weights are avoidable, we propose to substitute the standard self-attention with a local efficient one, setting the amount of context used based on the results of the analysis. With this approach, our model matches the baseline performance, and improves the efficiency by skipping the computation of those weights that standard attention discards., This work was partially funded by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Forecast of the global TEC by nearest neighbour technique

  • Monte Moreno, Enrique
  • Yang, Heng
  • Hernández Pajares, Manuel
We propose a method for Global Ionospheric Maps of Total Electron Content forecasting using the Nearest Neighbour method. The assumption is that in a database of global ionosphere maps spanning more than two solar cycles, one can select a set of past observations that have similar geomagnetic conditions to those of the current map. The assumption is that the current ionospheric condition can be expressed by a linear combination of conditions seen in the past. The average of these maps leads to common geomagnetic components being preserved and those not shared by several maps being reduced. The method is based on searching the historical database for the dates of the maps closest to the current map and using as a prediction the maps in the database that correspond to time shifts on the prediction horizons. In contrast to other methods of machine learning, the implementation only requires a distance computation and does not need a previous step of model training and adjustment for each prediction horizon. It also provides confidence intervals for the forecast. The method has been analyzed for two full years (2015 and 2018), for selected days of 2015 and 2018, i.e., two storm days and two non-storm days and the performance of the system has been compared with CODE (24- and 48-h forecast horizons)., This work was been partially supported by the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and done in the context of PITHIA NRF EU project. The work of Heng Yang was also partially supported by Natural Science Foundation of Chongqing, China (No. cstc2021jcyj-msxmX0191) and by the Science and Technology Research Program of Chongqing Municipal Education Commission of China (Grant No. KJ202101414)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Explaining how transformers use context to build predictions

  • Ferrando Monsonís, Javier
  • Gallego Olsina, Gerard Ion
  • Tsiamas, Ioannis
  • Ruiz Costa-jussà, Marta
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions., Javier Ferrando, Gerard I.Gállego and Ioannis Tsiamas are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00/AEI /10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Speech translation with foundation models and optimal transport: UPC at IWSLT23

  • Tsiamas, Ioannis
  • Gallego Olsina, Gerard Ion
  • Fonollosa, José A. R.
  • Ruiz Costa-jussà, Marta
This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023., The work done by Ioannis Tsiamas and Gerard I. Gállego was supported by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Self-supervised deep learning approaches to speaker recognition: A Ph.D. Thesis overview

  • Khan, Umair
  • Hernando Pericás, Francisco Javier
Recent advances in Deep Learning (DL) for speaker recognition have improved the performance but are constrained to the need of labels for the background data, which is difficult in prac- tice. In i-vector based speaker recognition, cosine (unsuper- vised) and PLDA (supervised) are the basic scoring techniques, with a big performance gap between the two. In this thesis we tried to fill this gap without using speaker labels in several ways. We applied Restricted Boltzmann Machine (RBM) vectors for the tasks of speaker clustering and tracking in TV broadcast shows. The experiments on AGORA database show that us- ing this approach we gain a relative improvement of 12% and 11% for speaker clustering and tracking tasks, respectively. We also applied DL techniques in order to increase the discrimina- tive power of i-vectors in speaker verification task, for which we have proposed the use of autoencoder in several ways, i.e., (1) as a pre-training for a Deep Neural Network (DNN), (2) as a near- est neighbor autoencoder for i-vectors, (3) as an average pooled nearest neighbor autoencoder. The experiments on VoxCeleb database show that we gain a relative improvement of 21%, 42% and 53%, using the three systems respectively. Finally we also proposed a self-supervised end-to-end speaker verification system. The architecture is based on a Convolutional Neural Network (CNN), trained as a siamese network with multiple branches. From the results we can see that our system shows comparable performance to a supervised baseline, This work was supported by the project PID2019-107579RBI00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Syntax-driven iterative expansion language models for controllable text generation

  • Casas Manzanares, Noé
  • Rodríguez Fonollosa, José Adrián
  • Ruiz Costa-Jussà, Marta
The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations., This work is partially supported by Lucy Software / United Language Group (ULG) and the Catalan Agency for Management of University and Research Grants (AGAUR) through an Industrial Ph.D. Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund through the postdoctoral senior grant Ramón y Cajal and by the Agencia Estatal de Investigación through the projects EUR2019-103819, PCIN2017-079 and PID2019-107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Mass-editing memory with attention in transformers: a cross-lingual exploration of knowledge

  • Tamayo Mela, Daniel
  • Gonzalez Aguirre, Aitor
  • Hernando Pericás, Francisco Javier
  • Villegas, Marta
Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT., This work has been promoted and financed by the Government of Catalonia through the Aina project, by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia Funded by EU– NextGenerationEU within the framework of the project ILENIA with reference 2022/TL22/00215337, 2022/TL22/00215336, 2022/TL22/00215335, 2022/TL22/00215334, as well as by the Spanish Ministerio de Ciencia e Innovación through the AdaVoice project (PID2019-107579RB-I00)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Sound-to-image translation through direct cross-modal connection using a convolutional–attention generative model

  • Areias Fanzeres, Leonardo
  • Nadeu Camprubí, Climent
  • Rodríguez Fonollosa, José Adrián
Sound plays a fundamental role in human perception, conveying information about events, objects, and spatial dynamics that may not be visually accessible. However, current technologies such as Acoustic Event Detection typically reduce complex soundscapes to textual labels, often failing to preserve their semantic richness. This limitation motivates the exploration of sound-to-image (S2I) translation as an alternative connection between audio and visual modalities. Unlike multimodal approaches guided by intermediary constraints during the learning process, we investigate S2I translation without class supervision, cluster-based alignment, or textual mediation, a paradigm we refer to as direct S2I translation. To the best of our knowledge, apart from our previous work, no prior study addresses S2I translation under this fully direct setting. We propose a convolutional–attention generative framework composed of an audio encoder and a densely connected GAN integrating self-attention and cross-attention mechanisms. The attention-based model is systematically compared with a purely convolutional baseline. Results show that introducing attention at early stages of the generator significantly improves translation performance, increasing the likelihood of producing interpretable and semantically coherent visual representations of sound. These findings indicate that attention strengthens semantic correspondence between audio and vision while preserving the fully direct nature of the translation process., This research was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under the PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00/AEI/10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

End-to-end speech translation with pre-trained models and adapters: UPC at IWSLT 2021

  • Gallego Olsina, Gerard Ion
  • Tsiamas, Ioannis
  • Escolano Peinado, Carlos
  • Rodríguez Fonollosa, José Adrián
  • Ruiz Costa-Jussà, Marta
This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuSTC test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation., This work was supported by the project ADAVOICE, PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 947657)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Automatic adaption of the step size in gradient descent training

  • Nogueiras Rodríguez, Albino
  • Nogueiras Marco, Ignasi
Selecting an appropriate step size is critical in Gradient Descent algorithms used to train Neural Networks for Deep Learning tasks. A small value of the step size leads to slow convergence, while a large value can lead to instability. In this paper, we present ACME, a gradient descent algorithm that automatically updates the step size to its optimum value. When combined with Adaptive Moment Estimation (ADAM) and/or Adaptive Sharpness Aware Minimization (ASAM), ACME achieves performances similar to them, but without the burden of carefully adjusting the learning rate or designing intricate learning rate scheduling policies. The code of ACME, ready to be used by itself or in combination with ADAM and/or ASAM, is available at https://github.com/ignasi00/acme.git., The contribution of Albino Nogueiras-Rodríguez to this paper was supported by the Spanish Government project PID2019-107579RB-I00 / AEI / 10.13039/501100011033. Ignasi Nogueiras-Marco acknowledges funding from “la Caixa” Foundation (ID 100,010,434) under agreement HR-18-0036, and from the Spanish Government through Grant PID2021-122893NB-C21, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Language modelling for speaker diarization in telephonic interviews

  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
  • Rodríguez Fonollosa, José Adrián
The aim of this paper is to investigate the benefit of combining both language and acoustic modelling for speaker diarization. Although conventional systems only use acoustic features, in some scenarios linguistic data contain high discriminative speaker information, even more reliable than the acoustic ones. In this study we analyze how an appropriate fusion of both kind of features is able to obtain good results in these cases. The proposed system is based on an iterative algorithm where a LSTM network is used as a speaker classifier. The network is fed with character-level word embeddings and a GMM based acoustic score created with the output labels from previous iterations. The presented algorithm has been evaluated in a Call-Center database, which is composed of telephone interview audios. The combination of acoustic features and linguistic content shows a 84.29% improvement in terms of a word-level DER as compared to a HMM/VB baseline system. The results of this study confirms that linguistic content can be efficiently used for some speaker recognition tasks., This work was partially supported by the Spanish Project DeepVoice (TEC2015-69266-P) and by the project PID2019-107579RBI00/ AEI /10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The TALP-UPC system description for WMT20 news translation task: multilingual adaptation for low resource MT

  • Escolano Peinado, Carlos
  • Ruiz Costa-Jussà, Marta
  • Rodríguez Fonollosa, José Adrián
In this article, we describe the TALP-UPC participation in the WMT20 news translation shared task for Tamil-English. Given the low amount of parallel training data, we resort to adapt the task to a multilingual system to benefit from the positive transfer from high resource languages. We use iterative backtranslation to fine-tune the system and benefit from the monolingual data available. In order to measure the effectivity of such methods, we compare our results to a bilingual baseline system., This work is supported in part by the Google Faculty Research Award 2019, the Spanish Ministerio de Ciencia e Innovacion, through the postdoctoral senior grant Ramon y Cajal and by the Agencia Estatal de Investigacion through the projects EUR2019-103819, PCIN-2017-079 and PID2019- 107579RB-I00 / AEI / 10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

From bilingual to multilingual neural-based machine translation by incremental training

  • Escolano Peinado, Carlos
  • Ruiz Costa-Jussà, Marta
  • Rodríguez Fonollosa, José Adrián
A common intermediate language representation in neural machine translation can be used to extend bilingual systems by incremental training. We propose a new architecture based on introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we can train multiple encoders and decoders for each language, sharing among them a common intermediate representation. Translation results on the low-resource tasks (Turkish-English and Kazakh-English tasks) show a BLEU improvement of up to 2.8 points. However, results on a larger dataset (Russian-English and Kazakh-English) show BLEU losses of a similar amount. While our system provides improvements only for the low-resource tasks in terms of translation quality, our system is capable of quickly deploying new language pairs without the need to retrain the rest of the system, which may be a game changer in some situations. Specifically, what is most relevant regarding our architecture is that it is capable of: reducing the number of production systems, with respect to the number of languages, from quadratic to linear; incrementally adding a new language to the system without retraining the languages already there; and allowing for translations from the new language to all the others present in the system., This work also is supported in part by the Google Faculty Research Awards and, also by, the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the projects EUR2019-103819, PCIN-2017-079 andPID2019-107579RB-I00., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

On the use of audio to improve dialogue policies

  • Roncel Diaz, Daniel
  • Costa, Federico
  • Hernando Pericás, Francisco Javier
With the significant progress of speech technologies, spoken goal-oriented dialogue systems are becoming increasingly popular. One of the main modules of a dialogue system is typically the dialogue policy, which is responsible for determining system actions. This component usually relies only on audio transcriptions, being strongly dependent on their quality and ignoring very important extralinguistic information embedded in the user’s speech. In this paper, we propose new architectures to add audio information by combining speech and text embeddings using a Double Multi-Head Attention component. Our experiments show that audio embedding-aware dialogue policies outperform text-based ones, particularly in noisy transcription scenarios, and that how text and audio embeddings are combined is crucial to improve performance. We obtained a 9.8% relative improvement in the User Request Score compared to an only-text-based dialogue system on the DSTC2 dataset., This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Spanish Ministerio de Ciencia e Innovación through the project AdaVoice PID2019-107579RB-I00. The second author is supported by a FI grant from the Catalan government., Award-winning, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

SegAugment: maximizing the utility of speech translation data with segmentation-based augmentations

  • Tsiamas, Ioannis
  • Rodríguez Fonollosa, José Adrián
  • Ruiz Costa-jussà, Marta
End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment obtains state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time., Work at UPC was supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)