Pasar al contenido principal

ARQUITECTURAS AVANZADAS DE APRENDIZAJE PROFUNDO APLICADAS AL PROCESADO DE VOZ, AUDIO Y LENGUAJE

PID2019-107579RB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2019
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POLITECNICA DE CATALUNYA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 28
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Real-time interpolation of global ionospheric maps by means of sparse representation

  • Yang, Heng
  • Monte-Moreno, Enric
  • Hernández-Pajares, Manuel
  • Roma-Dollase, David
In this paper, we propose a method for the generation of real-time global ionospheric map (RT-GIM) of vertical total electron content (VTEC) from GNSS measurements. The need for interpolation arises from the fact that the ionospheric pierce point (IPP) measurements from satellites to stations are not distributed uniformly over the ionosphere, leaving unfilled gaps at oceans or poles. The method we propose is based on using a high-quality historical database of post-processed GIMs that comprises more than two solar cycles, calculates the GIM by weighted superposition on a subset of the database with the compatible solar condition. The linear combination of GIMs in the database was obtained by minimizing a ℓ distance between VTEC measurements at the IPPs and the VTECs from the database, adding a ℓ penalization on the weights to assure a sparse solution. The process uses a Sun-fixed geomagnetic reference frame. This method uses the atomic decomposition/least absolute shrinkage and selection operator (LASSO), which will be denoted as atomic decomposition interpolator of GIMs (ADIGIM). As the computation is done in milliseconds, the interpolation is performed in real time. In this work, two products were developed, denoted as UADG and UARG, the UADG in real time and UARG with a latency of 24 h to benefit from the availability of a greater number of stations. The altimeter JASON3 VTEC measurements were used as reference. The quality of interpolated RT-GIMs from day 258 of 2019 to 155 of the year 2020 is compared with other RT/non-RT GIM products such as those from International GNSS Service (IGS), Centre National d’Etudes Spatiales (CNES), Chinese Academy of Sciences (CAS), Polytechnic University of Catalonia (UPC) and others. The RT ADIGIM performance proved to be better, nearly as good as the rapid or final GIMs computed retrospectively with delays of hours to days. Besides, the non-RT ADIGIM quality is as good or better than most GIM products. The oceanic regions have been included in the assessment which showed that ADIGIM interpolation gives the best estimation (referred to JASON3). The developed method, UADG, will constitute the next-generation UPC RT-GIM, and also UARG will improve the current product UQRG (the current UPC rapid GIM product computed retrospectively) due to its complementary information., This work has been partially supported by the Project PID2019-107579RB-I00 (MICINN). Also was partially supported by the 2017 SGR-0851 Grant of the Generalitat de Catalunya and by the EU Project 101007599 - PITHIA-NRF.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Forecast of the Global TEC by Nearest Neighbour Technique

  • Monte-Moreno, Enric
  • Yang, Heng
  • Hernández-Pajares, Manuel
We propose a method for Global Ionospheric Maps of Total Electron Content forecasting using the Nearest Neighbour method. The assumption is that in a database of global ionosphere maps spanning more than two solar cycles, one can select a set of past observations that have similar geomagnetic conditions to those of the current map. The assumption is that the current ionospheric condition can be expressed by a linear combination of conditions seen in the past. The average of these maps leads to common geomagnetic components being preserved and those not shared by several maps being reduced. The method is based on searching the historical database for the dates of the maps closest to the current map and using as a prediction the maps in the database that correspond to time shifts on the prediction horizons. In contrast to other methods of machine learning, the implementation only requires a distance computation and does not need a previous step of model training and adjustment for each prediction horizon. It also provides confidence intervals for the forecast. The method has been analyzed for two full years (2015 and 2018), for selected days of 2015 and 2018, i.e., two storm days and two non-storm days and the performance of the system has been compared with CODE (24-and 48-h forecast horizons)., This work was been partially supported by the project PID2019-107579RB-I00 (MICINN) and done in the context of PITHIA NRF EU project. The work of Heng Yang was also partially supported by Natural Science Foundation of Chongqing, China (No. cstc2021jcyj-msxmX0191), by the Science and Technology Research Program of Chongqing Municipal Education Commission of China (Grant No. KJQN202101414), and by the Cooperative Projects between Undergraduate Universities in Chongqing and Institutes affiliated with Chinese Academy of Sciences (No.HZ2021014).
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Forecast of the global TEC by nearest neighbour technique

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Yang, Heng
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
We propose a method for Global Ionospheric Maps of Total Electron Content forecasting using the Nearest Neighbour method. The assumption is that in a database of global ionosphere maps spanning more than two solar cycles, one can select a set of past observations that have similar geomagnetic conditions to those of the current map. The assumption is that the current ionospheric condition can be expressed by a linear combination of conditions seen in the past. The average of these maps leads to common geomagnetic components being preserved and those not shared by several maps being reduced. The method is based on searching the historical database for the dates of the maps closest to the current map and using as a prediction the maps in the database that correspond to time shifts on the prediction horizons. In contrast to other methods of machine learning, the implementation only requires a distance computation and does not need a previous step of model training and adjustment for each prediction horizon. It also provides confidence intervals for the forecast. The method has been analyzed for two full years (2015 and 2018), for selected days of 2015 and 2018, i.e., two storm days and two non-storm days and the performance of the system has been compared with CODE (24- and 48-h forecast horizons)., This work was been partially supported by the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and done in the context of PITHIA NRF EU project. The work of Heng Yang was also partially supported by Natural Science Foundation of Chongqing, China (No. cstc2021jcyj-msxmX0191) and by the Science and Technology Research Program of Chongqing Municipal Education Commission of China (Grant No. KJ202101414)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Promoting generalized cross-lingual question answering in few-resource scenarios via self-knowledge distillation, Fomentando la respuesta a preguntas cross-lingüística generalizada en escenarios con pocos recursos mediante auto-destilación del conocimiento

  • Carrino, Casimiro Pio
  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Gasco Sánchez, Luis
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
We address the challenge of Generalized Cross-Lingual Transfer (G-XLT) in extractive Question Answering, where question and context languages differ, a problem particularly difficult for low-resource languages. Working with only a thousand parallel QA samples, we combine cross-lingual sampling with self-knowledge distillation to regularize cross-lingual fine-tuning. We introduce the novel mean Average Precision at k (mAP@k) coefficient, which mitigates the negative impact of incorrect predictions during training and serves as a diagnostic tool providing early training guidance and reliable indicators of model learning. Evaluations on MLQA, XQuAD, and TyDiQA-GoldP datasets demonstrate that our approach consistently outperforms standard cross-entropy fine-tuning of the mBERT multilingual model. Our method represents a promising alternative to machine translation-based approaches, particularly valuable for low-resource languages where translation quality is poor, offering an efficient solution for cross-lingual transfer in data-scarce settings., This work was supported by the project PID2019-107579RB-I00 (MICINN)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Global teleconnections of rapid ionospheric fluctuations as observed via GNSS measurements

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Yang, Heng
The rapid fluctuations in the electron content of the ionosphere can significantly affect the performance and reliability of communication and GNSS systems. These fluctuations are closely linked to space weather and can result in simultaneous changes in the total electron content of the ionosphere across multiple regions globally. We show the existence of statistically significant synchronous connections between geographical locations when observing fluctuations in electron content. The two findings concerning rapid fluctuations in the electron content of the ionosphere are: a) Evidence of synchronous teleconnections in rapid local electron content variations across a wide range of geographical regions of the globe, and b) The influence of the solar cycle on the geographic distribution of teleconnections is relatively minor, primarily impacting the frequency of teleconnections per unit of time. We characterize teleconnection patterns for different space weather variables throughout the 2016-2022 study period, encompassing the descending and ascending phases of solar cycles 24 and 25. Previous research has mainly focused on the impact of ionospheric irregularities in limited regions, weather conditions, and short time periods, while our findings indicate the occurrence of this phenomenon globally and during both high and low space weather activity., The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191, and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423. This work was supported by the project PID2019-107579RB-I00 (MICINN). The work has been performed coinciding with the execution of the PITHIA-NRF H2020 project (H2020-INFRAIA-2018-2020101007599)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Language modelling for speaker diarization in telephonic interviews

  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
The aim of this paper is to investigate the benefit of combining both language and acoustic modelling for speaker diarization. Although conventional systems only use acoustic features, in some scenarios linguistic data contain high discriminative speaker information, even more reliable than the acoustic ones. In this study we analyze how an appropriate fusion of both kind of features is able to obtain good results in these cases. The proposed system is based on an iterative algorithm where a LSTM network is used as a speaker classifier. The network is fed with character-level word embeddings and a GMM based acoustic score created with the output labels from previous iterations. The presented algorithm has been evaluated in a Call-Center database, which is composed of telephone interview audios. The combination of acoustic features and linguistic content shows a 84.29% improvement in terms of a word-level DER as compared to a HMM/VB baseline system. The results of this study confirms that linguistic content can be efficiently used for some speaker recognition tasks., This work was partially supported by the Spanish Project DeepVoice (TEC2015-69266-P) and by the project PID2019-107579RBI00/ AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Self attention networks in speaker recognition

  • Safari, Pooyan
  • India Massana, Miquel Àngel|||0000-0002-3107-3662
  • Hernando Pericás, Francisco Javier|||0000-0002-1730-8154
Recently, there has been a significant surge of interest in Self-Attention Networks (SANs) based on the Transformer architecture. This can be attributed to their notable ability for parallelization and their impressive performance across various Natural Language Processing applications. On the other hand, the utilization of large-scale, multi-purpose language models trained through self-supervision is progressively more prevalent, for tasks like speech recognition. In this context, the pre-trained model, which has been trained on extensive speech data, can be fine-tuned for particular downstream tasks like speaker verification. These massive models typically rely on SANs as their foundational architecture. Therefore, studying the potential capabilities and training challenges of such models is of utmost importance for the future generation of speaker verification systems. In this direction, we propose a speaker embedding extractor based on SANs to obtain a discriminative speaker representation given non-fixed length speech utterances. With the advancements suggested in this work, we could achieve up to 41% relative performance improvement in terms of EER compared to the naive SAN which was proposed in our previous work. Moreover, we empirically show the training instability in such architectures in terms of rank collapse and further investigate the potential solutions to alleviate this shortcoming., This work was supported by the Project PID2019-107579RB-I00/AEI/10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Extensive study on the underlying gender bias in contextualized word embeddings

  • Basta, Christine Raouf Saad
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Casas Manzanares, Noé
Gender bias is affecting many natural language processing applications. While we are still far from proposing debiasing methods that will solve the problem, we are making progress analyzing the impact of this bias in current algorithms. This paper provides an extensive study of the underlying gender bias in popular contextualized word embeddings. Our study provides an insightful analysis of evaluation measures applied to several English data domains and the layers of the contextualized word embeddings. It is also adapted and extended to the Spanish language. Our study points out the advantages and limitations of the various evaluation measures that we are using and aims to standardize the evaluation of gender bias in contextualized word embeddings., This work is supported in part by the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship and the Industrial PhD Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the Projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

A genetic programming approach for estimating economic sentiment in the Baltic countries and the European Union

  • Claveria González, Oscar
  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Torra Porras, Salvador
In this study, we introduce a sentiment construction method based on the evolution of survey-based indicators. We make use of genetic algorithms to evolve qualitative expectations in order to generate country-specific empirical economic sentiment indicators in the three Baltic republics and the European Union. First, for each country we search for the non-linear combination of firms’ and households’ expectations that minimises a fitness function. Second, we compute the frequency with which each survey expectation appears in the evolved indicators and examine the lag structure per variable selected by the algorithm. The industry survey indicator with the highest predictive performance are production expectations, while in the case of the consumer survey the distribution between variables is multi-modal. Third, we evaluate the out-of-sample predictive performance of the generated indicators, obtaining more accurate estimates of year-on-year GDP growth rates than with the scaled industrial and consumer confidence indicators. Finally, we use non-linear constrained optimisation to combine the evolved expectations of firms and consumers and generate aggregate expectations of of year-on-year GDP growth. We find that, in most cases, aggregate expectations outperform recursive autoregressive predictions of economic growth., This work was supported by the Spanish Ministry of Science and Innovation under Grant PID2019-107579RB-I00., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Power-law distribution of rapid ionosphere electron content fluctuations via GNSS measurements

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Yang, Heng
The ionosphere is a dynamic region of the Earth’s upper atmosphere that exhibits various fluctuations in electron density. These fluctuations can be estimated using measurements from the global navigation satellite system (GNSS). In this article, our focus is on characterizing the probability distribution of total electron content (TEC) fluctuations and their temporal duration. The findings of this study, conducted over half a solar cycle and involving more than 100 stations distributed worldwide, demonstrate that the distribution of rapid ionosphere electron content fluctuations follows a power-law behavior, both in amplitude and duration. This explains the occurrence of occasional extreme values of amplitude and duration. In addition, despite significant differences in space weather conditions during the solar cycle, including geomagnetic storms, the power-law distributions of both the rate of change in TEC index amplitude and duration of bursts of activity, exhibit the same shape parameter ( α ), which is similar to the values observed during less active periods. Another finding is the statistical independence of amplitude and duration. The observed power-law distribution and its temporal and spatial dependencies provide practical insights for developing robust models and algorithms for ionospheric impact mitigation in GNSS positioning and navigation systems. Moreover, these findings have implications for space weather monitoring and forecasting, contributing to the assessment of ionospheric disturbances and for a wide range of technological systems, including telecommunication networks among GNSS., This work was supported in part by Project PID2019- 107579RB-I00/AEI/10.13039/501100011033 and in part by the PITHIA-NRF H2020 Project under Grant H2020-INFRAIA-2018-2020101007599. The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191; and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Real-time interpolation of global ionospheric maps by means of sparse representation

  • Yang, Heng
  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Roma Dollase, David|||0000-0003-4450-6073
In this paper, we propose a method for the generation of real-time global ionospheric map (RT-GIM) of vertical total electron content (VTEC) from GNSS measurements. The need for interpolation arises from the fact that the ionospheric pierce point (IPP) measurements from satellites to stations are not distributed uniformly over the ionosphere, leaving unfilled gaps at oceans or poles. The method we propose is based on using a high-quality historical database of post-processed GIMs that comprises more than two solar cycles, calculates the GIM by weighted superposition on a subset of the database with the compatible solar condition. The linear combination of GIMs in the database was obtained by minimizing a ℓ2 distance between VTEC measurements at the IPPs and the VTECs from the database, adding a ℓ1 penalization on the weights to assure a sparse solution. The process uses a Sun-fixed geomagnetic reference frame. This method uses the atomic decomposition/least absolute shrinkage and selection operator (LASSO), which will be denoted as atomic decomposition interpolator of GIMs (ADIGIM). As the computation is done in milliseconds, the interpolation is performed in real time. In this work, two products were developed, denoted as UADG and UARG, the UADG in real time and UARG with a latency of 24 h to benefit from the availability of a greater number of stations. The altimeter JASON3 VTEC measurements were used as reference. The quality of interpolated RT-GIMs from day 258 of 2019 to 155 of the year 2020 is compared with other RT/non-RT GIM products such as those from International GNSS Service (IGS), Centre National d’Etudes Spatiales (CNES), Chinese Academy of Sciences (CAS), Polytechnic University of Catalonia (UPC) and others. The RT ADIGIM performance proved to be better, nearly as good as the rapid or final GIMs computed retrospectively with delays of hours to days. Besides, the non-RT ADIGIM quality is as good or better than most GIM products. The oceanic regions have been included in the assessment which showed that ADIGIM interpolation gives the best estimation (referred to JASON3). The developed method, UADG, will constitute the next-generation UPC RT-GIM, and also UARG will improve the current product UQRG (the current UPC rapid GIM product computed retrospectively) due to its complementary information., This work has been partially supported by the Project PID2019-107579RB-I00 (MICINN). Also was partially supported by the 2017 SGR-0851 Grant of the Generalitat de Catalunya and by the EU Project 101007599 - PITHIA-NRF., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Estimation of polar depletion regions by VTEC contrast and watershed enhancing

  • Monte Moreno, Enrique|||0000-0002-4907-0494
  • Hernández Pajares, Manuel|||0000-0002-9687-5850
  • Lyu, Haixia
  • Yang, Heng
  • Aragon-Angel, Angela
This article presents a method for determining near-Pole ionization depletion regions and troughs from global navigation satellite system (GNSS) vertical total electron content (VTEC) maps. To define the regions, we use Image processing tools, considering the VTEC maps as gray-level images. In this work, watershed segmentation is applied to determine the edges of regions, based on the flow of areas between the lowest ionization zones (<1%) and the medium ionization regions (>10%). In order to enhance the contrast between low-ionization regions and those with a higher relative ionization, the histogram of the maps is equalized, which increases the contrast of the VTEC maps, which is the distance between low-ionization regions and those with a higher relative ionization are increased. The use of percentile thresholds makes this method independent of hypotheses about the generation of depletions, their morphology, or the number of depletion regions that can occur on a map. We test the performance by comparing the depletion regions identified in eight events reported in the literature under different conditions of the ionosphere. The events used as reference were measured by satellites, radar, and GNSS. The method that we propose allows for real-time processing of GNSS VTEC maps. The morphology of the detected regions matches the shape that is reported in the literature., This work was supported by the project PID2019- 107579RB-I00/AEI/10.13039/501100011033, and ECfunded activity PolarGIMs2, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Sound-to-image translation through direct cross-modal connection using a convolutional–attention generative model

  • Areias Fanzeres, Leonardo|||0000-0003-1179-6596
  • Nadeu Camprubí, Climent|||0000-0002-5863-0983
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
Sound plays a fundamental role in human perception, conveying information about events, objects, and spatial dynamics that may not be visually accessible. However, current technologies such as Acoustic Event Detection typically reduce complex soundscapes to textual labels, often failing to preserve their semantic richness. This limitation motivates the exploration of sound-to-image (S2I) translation as an alternative connection between audio and visual modalities. Unlike multimodal approaches guided by intermediary constraints during the learning process, we investigate S2I translation without class supervision, cluster-based alignment, or textual mediation, a paradigm we refer to as direct S2I translation. To the best of our knowledge, apart from our previous work, no prior study addresses S2I translation under this fully direct setting. We propose a convolutional–attention generative framework composed of an audio encoder and a densely connected GAN integrating self-attention and cross-attention mechanisms. The attention-based model is systematically compared with a purely convolutional baseline. Results show that introducing attention at early stages of the generator significantly improves translation performance, increasing the likelihood of producing interpretable and semantically coherent visual representations of sound. These findings indicate that attention strengthens semantic correspondence between audio and vision while preserving the fully direct nature of the translation process., This research was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under the PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00/AEI/10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

From bilingual to multilingual neural-based machine translation by incremental training

  • Escolano Peinado, Carlos|||0000-0001-6657-673X
  • Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
  • Rodríguez Fonollosa, José Adrián|||0000-0001-9513-7939
A common intermediate language representation in neural machine translation can be used to extend bilingual systems by incremental training. We propose a new architecture based on introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we can train multiple encoders and decoders for each language, sharing among them a common intermediate representation. Translation results on the low-resource tasks (Turkish-English and Kazakh-English tasks) show a BLEU improvement of up to 2.8 points. However, results on a larger dataset (Russian-English and Kazakh-English) show BLEU losses of a similar amount. While our system provides improvements only for the low-resource tasks in terms of translation quality, our system is capable of quickly deploying new language pairs without the need to retrain the rest of the system, which may be a game changer in some situations. Specifically, what is most relevant regarding our architecture is that it is capable of: reducing the number of production systems, with respect to the number of languages, from quadratic to linear; incrementally adding a new language to the system without retraining the languages already there; and allowing for translations from the new language to all the others present in the system., This work also is supported in part by the Google Faculty Research Awards and, also by, the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the projects EUR2019-103819, PCIN-2017-079 andPID2019-107579RB-I00., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Sound-to-imagination: an exploratory study on cross-modal translation using diverse audiovisual data

  • Areias Fanzeres, Leonardo|||0000-0003-1179-6596
  • Nadeu Camprubí, Climent|||0000-0002-5863-0983
The motivation of our research is to explore the possibilities of automatic sound-to-image (S2I) translation for enabling a human receiver to visually infer occurrences of sound-related events. We expect the computer to ‘imagine’ scenes from captured sounds, generating original images that depict the sound-emitting sources. Previous studies on similar topics opted for simplified approaches using data with low content diversity and/or supervision/self-supervision for training. In contrast, our approach involves performing S2I translation using thousands of distinct and unknown scenes, using sound class annotations solely for data preparation, just enough to ensure aural–visual semantic coherence. To model the translator, we employ an audio encoder and a conditional generative adversarial network (GAN) with a deep densely connected generator. Furthermore, we present a solution using informativity classifiers for quantitatively evaluating the generated images. This allows us to analyze the influence of network-bottleneck variation on the translation process, highlighting a potential trade-off between informativity and pixel space convergence. Despite the complexity of the specified S2I translation task, we were able to generalize the model enough to obtain more than 14%, on average, of interpretable and semantically coherent images translated from unknown sounds., The present work was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI), project PID2019-107579RBI00/AEI/10.13039/501100011033., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Self attention networks in speaker recognition

  • Safari, Pooyan
  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
Recently, there has been a significant surge of interest in Self-Attention Networks (SANs) based on the Transformer architecture. This can be attributed to their notable ability for parallelization and their impressive performance across various Natural Language Processing applications. On the other hand, the utilization of large-scale, multi-purpose language models trained through self-supervision is progressively more prevalent, for tasks like speech recognition. In this context, the pre-trained model, which has been trained on extensive speech data, can be fine-tuned for particular downstream tasks like speaker verification. These massive models typically rely on SANs as their foundational architecture. Therefore, studying the potential capabilities and training challenges of such models is of utmost importance for the future generation of speaker verification systems. In this direction, we propose a speaker embedding extractor based on SANs to obtain a discriminative speaker representation given non-fixed length speech utterances. With the advancements suggested in this work, we could achieve up to 41% relative performance improvement in terms of EER compared to the naive SAN which was proposed in our previous work. Moreover, we empirically show the training instability in such architectures in terms of rank collapse and further investigate the potential solutions to alleviate this shortcoming., This work was supported by the Project PID2019-107579RB-I00/AEI/10.13039/501100011033, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Estimation of polar depletion regions by VTEC contrast and watershed enhancing

  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Lyu, Haixia
  • Yang, Heng
  • Aragon-Angel, Angela
This article presents a method for determining near-Pole ionization depletion regions and troughs from global navigation satellite system (GNSS) vertical total electron content (VTEC) maps. To define the regions, we use Image processing tools, considering the VTEC maps as gray-level images. In this work, watershed segmentation is applied to determine the edges of regions, based on the flow of areas between the lowest ionization zones (<1%) and the medium ionization regions (>10%). In order to enhance the contrast between low-ionization regions and those with a higher relative ionization, the histogram of the maps is equalized, which increases the contrast of the VTEC maps, which is the distance between low-ionization regions and those with a higher relative ionization are increased. The use of percentile thresholds makes this method independent of hypotheses about the generation of depletions, their morphology, or the number of depletion regions that can occur on a map. We test the performance by comparing the depletion regions identified in eight events reported in the literature under different conditions of the ionosphere. The events used as reference were measured by satellites, radar, and GNSS. The method that we propose allows for real-time processing of GNSS VTEC maps. The morphology of the detected regions matches the shape that is reported in the literature., This work was supported by the project PID2019- 107579RB-I00/AEI/10.13039/501100011033, and ECfunded activity PolarGIMs2, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Promoting generalized cross-lingual question answering in few-resource scenarios via self-knowledge distillation, Fomentando la respuesta a preguntas cross-lingüística generalizada en escenarios con pocos recursos mediante auto-destilación del conocimiento

  • Carrino, Casimiro Pio
  • Escolano Peinado, Carlos
  • Gasco Sánchez, Luis
  • Rodríguez Fonollosa, José Adrián
We address the challenge of Generalized Cross-Lingual Transfer (G-XLT) in extractive Question Answering, where question and context languages differ, a problem particularly difficult for low-resource languages. Working with only a thousand parallel QA samples, we combine cross-lingual sampling with self-knowledge distillation to regularize cross-lingual fine-tuning. We introduce the novel mean Average Precision at k (mAP@k) coefficient, which mitigates the negative impact of incorrect predictions during training and serves as a diagnostic tool providing early training guidance and reliable indicators of model learning. Evaluations on MLQA, XQuAD, and TyDiQA-GoldP datasets demonstrate that our approach consistently outperforms standard cross-entropy fine-tuning of the mBERT multilingual model. Our method represents a promising alternative to machine translation-based approaches, particularly valuable for low-resource languages where translation quality is poor, offering an efficient solution for cross-lingual transfer in data-scarce settings., This work was supported by the project PID2019-107579RB-I00 (MICINN)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A genetic programming approach for estimating economic sentiment in the Baltic countries and the European Union

  • Claveria González, Oscar
  • Monte Moreno, Enrique
  • Torra Porras, Salvador
In this study, we introduce a sentiment construction method based on the evolution of survey-based indicators. We make use of genetic algorithms to evolve qualitative expectations in order to generate country-specific empirical economic sentiment indicators in the three Baltic republics and the European Union. First, for each country we search for the non-linear combination of firms’ and households’ expectations that minimises a fitness function. Second, we compute the frequency with which each survey expectation appears in the evolved indicators and examine the lag structure per variable selected by the algorithm. The industry survey indicator with the highest predictive performance are production expectations, while in the case of the consumer survey the distribution between variables is multi-modal. Third, we evaluate the out-of-sample predictive performance of the generated indicators, obtaining more accurate estimates of year-on-year GDP growth rates than with the scaled industrial and consumer confidence indicators. Finally, we use non-linear constrained optimisation to combine the evolved expectations of firms and consumers and generate aggregate expectations of of year-on-year GDP growth. We find that, in most cases, aggregate expectations outperform recursive autoregressive predictions of economic growth., This work was supported by the Spanish Ministry of Science and Innovation under Grant PID2019-107579RB-I00., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Real-time interpolation of global ionospheric maps by means of sparse representation

  • Yang, Heng
  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Roma Dollase, David
In this paper, we propose a method for the generation of real-time global ionospheric map (RT-GIM) of vertical total electron content (VTEC) from GNSS measurements. The need for interpolation arises from the fact that the ionospheric pierce point (IPP) measurements from satellites to stations are not distributed uniformly over the ionosphere, leaving unfilled gaps at oceans or poles. The method we propose is based on using a high-quality historical database of post-processed GIMs that comprises more than two solar cycles, calculates the GIM by weighted superposition on a subset of the database with the compatible solar condition. The linear combination of GIMs in the database was obtained by minimizing a ℓ2 distance between VTEC measurements at the IPPs and the VTECs from the database, adding a ℓ1 penalization on the weights to assure a sparse solution. The process uses a Sun-fixed geomagnetic reference frame. This method uses the atomic decomposition/least absolute shrinkage and selection operator (LASSO), which will be denoted as atomic decomposition interpolator of GIMs (ADIGIM). As the computation is done in milliseconds, the interpolation is performed in real time. In this work, two products were developed, denoted as UADG and UARG, the UADG in real time and UARG with a latency of 24 h to benefit from the availability of a greater number of stations. The altimeter JASON3 VTEC measurements were used as reference. The quality of interpolated RT-GIMs from day 258 of 2019 to 155 of the year 2020 is compared with other RT/non-RT GIM products such as those from International GNSS Service (IGS), Centre National d’Etudes Spatiales (CNES), Chinese Academy of Sciences (CAS), Polytechnic University of Catalonia (UPC) and others. The RT ADIGIM performance proved to be better, nearly as good as the rapid or final GIMs computed retrospectively with delays of hours to days. Besides, the non-RT ADIGIM quality is as good or better than most GIM products. The oceanic regions have been included in the assessment which showed that ADIGIM interpolation gives the best estimation (referred to JASON3). The developed method, UADG, will constitute the next-generation UPC RT-GIM, and also UARG will improve the current product UQRG (the current UPC rapid GIM product computed retrospectively) due to its complementary information., This work has been partially supported by the Project PID2019-107579RB-I00 (MICINN). Also was partially supported by the 2017 SGR-0851 Grant of the Generalitat de Catalunya and by the EU Project 101007599 - PITHIA-NRF., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Global teleconnections of rapid ionospheric fluctuations as observed via GNSS measurements

  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Yang, Heng
The rapid fluctuations in the electron content of the ionosphere can significantly affect the performance and reliability of communication and GNSS systems. These fluctuations are closely linked to space weather and can result in simultaneous changes in the total electron content of the ionosphere across multiple regions globally. We show the existence of statistically significant synchronous connections between geographical locations when observing fluctuations in electron content. The two findings concerning rapid fluctuations in the electron content of the ionosphere are: a) Evidence of synchronous teleconnections in rapid local electron content variations across a wide range of geographical regions of the globe, and b) The influence of the solar cycle on the geographic distribution of teleconnections is relatively minor, primarily impacting the frequency of teleconnections per unit of time. We characterize teleconnection patterns for different space weather variables throughout the 2016-2022 study period, encompassing the descending and ascending phases of solar cycles 24 and 25. Previous research has mainly focused on the impact of ionospheric irregularities in limited regions, weather conditions, and short time periods, while our findings indicate the occurrence of this phenomenon globally and during both high and low space weather activity., The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191, and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423. This work was supported by the project PID2019-107579RB-I00 (MICINN). The work has been performed coinciding with the execution of the PITHIA-NRF H2020 project (H2020-INFRAIA-2018-2020101007599)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Power-law distribution of rapid ionosphere electron content fluctuations via GNSS measurements

  • Monte Moreno, Enrique
  • Hernández Pajares, Manuel
  • Yang, Heng
The ionosphere is a dynamic region of the Earth’s upper atmosphere that exhibits various fluctuations in electron density. These fluctuations can be estimated using measurements from the global navigation satellite system (GNSS). In this article, our focus is on characterizing the probability distribution of total electron content (TEC) fluctuations and their temporal duration. The findings of this study, conducted over half a solar cycle and involving more than 100 stations distributed worldwide, demonstrate that the distribution of rapid ionosphere electron content fluctuations follows a power-law behavior, both in amplitude and duration. This explains the occurrence of occasional extreme values of amplitude and duration. In addition, despite significant differences in space weather conditions during the solar cycle, including geomagnetic storms, the power-law distributions of both the rate of change in TEC index amplitude and duration of bursts of activity, exhibit the same shape parameter ( α ), which is similar to the values observed during less active periods. Another finding is the statistical independence of amplitude and duration. The observed power-law distribution and its temporal and spatial dependencies provide practical insights for developing robust models and algorithms for ionospheric impact mitigation in GNSS positioning and navigation systems. Moreover, these findings have implications for space weather monitoring and forecasting, contributing to the assessment of ionospheric disturbances and for a wide range of technological systems, including telecommunication networks among GNSS., This work was supported in part by Project PID2019- 107579RB-I00/AEI/10.13039/501100011033 and in part by the PITHIA-NRF H2020 Project under Grant H2020-INFRAIA-2018-2020101007599. The work of Heng Yang was supported in part by the Natural Science Foundation of Chongqing, China, under Grant cstc2021jcyj-msxmX0191; and in part by the Science and Technology Research Program of Chongqing Municipal Education Commission of China under Grant KJQN202101414 and Grant KJQN202201423., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Sound-to-imagination: an exploratory study on cross-modal translation using diverse audiovisual data

  • Areias Fanzeres, Leonardo
  • Nadeu Camprubí, Climent
The motivation of our research is to explore the possibilities of automatic sound-to-image (S2I) translation for enabling a human receiver to visually infer occurrences of sound-related events. We expect the computer to ‘imagine’ scenes from captured sounds, generating original images that depict the sound-emitting sources. Previous studies on similar topics opted for simplified approaches using data with low content diversity and/or supervision/self-supervision for training. In contrast, our approach involves performing S2I translation using thousands of distinct and unknown scenes, using sound class annotations solely for data preparation, just enough to ensure aural–visual semantic coherence. To model the translator, we employ an audio encoder and a conditional generative adversarial network (GAN) with a deep densely connected generator. Furthermore, we present a solution using informativity classifiers for quantitatively evaluating the generated images. This allows us to analyze the influence of network-bottleneck variation on the translation process, highlighting a potential trade-off between informativity and pixel space convergence. Despite the complexity of the specified S2I translation task, we were able to generalize the model enough to obtain more than 14%, on average, of interpretable and semantically coherent images translated from unknown sounds., The present work was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI), project PID2019-107579RBI00/AEI/10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Extensive study on the underlying gender bias in contextualized word embeddings

  • Basta, Christine Raouf Saad
  • Ruiz Costa-Jussà, Marta
  • Casas Manzanares, Noé
Gender bias is affecting many natural language processing applications. While we are still far from proposing debiasing methods that will solve the problem, we are making progress analyzing the impact of this bias in current algorithms. This paper provides an extensive study of the underlying gender bias in popular contextualized word embeddings. Our study provides an insightful analysis of evaluation measures applied to several English data domains and the layers of the contextualized word embeddings. It is also adapted and extended to the Spanish language. Our study points out the advantages and limitations of the various evaluation measures that we are using and aims to standardize the evaluation of gender bias in contextualized word embeddings., This work is supported in part by the Catalan Agency for Management of University and Research Grants (AGAUR) through the FI PhD Scholarship and the Industrial PhD Grant. This work also is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the Projects EUR2019-103819, PCIN-2017-079 and PID2019-107579RB-I00., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Forecast of the global TEC by nearest neighbour technique

  • Monte Moreno, Enrique
  • Yang, Heng
  • Hernández Pajares, Manuel
We propose a method for Global Ionospheric Maps of Total Electron Content forecasting using the Nearest Neighbour method. The assumption is that in a database of global ionosphere maps spanning more than two solar cycles, one can select a set of past observations that have similar geomagnetic conditions to those of the current map. The assumption is that the current ionospheric condition can be expressed by a linear combination of conditions seen in the past. The average of these maps leads to common geomagnetic components being preserved and those not shared by several maps being reduced. The method is based on searching the historical database for the dates of the maps closest to the current map and using as a prediction the maps in the database that correspond to time shifts on the prediction horizons. In contrast to other methods of machine learning, the implementation only requires a distance computation and does not need a previous step of model training and adjustment for each prediction horizon. It also provides confidence intervals for the forecast. The method has been analyzed for two full years (2015 and 2018), for selected days of 2015 and 2018, i.e., two storm days and two non-storm days and the performance of the system has been compared with CODE (24- and 48-h forecast horizons)., This work was been partially supported by the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033 and done in the context of PITHIA NRF EU project. The work of Heng Yang was also partially supported by Natural Science Foundation of Chongqing, China (No. cstc2021jcyj-msxmX0191) and by the Science and Technology Research Program of Chongqing Municipal Education Commission of China (Grant No. KJ202101414)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Sound-to-image translation through direct cross-modal connection using a convolutional–attention generative model

  • Areias Fanzeres, Leonardo
  • Nadeu Camprubí, Climent
  • Rodríguez Fonollosa, José Adrián
Sound plays a fundamental role in human perception, conveying information about events, objects, and spatial dynamics that may not be visually accessible. However, current technologies such as Acoustic Event Detection typically reduce complex soundscapes to textual labels, often failing to preserve their semantic richness. This limitation motivates the exploration of sound-to-image (S2I) translation as an alternative connection between audio and visual modalities. Unlike multimodal approaches guided by intermediary constraints during the learning process, we investigate S2I translation without class supervision, cluster-based alignment, or textual mediation, a paradigm we refer to as direct S2I translation. To the best of our knowledge, apart from our previous work, no prior study addresses S2I translation under this fully direct setting. We propose a convolutional–attention generative framework composed of an audio encoder and a densely connected GAN integrating self-attention and cross-attention mechanisms. The attention-based model is systematically compared with a purely convolutional baseline. Results show that introducing attention at early stages of the generator significantly improves translation performance, increasing the likelihood of producing interpretable and semantically coherent visual representations of sound. These findings indicate that attention strengthens semantic correspondence between audio and vision while preserving the fully direct nature of the translation process., This research was supported in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under the PhD grant 200884/2015-8. Also, the work was partly supported by the Spanish State Research Agency (AEI) project PID2019-107579RB-I00/AEI/10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Language modelling for speaker diarization in telephonic interviews

  • India Massana, Miquel Àngel
  • Hernando Pericás, Francisco Javier
  • Rodríguez Fonollosa, José Adrián
The aim of this paper is to investigate the benefit of combining both language and acoustic modelling for speaker diarization. Although conventional systems only use acoustic features, in some scenarios linguistic data contain high discriminative speaker information, even more reliable than the acoustic ones. In this study we analyze how an appropriate fusion of both kind of features is able to obtain good results in these cases. The proposed system is based on an iterative algorithm where a LSTM network is used as a speaker classifier. The network is fed with character-level word embeddings and a GMM based acoustic score created with the output labels from previous iterations. The presented algorithm has been evaluated in a Call-Center database, which is composed of telephone interview audios. The combination of acoustic features and linguistic content shows a 84.29% improvement in terms of a word-level DER as compared to a HMM/VB baseline system. The results of this study confirms that linguistic content can be efficiently used for some speaker recognition tasks., This work was partially supported by the Spanish Project DeepVoice (TEC2015-69266-P) and by the project PID2019-107579RBI00/ AEI /10.13039/501100011033., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

From bilingual to multilingual neural-based machine translation by incremental training

  • Escolano Peinado, Carlos
  • Ruiz Costa-Jussà, Marta
  • Rodríguez Fonollosa, José Adrián
A common intermediate language representation in neural machine translation can be used to extend bilingual systems by incremental training. We propose a new architecture based on introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we can train multiple encoders and decoders for each language, sharing among them a common intermediate representation. Translation results on the low-resource tasks (Turkish-English and Kazakh-English tasks) show a BLEU improvement of up to 2.8 points. However, results on a larger dataset (Russian-English and Kazakh-English) show BLEU losses of a similar amount. While our system provides improvements only for the low-resource tasks in terms of translation quality, our system is capable of quickly deploying new language pairs without the need to retrain the rest of the system, which may be a game changer in some situations. Specifically, what is most relevant regarding our architecture is that it is capable of: reducing the number of production systems, with respect to the number of languages, from quadratic to linear; incrementally adding a new language to the system without retraining the languages already there; and allowing for translations from the new language to all the others present in the system., This work also is supported in part by the Google Faculty Research Awards and, also by, the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund, the Agencia Estatal de Investigación through the postdoctoral senior grant Ramón y Cajal and the projects EUR2019-103819, PCIN-2017-079 andPID2019-107579RB-I00., Peer Reviewed, Postprint (published version)