Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 22
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

On the evolution of syntactic information encoded by BERT’s contextualized representations

  • Pérez-Mayos, Laura
  • Carlini, Roberto
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), celebrat del 19 al 23 d'abril de 2021 de manera virtual., The adaptation of pretrained language models to solve supervised tasks has become a baseline in NLP, and many recent works have focused on studying how linguistic information is encoded in the pretrained sentence representations. Among other information, it has been shown that entire syntax trees are implicitly embedded in the geometry of such models. As these models are often fine-tuned, it becomes increasingly important to understand how the encoded knowledge evolves along the fine-tuning. In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six different tasks, covering all levels of the linguistic structure. Experimental results show that the encoded syntactic information is forgotten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semantics-related tasks) in different ways along the finetuning process depending on the task., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Assessing the syntactic capabilities of transformer-based multilingual language models

  • Pérez-Mayos, Laura
  • Táboas García, Alba
  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, celebrat de l'1 al 6 d'agost de 2021 de manera virtual., Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of crosslingual transfer tasks. However, it remains unknown whether the optimization for different languages conditions the capacity of the models to generalize over syntactic structures, and how languages with syntactic phenomena of different complexity are affected. In this work, we explore the syntactic generalization capabilities of the monolingual and multilingual versions of BERT and RoBERTa. More specifically, we evaluate the syntactic generalization potential of the models on English and Spanish tests, comparing the syntactic abilities of monolingual and multilingual models on the same language (English), and of multilingual models on two different languages (English and Spanish). For English, we use the available SyntaxGym test suite; for Spanish, we introduce SyntaxGymES, a novel ensemble of targeted syntactic tests in Spanish, designed to evaluate the syntactic generalization capabilities of language models through the SyntaxGym online platform., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Multilingual extraction and categorization of lexical collocations with graph-aware transformers

  • Espinosa-Anke, Luis
  • Shvets, Alexander
  • Mohammadshahi, Alireza
  • Henderson, James
  • Wanner, Leo
Comunicació presentada a 11th Joint Conference on Lexical and Computational Semantics (SEM 2022), celebrat del 14 al 15 de juliol de 2022 a Seattle, Estats Units., Recognizing and categorizing lexical collocations in context is useful for language learning, dictionary compilation and downstream NLP. However, it is a challenging task due to the varying degrees of frozenness lexical collocations exhibit. In this paper, we put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French., The work by Alexander Shvets and Leo Wanner has been supported by the European Commission in the context of the Horizon 2020 Research Program under the grant numbers 825079 and 870930. Alireza Mohammadshahi is supported by the Swiss National Science Foundation (grant number CRSII5- 180320).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

CollFrEn: rich bilingual English–French collocation resource

  • Fisas Elizalde, Beatriz
  • Espinosa-Anke, Luis
  • Codina Filbà, Joan
  • Wanner, Leo
Comunicació presentada a: Joint Workshop on Multiword Expressions and Electronic Lexicons celebrat el 13 de desembre de 2020 de manera virtual., Collocations in the sense of idiosyncratic lexical co-occurrences of two syntactically bound words traditionally pose a challenge to language learners and many Natural Language Processing (NLP) applications alike. Reliable ground truth (i.e., ideally manually compiled) resources are thus of high value. We present a manually compiled bilingual English–French collocation resource with 7,480 collocations in English and 6,733 in French. Each collocation is enriched with information that facilitates its downstream exploitation in NLP tasks such as machine translation, word sense disambiguation, natural language generation, relation classification, and so forth. Our proposed enrichment covers: the semantic category of the collocation (its lexical function), its vector space representation (for each individual word as well as their joint collocation embedding), a subcategorization pattern of both its elements, as well as their corresponding BabelNet id, and finally, indices of their occurrences in large scale reference corpora., This work has been supported by the European Commission in the context of its H2020 Program under the contract numbers 870930-RIA, 825079-STARTS, and 779962-RIA.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

The Information structure–prosody interface in text-to-speech technologies: an empirical perspective

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogues. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel’ˇcuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-–prosody interface, This work has been partially funded by the European Commission in the context of its H2020 Programme under the contract numbers H2020-645012-RIA (KRISTINA) and H2020-870930-IA (WELCOME). The second author has been funded by the Agencia Estatal de Investigaci´on (AEI), Ministerio de Ciencia, Innovaci´on y Universidades and the Fondo Social Europeo (FSE), grant RYC2015-17239 (AEI/FSE, UE).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Evaluating language models for the retrieval and categorization of lexical collocations

  • Espinosa-Anke, Luis
  • Codina Filbà, Joan
  • Wanner, Leo
Comunicació presentada a: EACL 2021 celebrat del 19 a 23 d'abril de 2021 en línia., Lexical collocations are idiosyncratic combinations of two syntactically bound lexical items (e.g., “heavy rain”, “take a step” or “undergo surgery”). Understanding their degree of compositionality and idiosyncrasy, as well their underlying semantics, is crucial for language learners, lexicographers and downstream NLP applications alike. In this paper we analyse a suite of language models for collocation understanding. We first construct a dataset of apparitions of lexical collocations in context, categorized into 16 representative semantic categories. Then, we perform two experiments: (1) unsupervised collocate retrieval, and (2) supervised collocation classification in context. We find that most models perform well in distinguishing light verb constructions, especially if the collocation’s first argument acts as a subject, but often fail to distinguish, first, different syntactic structures within the same semantic category, and second, finer-grained categories which restrict the set of correct collocates., This work was partially supported by the European Commission via its H2020 Program under the contract number 870930.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

ThemePro: a toolkit for the analysis of thematic progression

  • Wanner, Leo
  • Domínguez Bajo, Mónica
  • Soler Company, Juan
Comunicació presentada a: 12th Language Resources and Evaluation Conference, celebrat en línia del 13 al 15 de maig de 2020., This paper introduces ThemePro, a toolkit for the automatic analysis of thematic progression. Thematic progression is relevant to natural language processing (NLP) applications dealing, among others, with discourse structure, argumentation structure, natural language generation, summarization and topic detection. A web platform demonstrates the potential of this toolkit and provides a visualization of the results including syntactic trees, hierarchical thematicity over propositions and thematic progression over whole texts., This work is part of the WELCOME project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number 870930.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

The relation dimension in the identification and classification of lexically restricted word co-occurrences in text corpora

  • Shvets, Alexander
  • Wanner, Leo
The speech of native speakers is full of idiosyncrasies. Especially prominent are lexically restricted binary word co-occurrences of the type high esteem, strong tea, run [an] experiment, war break(s) out, etc. In lexicography, such co-occurrences are referred to as collocations. Due to their semi-decompositional nature, collocations are of high relevance to a large number of natural language processing applications as well as to second language learning. A substantial body of work exists on the automatic recognition of collocations in textual material and, increasingly also on their semantic classification, even if not yet in the mainstream research. Especially classification with respect to the lexical function (LF) taxonomy, which is the most detailed semantically oriented taxonomy of collocations available to date, proved to be of real use to human speakers and machines alike. The most recent approaches in the field are based on multilingual neural graph transformer models that use explicit syntactic dependencies. Our goal is to explore whether the extension of such a model by a semantic relation extraction network improves its classification performance or whether it already learns the corresponding semantic relations from the dependencies and the sentential contexts, such that an additional relation extraction network will not improve the overall performance. The experiments show that the semantic relation extraction layer indeed improves the overall performance of a graph transformer. However, this improvement is not very significant, such that we can conclude that graph transformers already learn to a certain extent the semantics of the dependencies between the collocation elements., This research was funded by the European Commission in the context of its H2020 Research and Development Program under the contract number 870930.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

ThemePro 2.0: showcasing the role of thematic progression in engaging human-computer interaction

  • Domínguez Bajo, Mónica
  • Soler Company, Juan
  • Wanner, Leo
Comunicació presentada a: INTERSPEECH 2021 celebrat del 30 d'agost a 3 de setembre de 2021 a Brno, República Txeca., Structuring speech into informative units is certainly a desirable feature in efficient human-machine communication. This paper introduces ThemePro 2.0, a toolkit that pre-processes long monologues into smaller cohesive units to be consumed by the text-to-speech module within a conversational agent. The methodology used is based upon the text’s discourse structure modelled as thematic progression patterns. As shown in the demonstration, thematic progression modelling captures the underlying information structure at the discourse level and is, therefore, instrumental for cohesive speech output in the TTS component., This work has received funding from the European Commission under Grant n. H2020-870930.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

How much pretraining data do language models need to learn syntax?

  • Pérez-Mayos, Laura
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), celebrat del 7 a l'11 de novembre de 2021 de manera virtual., Transformers-based pretrained language models achieve outstanding results in many wellknown NLU benchmarks. However, while pretraining methods are very convenient, they are expensive in terms of time and resources. This calls for a study of the impact of pretraining data size on the knowledge of the models. We explore this impact on the syntactic capabilities of RoBERTa, using models trained on incremental sizes of raw text data. First, we use syntactic structural probes to determine whether models pretrained on more data encode a higher amount of syntactic information. Second, we perform a targeted syntactic evaluation to analyze the impact of pretraining data size on the syntactic generalization performance of the models. Third, we compare the performance of the different models on three downstream applications: part-of-speech tagging, dependency parsing and paraphrase identification. We complement our study with an analysis of the cost-benefit trade-off of training such models. Our experiments show that while models pretrained on more data encode more syntactic knowledge and perform better on downstream applications, they do not always offer a better performance across the different syntactic phenomena and come at a higher financial and environmental cost., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 786731, 825079, 870930, and 952133. This work has been partially supported by the ICT PhD program of Universitat Pompeu Fabra through a travel grant.
Dipòsit Digital de la UB
Publication Artículo científico (JournalArticle)

The Information Structure-prosody interface in text-to-speech technologies. An empirical perspective

  • Domínguez, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet, these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogs. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel'čuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-prosody interface.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Evaluating language models for the retrieval and categorization of lexical collocations

  • Espinosa-Anke, Luis
  • Codina Filbà, Joan
  • Wanner, Leo
Comunicació presentada a: EACL 2021 celebrat del 19 a 23 d'abril de 2021 en línia., Lexical collocations are idiosyncratic combinations of two syntactically bound lexical items (e.g., “heavy rain”, “take a step” or “undergo surgery”). Understanding their degree of compositionality and idiosyncrasy, as well their underlying semantics, is crucial for language learners, lexicographers and downstream NLP applications alike. In this paper we analyse a suite of language models for collocation understanding. We first construct a dataset of apparitions of lexical collocations in context, categorized into 16 representative semantic categories. Then, we perform two experiments: (1) unsupervised collocate retrieval, and (2) supervised collocation classification in context. We find that most models perform well in distinguishing light verb constructions, especially if the collocation’s first argument acts as a subject, but often fail to distinguish, first, different syntactic structures within the same semantic category, and second, finer-grained categories which restrict the set of correct collocates., This work was partially supported by the European Commission via its H2020 Program under the contract number 870930.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Multilingual extraction and categorization of lexical collocations with graph-aware transformers

  • Espinosa-Anke, Luis
  • Shvets, Alexander
  • Mohammadshahi, Alireza
  • Henderson, James
  • Wanner, Leo
Comunicació presentada a 11th Joint Conference on Lexical and Computational Semantics (SEM 2022), celebrat del 14 al 15 de juliol de 2022 a Seattle, Estats Units., Recognizing and categorizing lexical collocations in context is useful for language learning, dictionary compilation and downstream NLP. However, it is a challenging task due to the varying degrees of frozenness lexical collocations exhibit. In this paper, we put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French., The work by Alexander Shvets and Leo Wanner has been supported by the European Commission in the context of the Horizon 2020 Research Program under the grant numbers 825079 and 870930. Alireza Mohammadshahi is supported by the Swiss National Science Foundation (grant number CRSII5- 180320).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

The Information Structure-prosody interface in text-to-speech technologies. An empirical perspective

  • Domínguez, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet, these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogs. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel'čuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-prosody interface.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

On the evolution of syntactic information encoded by BERT’s contextualized representations

  • Pérez-Mayos, Laura
  • Carlini, Roberto
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), celebrat del 19 al 23 d'abril de 2021 de manera virtual., The adaptation of pretrained language models to solve supervised tasks has become a baseline in NLP, and many recent works have focused on studying how linguistic information is encoded in the pretrained sentence representations. Among other information, it has been shown that entire syntax trees are implicitly embedded in the geometry of such models. As these models are often fine-tuned, it becomes increasingly important to understand how the encoded knowledge evolves along the fine-tuning. In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six different tasks, covering all levels of the linguistic structure. Experimental results show that the encoded syntactic information is forgotten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semantics-related tasks) in different ways along the finetuning process depending on the task., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

How much pretraining data do language models need to learn syntax?

  • Pérez-Mayos, Laura
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), celebrat del 7 a l'11 de novembre de 2021 de manera virtual., Transformers-based pretrained language models achieve outstanding results in many wellknown NLU benchmarks. However, while pretraining methods are very convenient, they are expensive in terms of time and resources. This calls for a study of the impact of pretraining data size on the knowledge of the models. We explore this impact on the syntactic capabilities of RoBERTa, using models trained on incremental sizes of raw text data. First, we use syntactic structural probes to determine whether models pretrained on more data encode a higher amount of syntactic information. Second, we perform a targeted syntactic evaluation to analyze the impact of pretraining data size on the syntactic generalization performance of the models. Third, we compare the performance of the different models on three downstream applications: part-of-speech tagging, dependency parsing and paraphrase identification. We complement our study with an analysis of the cost-benefit trade-off of training such models. Our experiments show that while models pretrained on more data encode more syntactic knowledge and perform better on downstream applications, they do not always offer a better performance across the different syntactic phenomena and come at a higher financial and environmental cost., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 786731, 825079, 870930, and 952133. This work has been partially supported by the ICT PhD program of Universitat Pompeu Fabra through a travel grant.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

ThemePro: a toolkit for the analysis of thematic progression

  • Wanner, Leo
  • Domínguez Bajo, Mónica
  • Soler Company, Juan
Comunicació presentada a: 12th Language Resources and Evaluation Conference, celebrat en línia del 13 al 15 de maig de 2020., This paper introduces ThemePro, a toolkit for the automatic analysis of thematic progression. Thematic progression is relevant to natural language processing (NLP) applications dealing, among others, with discourse structure, argumentation structure, natural language generation, summarization and topic detection. A web platform demonstrates the potential of this toolkit and provides a visualization of the results including syntactic trees, hierarchical thematicity over propositions and thematic progression over whole texts., This work is part of the WELCOME project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number 870930.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

CollFrEn: rich bilingual English–French collocation resource

  • Fisas Elizalde, Beatriz
  • Espinosa-Anke, Luis
  • Codina Filbà, Joan
  • Wanner, Leo
Comunicació presentada a: Joint Workshop on Multiword Expressions and Electronic Lexicons celebrat el 13 de desembre de 2020 de manera virtual., Collocations in the sense of idiosyncratic lexical co-occurrences of two syntactically bound words traditionally pose a challenge to language learners and many Natural Language Processing (NLP) applications alike. Reliable ground truth (i.e., ideally manually compiled) resources are thus of high value. We present a manually compiled bilingual English–French collocation resource with 7,480 collocations in English and 6,733 in French. Each collocation is enriched with information that facilitates its downstream exploitation in NLP tasks such as machine translation, word sense disambiguation, natural language generation, relation classification, and so forth. Our proposed enrichment covers: the semantic category of the collocation (its lexical function), its vector space representation (for each individual word as well as their joint collocation embedding), a subcategorization pattern of both its elements, as well as their corresponding BabelNet id, and finally, indices of their occurrences in large scale reference corpora., This work has been supported by the European Commission in the context of its H2020 Program under the contract numbers 870930-RIA, 825079-STARTS, and 779962-RIA.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

The relation dimension in the identification and classification of lexically restricted word co-occurrences in text corpora

  • Shvets, Alexander
  • Wanner, Leo
The speech of native speakers is full of idiosyncrasies. Especially prominent are lexically restricted binary word co-occurrences of the type high esteem, strong tea, run [an] experiment, war break(s) out, etc. In lexicography, such co-occurrences are referred to as collocations. Due to their semi-decompositional nature, collocations are of high relevance to a large number of natural language processing applications as well as to second language learning. A substantial body of work exists on the automatic recognition of collocations in textual material and, increasingly also on their semantic classification, even if not yet in the mainstream research. Especially classification with respect to the lexical function (LF) taxonomy, which is the most detailed semantically oriented taxonomy of collocations available to date, proved to be of real use to human speakers and machines alike. The most recent approaches in the field are based on multilingual neural graph transformer models that use explicit syntactic dependencies. Our goal is to explore whether the extension of such a model by a semantic relation extraction network improves its classification performance or whether it already learns the corresponding semantic relations from the dependencies and the sentential contexts, such that an additional relation extraction network will not improve the overall performance. The experiments show that the semantic relation extraction layer indeed improves the overall performance of a graph transformer. However, this improvement is not very significant, such that we can conclude that graph transformers already learn to a certain extent the semantics of the dependencies between the collocation elements., This research was funded by the European Commission in the context of its H2020 Research and Development Program under the contract number 870930.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Assessing the syntactic capabilities of transformer-based multilingual language models

  • Pérez-Mayos, Laura
  • Táboas García, Alba
  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, celebrat de l'1 al 6 d'agost de 2021 de manera virtual., Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of crosslingual transfer tasks. However, it remains unknown whether the optimization for different languages conditions the capacity of the models to generalize over syntactic structures, and how languages with syntactic phenomena of different complexity are affected. In this work, we explore the syntactic generalization capabilities of the monolingual and multilingual versions of BERT and RoBERTa. More specifically, we evaluate the syntactic generalization potential of the models on English and Spanish tests, comparing the syntactic abilities of monolingual and multilingual models on the same language (English), and of multilingual models on two different languages (English and Spanish). For English, we use the available SyntaxGym test suite; for Spanish, we introduce SyntaxGymES, a novel ensemble of targeted syntactic tests in Spanish, designed to evaluate the syntactic generalization capabilities of language models through the SyntaxGym online platform., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

ThemePro 2.0: showcasing the role of thematic progression in engaging human-computer interaction

  • Domínguez Bajo, Mónica
  • Soler Company, Juan
  • Wanner, Leo
Comunicació presentada a: INTERSPEECH 2021 celebrat del 30 d'agost a 3 de setembre de 2021 a Brno, República Txeca., Structuring speech into informative units is certainly a desirable feature in efficient human-machine communication. This paper introduces ThemePro 2.0, a toolkit that pre-processes long monologues into smaller cohesive units to be consumed by the text-to-speech module within a conversational agent. The methodology used is based upon the text’s discourse structure modelled as thematic progression patterns. As shown in the demonstration, thematic progression modelling captures the underlying information structure at the discourse level and is, therefore, instrumental for cohesive speech output in the TTS component., This work has received funding from the European Commission under Grant n. H2020-870930.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

The Information structure–prosody interface in text-to-speech technologies: an empirical perspective

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogues. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel’ˇcuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-–prosody interface, This work has been partially funded by the European Commission in the context of its H2020 Programme under the contract numbers H2020-645012-RIA (KRISTINA) and H2020-870930-IA (WELCOME). The second author has been funded by the Agencia Estatal de Investigaci´on (AEI), Ministerio de Ciencia, Innovaci´on y Universidades and the Fondo Social Europeo (FSE), grant RYC2015-17239 (AEI/FSE, UE).