Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 12
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets?

  • Fortuna, Paula
  • Soler Company, Juan
  • Wanner, Leo
A considerable body of research deals with the automatic identification of hate speech and related phenomena. However, cross-dataset model generalization remains a challenge. In this context, we address two still open central questions: (i) to what extent does the generalization depend on the model and the composition and annotation of the training data in terms of different categories?, and (ii) do specific features of the datasets or models influence the generalization potential? To answer (i), we experiment with BERT, ALBERT, fastText, and SVM models trained on nine common public English datasets, whose class (or category) labels are standardized (and thus made comparable), in intra- and cross-dataset setups. The experiments show that indeed the generalization varies from model to model and that some of the categories (e.g., ‘toxic’, ‘abusive’, or ‘offensive’) serve better as cross-dataset training categories than others (e.g., ‘hate speech’). To answer (ii), we use a Random Forest model for assessing the relevance of different model and dataset features during the prediction of the performance of 450 BERT, 450 ALBERT, 450 fastText, and 348 SVM binary abusive language classifiers (1698 in total). We find that in order to generalize well, a model already needs to perform well in an intra-dataset scenario. Furthermore, we find that some other parameters are equally decisive for the success of the generalization, including, e.g., the training and target categories and the percentage of the out-of-domain vocabulary., The first author is supported by the research grant SFRH/BD/143623/2019, provided by the Portuguese national funding agency for science, research and technology, Fundação para a Ciência e a Tecnologia (FCT), within the scope of Operational Program Human Capital (POCH), supported by the European Social Fund and by national funds from MCTES. The work of the second and third authors has been supported by the European Commission in the context of the H2020 Research Program under the contract numbers 700024 and 786731.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

On the evolution of syntactic information encoded by BERT’s contextualized representations

  • Pérez-Mayos, Laura
  • Carlini, Roberto
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), celebrat del 19 al 23 d'abril de 2021 de manera virtual., The adaptation of pretrained language models to solve supervised tasks has become a baseline in NLP, and many recent works have focused on studying how linguistic information is encoded in the pretrained sentence representations. Among other information, it has been shown that entire syntax trees are implicitly embedded in the geometry of such models. As these models are often fine-tuned, it becomes increasingly important to understand how the encoded knowledge evolves along the fine-tuning. In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six different tasks, covering all levels of the linguistic structure. Experimental results show that the encoded syntactic information is forgotten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semantics-related tasks) in different ways along the finetuning process depending on the task., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Assessing the syntactic capabilities of transformer-based multilingual language models

  • Pérez-Mayos, Laura
  • Táboas García, Alba
  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, celebrat de l'1 al 6 d'agost de 2021 de manera virtual., Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of crosslingual transfer tasks. However, it remains unknown whether the optimization for different languages conditions the capacity of the models to generalize over syntactic structures, and how languages with syntactic phenomena of different complexity are affected. In this work, we explore the syntactic generalization capabilities of the monolingual and multilingual versions of BERT and RoBERTa. More specifically, we evaluate the syntactic generalization potential of the models on English and Spanish tests, comparing the syntactic abilities of monolingual and multilingual models on the same language (English), and of multilingual models on two different languages (English and Spanish). For English, we use the available SyntaxGym test suite; for Spanish, we introduce SyntaxGymES, a novel ensemble of targeted syntactic tests in Spanish, designed to evaluate the syntactic generalization capabilities of language models through the SyntaxGym online platform., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

PyToBI: a toolkit for ToBI labeling under Python

  • Domínguez Bajo, Mónica
  • Rohrer, Patrick Louis
  • Soler Company, Juan
Comunicació presentada a: Interspeech 2019, celebrada del 15 al 19 de setembre de 2019 a Graz, Àustria., PyToBI is introduced as a user-friendly toolkit for the automatic annotation of intonation contours using the Tones and Breaks Indexes convention, known as ToBI., This project has received partial funding from the European Unions Horizon 2020 research and innovation program under grant agreement No 786731.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

How much pretraining data do language models need to learn syntax?

  • Pérez-Mayos, Laura
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), celebrat del 7 a l'11 de novembre de 2021 de manera virtual., Transformers-based pretrained language models achieve outstanding results in many wellknown NLU benchmarks. However, while pretraining methods are very convenient, they are expensive in terms of time and resources. This calls for a study of the impact of pretraining data size on the knowledge of the models. We explore this impact on the syntactic capabilities of RoBERTa, using models trained on incremental sizes of raw text data. First, we use syntactic structural probes to determine whether models pretrained on more data encode a higher amount of syntactic information. Second, we perform a targeted syntactic evaluation to analyze the impact of pretraining data size on the syntactic generalization performance of the models. Third, we compare the performance of the different models on three downstream applications: part-of-speech tagging, dependency parsing and paraphrase identification. We complement our study with an analysis of the cost-benefit trade-off of training such models. Our experiments show that while models pretrained on more data encode more syntactic knowledge and perform better on downstream applications, they do not always offer a better performance across the different syntactic phenomena and come at a higher financial and environmental cost., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 786731, 825079, 870930, and 952133. This work has been partially supported by the ICT PhD program of Universitat Pompeu Fabra through a travel grant.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Targets and aspects in social media hate speech

  • Shvets, Alexander
  • Fortuna, Paula
  • Wanner, Leo
Comunicació presentada a The 5th Workshop on Online Abuse and Harms (WOAH 2021), celebrat el 6 d'agost de 2021 de manera virtual., Mainstream research on hate speech focused so far predominantly on the task of classifying mainly social media posts with respect to predefined typologies of rather coarse-grained hate speech categories. This may be sufficient if the goal is to detect and delete abusive language posts. However, removal is not always possible due to the legislation of a country. Also, there is evidence that hate speech cannot be successfully combated by merely removing hate speech posts; they should be countered by education and counter-narratives. For this purpose, we need to identify (i) who is the target in a given hate speech post, and (ii) what aspects (or characteristics) of the target are attributed to the target in the post. As the first approximation, we propose to adapt a generic state-of-the-art concept extraction model to the hate speech domain. The outcome of the experiments is promising and can serve as inspiration for further work on the task., Paula Fortuna is supported by the research grant SFRH/BD/143623/2019, provided by the Portuguese national funding agency for science, research and technology, Fundação para a Ciência e a Tecnologia (FCT), within the scope of the Operational Program Human Capital (POCH), supported by the European Social Fund and by national funds from MCTES. The work of Alexander Shvets, Juan Soler-Company, and Leo Wanner has been supported by the European Commission in the context of the H2020 Research Program under the contract numbers 700024, 786731, and 825079.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Targets and aspects in social media hate speech

  • Shvets, Alexander
  • Fortuna, Paula
  • Wanner, Leo
Comunicació presentada a The 5th Workshop on Online Abuse and Harms (WOAH 2021), celebrat el 6 d'agost de 2021 de manera virtual., Mainstream research on hate speech focused so far predominantly on the task of classifying mainly social media posts with respect to predefined typologies of rather coarse-grained hate speech categories. This may be sufficient if the goal is to detect and delete abusive language posts. However, removal is not always possible due to the legislation of a country. Also, there is evidence that hate speech cannot be successfully combated by merely removing hate speech posts; they should be countered by education and counter-narratives. For this purpose, we need to identify (i) who is the target in a given hate speech post, and (ii) what aspects (or characteristics) of the target are attributed to the target in the post. As the first approximation, we propose to adapt a generic state-of-the-art concept extraction model to the hate speech domain. The outcome of the experiments is promising and can serve as inspiration for further work on the task., Paula Fortuna is supported by the research grant SFRH/BD/143623/2019, provided by the Portuguese national funding agency for science, research and technology, Fundação para a Ciência e a Tecnologia (FCT), within the scope of the Operational Program Human Capital (POCH), supported by the European Social Fund and by national funds from MCTES. The work of Alexander Shvets, Juan Soler-Company, and Leo Wanner has been supported by the European Commission in the context of the H2020 Research Program under the contract numbers 700024, 786731, and 825079.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

PyToBI: a toolkit for ToBI labeling under Python

  • Domínguez Bajo, Mónica
  • Rohrer, Patrick Louis
  • Soler Company, Juan
Comunicació presentada a: Interspeech 2019, celebrada del 15 al 19 de setembre de 2019 a Graz, Àustria., PyToBI is introduced as a user-friendly toolkit for the automatic annotation of intonation contours using the Tones and Breaks Indexes convention, known as ToBI., This project has received partial funding from the European Unions Horizon 2020 research and innovation program under grant agreement No 786731.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets?

  • Fortuna, Paula
  • Soler Company, Juan
  • Wanner, Leo
A considerable body of research deals with the automatic identification of hate speech and related phenomena. However, cross-dataset model generalization remains a challenge. In this context, we address two still open central questions: (i) to what extent does the generalization depend on the model and the composition and annotation of the training data in terms of different categories?, and (ii) do specific features of the datasets or models influence the generalization potential? To answer (i), we experiment with BERT, ALBERT, fastText, and SVM models trained on nine common public English datasets, whose class (or category) labels are standardized (and thus made comparable), in intra- and cross-dataset setups. The experiments show that indeed the generalization varies from model to model and that some of the categories (e.g., ‘toxic’, ‘abusive’, or ‘offensive’) serve better as cross-dataset training categories than others (e.g., ‘hate speech’). To answer (ii), we use a Random Forest model for assessing the relevance of different model and dataset features during the prediction of the performance of 450 BERT, 450 ALBERT, 450 fastText, and 348 SVM binary abusive language classifiers (1698 in total). We find that in order to generalize well, a model already needs to perform well in an intra-dataset scenario. Furthermore, we find that some other parameters are equally decisive for the success of the generalization, including, e.g., the training and target categories and the percentage of the out-of-domain vocabulary., The first author is supported by the research grant SFRH/BD/143623/2019, provided by the Portuguese national funding agency for science, research and technology, Fundação para a Ciência e a Tecnologia (FCT), within the scope of Operational Program Human Capital (POCH), supported by the European Social Fund and by national funds from MCTES. The work of the second and third authors has been supported by the European Commission in the context of the H2020 Research Program under the contract numbers 700024 and 786731.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

On the evolution of syntactic information encoded by BERT’s contextualized representations

  • Pérez-Mayos, Laura
  • Carlini, Roberto
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), celebrat del 19 al 23 d'abril de 2021 de manera virtual., The adaptation of pretrained language models to solve supervised tasks has become a baseline in NLP, and many recent works have focused on studying how linguistic information is encoded in the pretrained sentence representations. Among other information, it has been shown that entire syntax trees are implicitly embedded in the geometry of such models. As these models are often fine-tuned, it becomes increasingly important to understand how the encoded knowledge evolves along the fine-tuning. In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six different tasks, covering all levels of the linguistic structure. Experimental results show that the encoded syntactic information is forgotten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semantics-related tasks) in different ways along the finetuning process depending on the task., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

How much pretraining data do language models need to learn syntax?

  • Pérez-Mayos, Laura
  • Ballesteros, Miguel
  • Wanner, Leo
Comunicació presentada a 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), celebrat del 7 a l'11 de novembre de 2021 de manera virtual., Transformers-based pretrained language models achieve outstanding results in many wellknown NLU benchmarks. However, while pretraining methods are very convenient, they are expensive in terms of time and resources. This calls for a study of the impact of pretraining data size on the knowledge of the models. We explore this impact on the syntactic capabilities of RoBERTa, using models trained on incremental sizes of raw text data. First, we use syntactic structural probes to determine whether models pretrained on more data encode a higher amount of syntactic information. Second, we perform a targeted syntactic evaluation to analyze the impact of pretraining data size on the syntactic generalization performance of the models. Third, we compare the performance of the different models on three downstream applications: part-of-speech tagging, dependency parsing and paraphrase identification. We complement our study with an analysis of the cost-benefit trade-off of training such models. Our experiments show that while models pretrained on more data encode more syntactic knowledge and perform better on downstream applications, they do not always offer a better performance across the different syntactic phenomena and come at a higher financial and environmental cost., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 786731, 825079, 870930, and 952133. This work has been partially supported by the ICT PhD program of Universitat Pompeu Fabra through a travel grant.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Assessing the syntactic capabilities of transformer-based multilingual language models

  • Pérez-Mayos, Laura
  • Táboas García, Alba
  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, celebrat de l'1 al 6 d'agost de 2021 de manera virtual., Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of crosslingual transfer tasks. However, it remains unknown whether the optimization for different languages conditions the capacity of the models to generalize over syntactic structures, and how languages with syntactic phenomena of different complexity are affected. In this work, we explore the syntactic generalization capabilities of the monolingual and multilingual versions of BERT and RoBERTa. More specifically, we evaluate the syntactic generalization potential of the models on English and Spanish tests, comparing the syntactic abilities of monolingual and multilingual models on the same language (English), and of multilingual models on two different languages (English and Spanish). For English, we use the available SyntaxGym test suite; for Spanish, we introduce SyntaxGymES, a novel ensemble of targeted syntactic tests in Spanish, designed to evaluate the syntactic generalization capabilities of language models through the SyntaxGym online platform., This work has been partially funded by the European Commission via its H2020 Research Program under the contract numbers 779962, 786731, 825079, and 870930.