RUA. Repositorio Institucional de la Universidad de Alicante
Publicación
Otros (Other)
Lessons learned from the evaluation of Spanish Language Models, Conclusiones de la evaluación de Modelos del Lenguaje en Español
Given the impact of language models on the field of Natural Language Processing, a number of Spanish encoder-only masked language models (aka BERTs) have been trained and released. These models were developed either within large projects using very large private corpora or by means of smaller scale academic efforts leveraging freely available data. In this paper we present a comprehensive head-to-head comparison of language models for Spanish with the following results: (i) Previously ignored multilingual models from large companies fare better than monolingual models, substantially changing the evaluation landscape of language models in Spanish; (ii) Results across the monolingual models are not conclusive, with supposedly smaller and inferior models performing competitively. Based on these empirical results, we argue for the need of more research to understand the factors underlying them. In this sense, the effect of corpus size, quality and pre-training techniques need to be further investigated to be able to obtain Spanish monolingual models significantly better than the multilingual ones released by large private companies, specially in the face of rapid ongoing progress in the field. The recent activity in the development of language technology for Spanish is to be welcomed, but our results show that building language models remains an open, resource-heavy problem which requires to marry resources (monetary and/or computational) with the best research expertise and practice.
Proyecto:
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2021-127777OB-C21
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 TED2021-130810B-C21
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PCI2020-120717-2
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2013-2016 RYC-2017-23647