Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 2
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

From formal to informal: how register shift confuses LLMs in Spanish dialect recognition

  • Zander-Marzano, Samuel Axel
  • Amores-Giner, Pau
  • Hernández-Orallo, José|||0000-0001-9746-7632
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Mehrbakhsh, Behzad|||0000-0001-9017-989X
[EN] Language dialects and registers represent complex data shifts. The use of LLMs can take advantage of more abstract semantic features compared to traditional approaches based on word frequencies. We construct a register-aware benchmark spanning five countries (Argentina, Chile, Colombia, Mexico, Spain) and three registers (formal, informal, mixed). Four open small LLMs (LLaMA, Gemma, Mistral, DeepSeek) are fine-tuned for 5-way classification, evaluated with all train-test register pairings, and compared with an inexpensive classical TF-IDF + Na & iuml;ve Bayes setting. Several main findings emerge: (i) We find a specialization-generalization trade-off and asymmetric cross-register transfer: informal ->\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ ightarrow $$\end{document} formal degrades far less than formal ->\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ ightarrow $$\end{document} informal; (ii) misclassifications are structured: confusions concentrate in linguistically proximate pairs (e.g., Mexico-Colombia) and align with a low-dimensional geometry from Jensen-Shannon distances; (iii) errors stem from two main sources: over-reliance on toponyms/proper nouns (often defaulting to Spain), and interference from shared slang/morphology., This work has been supported by Open Philanthropy Long-term Future Fund, CIPROM/2022/6 (FASSLOW), funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN), funded by MCIN/AEI/ 10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024, and by the VANTAGE Project, Grant Agreement No. 101249800, funded by the EU through the ECCC under the Digital Europe Programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

What Should an AI Assessor Optimise for?

  • Romero-Alvarado, Daniel
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] An AI assessor is an external, ideally independent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can leverage information from the test results of many other AI systems and have the flexibility of being trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Using twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and nonmonotonic mappings and find that, contrary to intuition, optimising for more informative metrics (i.e., yielding a better-conditioned supervision signal) is not universally preferred. Surprisingly, some monotonic transformations are promising. For example, logistic loss is useful for minimising absolute or quadratic errors in regression, and logarithmic score helps maximise quadratic or spherical scores in classification., This work has been supported by CIPROM/2022/6 (FASSLOW), funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN), funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe; and by the VANTAGE Project, Grant Agreement No. 101249800, funded by the European Union through ECCC; and by EUR2024-153548 (PREDAIT) Towards Predictable AI" from Spanish Europe Excelencia" 2024; and the HIDDEN project is funded by the European Union, under grant agreement No 101202228. JHO s research is supported by OpenAI s grant to the AI Progress through the Lens of Predictable AI Ecosystems" programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.