Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 5
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

General scales unlock AI evaluation with explanatory and predictive power

  • Zhou, Lexin
  • Pacchiardi, Lorenzo
  • Collins, Katherine M.
  • Zhang, Seraphina
  • Zhao, Qinlin
  • Huang, Yitian
  • Sun, Luning
  • Prunty, Jonathan E.
  • Li, Zongqian
  • Sanchez-Garcia, Pablo
  • Casares, Pablo A. M.
  • Zu, Jiyun
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Moros-Daval, Yael
  • Jiang-Chen, Ke-Xin
  • Burden, John
  • Mehrbakhsh, Behzad|||0000-0001-9017-989X
  • Stillwell, David
  • Cebrian, Manuel
  • Wang, Jindong
  • Henderson, Peter
  • Wu, Sherry Tongshuang
  • Kyllonen, Patrick C.
  • Cheke, Lucy
  • Xie, Xing
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Ensuring safe and effective use of artificial intelligence (AI) requires understanding and anticipating its performance on new tasks, from advanced scientific challenges to transformed workplace activities1, 2-3. So far, benchmarking has guided progress in AI but has offered limited explanatory and predictive power for general-purpose AI systems4, 5, 6, 7-8, attributed to limited transferability across specific tasks9, 10-11. Here we introduce general scales for AI evaluation that elicit demand profiles explaining what capabilities common AI benchmarks truly measure, extract ability profiles quantifying the general strengths and limits of AI systems and robustly predict AI performance for new task instances. Our fully automated methodology builds on 18 rubrics, capturing a broad range of cognitive and intellectual demands, which place different task instances on the same general scales, illustrated on 15 large language models (LLMs) and 63 tasks. Both the demand and the ability profiles on these scales bring new insights such as construct validity through benchmark sensitivity and specificity and explain conflicting claims about whether AI has reasoning capabilities. Ultimately, high predictive power at the instance level becomes possible using the general scales, providing superior estimates over strong black-box baseline predictors, especially in out-of-distribution settings (new tasks and benchmarks). The scales, rubrics, battery, techniques and results presented here constitute a solid foundation for a science of AI evaluation, underpinning the reliable deployment of AI in the years ahead., We thank the POLARIS Lab and the Computational Cognitive Science Lab at Princeton University, the European AI Office, the UK AISI, the Future of Life Institute, the OECD Future of Skills team, S. Chang, M. Zilka, J. Lian, R. Burnell, W. Schellaert, A. Gomez, M. Tesic, C. Li and P. Romero for their valuable help and feedback at certain stages of the project. We thank OpenAI for granting us research access to several LLMs to conduct the experiments in this paper and DeepSeek and Meta for giving us access to the weights of their models. We acknowledge support from the following institutions: Microsoft Accelerate Foundation Models Research (AFMR) grant programme, Long-Term Future Scholarship financed by Coefficient Giving (formerly Open Philanthropy) and the Spanish Government's Knowledge Generation Projects (PID2023-150271NB-C21). This work was also supported by CIPROM/2022/6 (FASSLOW), IDIFEDER/2021/05 (CLUSTERIA), CIACIF/2023/276, financed by Generalitat Valenciana, the EC H2020-EU grant agreement no. 952215 (TAILOR), Spanish grants PID2021-122830OB-C42 (SFERA), PID2023-150271NB-C21 and PID2024-162030OB-100 (ROBIN) financed by MCIN/AEI/10.13039/501100011033 and 'ERDF A way of making Europe', Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, INCIBE's Chair financed by the EU's NextGenerationEU, EUR2024-153548 (PREDAIT) 'Towards Predictable AI' from 'Spanish Europe Excelencia' 2024, Spanish National Research Council (CSIC) Special Intramural Projects programme and the Cambridge Trust. M.C. declares support from Google.org through the Silicon Valley Community Foundation by means of a grant to Fundacion General CSIC. The research of J.H.-O. is supported by OpenAI's grant to the 'AI Progress through the Lens of Predictable AI Ecosystems' programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

From formal to informal: how register shift confuses LLMs in Spanish dialect recognition

  • Zander-Marzano, Samuel Axel
  • Amores-Giner, Pau
  • Hernández-Orallo, José|||0000-0001-9746-7632
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Mehrbakhsh, Behzad|||0000-0001-9017-989X
[EN] Language dialects and registers represent complex data shifts. The use of LLMs can take advantage of more abstract semantic features compared to traditional approaches based on word frequencies. We construct a register-aware benchmark spanning five countries (Argentina, Chile, Colombia, Mexico, Spain) and three registers (formal, informal, mixed). Four open small LLMs (LLaMA, Gemma, Mistral, DeepSeek) are fine-tuned for 5-way classification, evaluated with all train-test register pairings, and compared with an inexpensive classical TF-IDF + Na & iuml;ve Bayes setting. Several main findings emerge: (i) We find a specialization-generalization trade-off and asymmetric cross-register transfer: informal ->\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ ightarrow $$\end{document} formal degrades far less than formal ->\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ ightarrow $$\end{document} informal; (ii) misclassifications are structured: confusions concentrate in linguistically proximate pairs (e.g., Mexico-Colombia) and align with a low-dimensional geometry from Jensen-Shannon distances; (iii) errors stem from two main sources: over-reliance on toponyms/proper nouns (often defaulting to Spain), and interference from shared slang/morphology., This work has been supported by Open Philanthropy Long-term Future Fund, CIPROM/2022/6 (FASSLOW), funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN), funded by MCIN/AEI/ 10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024, and by the VANTAGE Project, Grant Agreement No. 101249800, funded by the EU through the ECCC under the Digital Europe Programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Predictable artificial intelligence

  • Zhou, Lexin
  • Casares, Pablo A. M.
  • Burden, John
  • Burnell, Ryan
  • Cheke, Lucy
  • Marcoci, Alexandru
  • Heigeartaigh, Sean O.
  • Rutar, Danaja
  • Schellaert, Wout Willy M.
  • Voudouris, Konstantinos
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Ferri Ramírez, César|||0000-0002-8975-1120
  • Mehrbakhsh, Behzad|||0000-0001-9017-989X
  • Moros-Daval, Yael
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Many areas of artificial intelligence, and machine learning in particular, aim at being probably correct, i.e., valid on average, rather than pursuing the idealistic goal of being provably valid for all inputs. However, AI systems could still be predictably valid, such as an imperfect robot deliverer for which we can reliably and precisely predict the task instances for which it is correct and safe, its valid operating range. "Predictable AI" is a nascent research area that explores ways of anticipating key validity indicators (e.g., performance, safety) of present and future AI ecosystems. We argue that achieving predictability is crucial for fostering trust, liability, control, alignment and safety of AI, and thus should be prioritised over performance. We formally characterise predictability, explore its most relevant components, illustrate what can be predicted, describe alternative candidates for predictors, as well as the trade-offs between maximising validity and predictability. To illustrate these concepts, we bring an array of illustrative examples covering diverse ecosystem configurations. "Predictable AI" is related to other areas of technical and non-technical AI research, but have distinctive questions, hypotheses, techniques and challenges. This paper aims to elucidate them, calls for identifying paths towards a landscape of predictably valid AI systems and outlines the potential impact of this emergent field., This work has been supported by Open Philanthropy Long-term Future Fund, CIPROM/2022/6 (FASSLOW) , funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN) , funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024.JHO's research is supported by OpenAI's grant to the 'AI Progress through the Lens of Predictable AI Ecosystems' programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Evaluating General-Purpose AI with Psychometrics

  • Wang, Xiting
  • Jiang, Liming
  • Stillwell, David
  • Chen, Shiqiang
  • Sun, Luning
  • Luo, Fang
  • Xie, Xing
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] IN THE RAPIDLY evolving field of artificial intelligence (AI), rigorous evaluation is critical. Evaluation helps prevent disastrous outcomes in high-stakes applications such as autonomous driving and medical diagnosis,7 and helps stop the spread of racist, sexist, ableist, extremist, and other harmful ideologies.4 It also plays a key role in understanding how intelligent AI is, preventing the misallocation of resources, and guiding improvements in future model development. Despite its importance, rigorous evaluation of versatile general-purpose AI systems such as ChatGPT and Gemini is challenging. The versatility of these systems, which can handle diverse tasks specified, This work was supervised by Luning Sun, Fang Luo, and Xing Xie. The authors would like to thank Jinyan Fan, Marija Slavkovik, Clemens Stachl, Alina A von Davier, Xiangen Hu, Yu Lu, Bryan Maddox, Mengxiao Zhu, Meng Li, Li Dong, Jindong Wang, Igor Sterner, Greg Serapio-Garcia, Peter Romero, Fengli Xu, Fernando Martinez Plumed, and Lidong Zhou for the discussions and comments on early versions of the manuscript. This research was funded by Beijing Natural Science Foundation L247011; Microsoft Research Asia Collaborative Research Program grant FY23-Research-Sponsorship-422 "The convergence of assessing human and big model capabilities"; CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN) funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024; National Natural Science Foundation of China (NSFC) (NO. 62476279, NO. 92470205, NO. U2436209); Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China; The Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China No. 24XNKJ18; Fund for building world-class universities (disciplines) of Renmin University of China and Public Computing Cloud, Renmin University of China. L.S. and D.S. gratefully acknowledge financial support from Invesco through their philanthropic donation to Cambridge Judge Business School.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

What Should an AI Assessor Optimise for?

  • Romero-Alvarado, Daniel
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] An AI assessor is an external, ideally independent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can leverage information from the test results of many other AI systems and have the flexibility of being trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Using twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and nonmonotonic mappings and find that, contrary to intuition, optimising for more informative metrics (i.e., yielding a better-conditioned supervision signal) is not universally preferred. Surprisingly, some monotonic transformations are promising. For example, logistic loss is useful for minimising absolute or quadratic errors in regression, and logarithmic score helps maximise quadratic or spherical scores in classification., This work has been supported by CIPROM/2022/6 (FASSLOW), funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN), funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe; and by the VANTAGE Project, Grant Agreement No. 101249800, funded by the European Union through ECCC; and by EUR2024-153548 (PREDAIT) Towards Predictable AI" from Spanish Europe Excelencia" 2024; and the HIDDEN project is funded by the European Union, under grant agreement No 101202228. JHO s research is supported by OpenAI s grant to the AI Progress through the Lens of Predictable AI Ecosystems" programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.