RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication
Artículo científico (JournalArticle)
General scales unlock AI evaluation with explanatory and predictive power
[EN] Ensuring safe and effective use of artificial intelligence (AI) requires understanding and anticipating its performance on new tasks, from advanced scientific challenges to transformed workplace activities1, 2-3. So far, benchmarking has guided progress in AI but has offered limited explanatory and predictive power for general-purpose AI systems4, 5, 6, 7-8, attributed to limited transferability across specific tasks9, 10-11. Here we introduce general scales for AI evaluation that elicit demand profiles explaining what capabilities common AI benchmarks truly measure, extract ability profiles quantifying the general strengths and limits of AI systems and robustly predict AI performance for new task instances. Our fully automated methodology builds on 18 rubrics, capturing a broad range of cognitive and intellectual demands, which place different task instances on the same general scales, illustrated on 15 large language models (LLMs) and 63 tasks. Both the demand and the ability profiles on these scales bring new insights such as construct validity through benchmark sensitivity and specificity and explain conflicting claims about whether AI has reasoning capabilities. Ultimately, high predictive power at the instance level becomes possible using the general scales, providing superior estimates over strong black-box baseline predictors, especially in out-of-distribution settings (new tasks and benchmarks). The scales, rubrics, battery, techniques and results presented here constitute a solid foundation for a science of AI evaluation, underpinning the reliable deployment of AI in the years ahead., We thank the POLARIS Lab and the Computational Cognitive Science Lab at Princeton University, the European AI Office, the UK AISI, the Future of Life Institute, the OECD Future of Skills team, S. Chang, M. Zilka, J. Lian, R. Burnell, W. Schellaert, A. Gomez, M. Tesic, C. Li and P. Romero for their valuable help and feedback at certain stages of the project. We thank OpenAI for granting us research access to several LLMs to conduct the experiments in this paper and DeepSeek and Meta for giving us access to the weights of their models. We acknowledge support from the following institutions: Microsoft Accelerate Foundation Models Research (AFMR) grant programme, Long-Term Future Scholarship financed by Coefficient Giving (formerly Open Philanthropy) and the Spanish Government's Knowledge Generation Projects (PID2023-150271NB-C21). This work was also supported by CIPROM/2022/6 (FASSLOW), IDIFEDER/2021/05 (CLUSTERIA), CIACIF/2023/276, financed by Generalitat Valenciana, the EC H2020-EU grant agreement no. 952215 (TAILOR), Spanish grants PID2021-122830OB-C42 (SFERA), PID2023-150271NB-C21 and PID2024-162030OB-100 (ROBIN) financed by MCIN/AEI/10.13039/501100011033 and 'ERDF A way of making Europe', Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, INCIBE's Chair financed by the EU's NextGenerationEU, EUR2024-153548 (PREDAIT) 'Towards Predictable AI' from 'Spanish Europe Excelencia' 2024, Spanish National Research Council (CSIC) Special Intramural Projects programme and the Cambridge Trust. M.C. declares support from Google.org through the Silicon Valley Community Foundation by means of a grant to Fundacion General CSIC. The research of J.H.-O. is supported by OpenAI's grant to the 'AI Progress through the Lens of Predictable AI Ecosystems' programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge.
Project:
European Commission / H2020 952215
,
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2021-122830OB-C42
,
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2023-150271NB-C21
,
Generalitat Valenciana CIPROM/2022/6
,
Universitat Politècnica de València TSI-100930-2023-9
,
Generalitat Valenciana CIACIF/2023/276
,
Agencia Estatal de Investigación EUR2024-153548
,
Generalitat Valenciana IDIFEDER/2021/05
,
Agencia Estatal de Investigación PID2024-162030OB-I00