RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication
Artículo científico (JournalArticle)
Evaluating General-Purpose AI with Psychometrics
[EN] IN THE RAPIDLY evolving field of artificial intelligence (AI), rigorous evaluation is critical. Evaluation helps prevent disastrous outcomes in high-stakes applications such as autonomous driving and medical diagnosis,7 and helps stop the spread of racist, sexist, ableist, extremist, and other harmful ideologies.4 It also plays a key role in understanding how intelligent AI is, preventing the misallocation of resources, and guiding improvements in future model development. Despite its importance, rigorous evaluation of versatile general-purpose AI systems such as ChatGPT and Gemini is challenging. The versatility of these systems, which can handle diverse tasks specified, This work was supervised by Luning Sun, Fang Luo, and Xing Xie. The authors would like to thank Jinyan Fan, Marija Slavkovik, Clemens Stachl, Alina A von Davier, Xiangen Hu, Yu Lu, Bryan Maddox, Mengxiao Zhu, Meng Li, Li Dong, Jindong Wang, Igor Sterner, Greg Serapio-Garcia, Peter Romero, Fengli Xu, Fernando Martinez Plumed, and Lidong Zhou for the discussions and comments on early versions of the manuscript. This research was funded by Beijing Natural Science Foundation L247011; Microsoft Research Asia Collaborative Research Program grant FY23-Research-Sponsorship-422 "The convergence of assessing human and big model capabilities"; CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN) funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024; National Natural Science Foundation of China (NSFC) (NO. 62476279, NO. 92470205, NO. U2436209); Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China; The Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China No. 24XNKJ18; Fund for building world-class universities (disciplines) of Renmin University of China and Public Computing Cloud, Renmin University of China. L.S. and D.S. gratefully acknowledge financial support from Invesco through their philanthropic donation to Cambridge Judge Business School.
Project:
Generalitat Valenciana CIPROM/2022/6
,
Agencia Estatal de Investigación PID2024-162030OB-I00
,
Universitat Politècnica de València TSI-100930-2023-9
,
Agencia Estatal de Investigación EUR2024-153548
,
National Natural Science Foundation of China 62476279
,
National Natural Science Foundation of China 92470205
,
National Natural Science Foundation of China U2436209