Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 2
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Evaluating General-Purpose AI with Psychometrics

  • Wang, Xiting
  • Jiang, Liming
  • Stillwell, David
  • Chen, Shiqiang
  • Sun, Luning
  • Luo, Fang
  • Xie, Xing
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] IN THE RAPIDLY evolving field of artificial intelligence (AI), rigorous evaluation is critical. Evaluation helps prevent disastrous outcomes in high-stakes applications such as autonomous driving and medical diagnosis,7 and helps stop the spread of racist, sexist, ableist, extremist, and other harmful ideologies.4 It also plays a key role in understanding how intelligent AI is, preventing the misallocation of resources, and guiding improvements in future model development. Despite its importance, rigorous evaluation of versatile general-purpose AI systems such as ChatGPT and Gemini is challenging. The versatility of these systems, which can handle diverse tasks specified, This work was supervised by Luning Sun, Fang Luo, and Xing Xie. The authors would like to thank Jinyan Fan, Marija Slavkovik, Clemens Stachl, Alina A von Davier, Xiangen Hu, Yu Lu, Bryan Maddox, Mengxiao Zhu, Meng Li, Li Dong, Jindong Wang, Igor Sterner, Greg Serapio-Garcia, Peter Romero, Fengli Xu, Fernando Martinez Plumed, and Lidong Zhou for the discussions and comments on early versions of the manuscript. This research was funded by Beijing Natural Science Foundation L247011; Microsoft Research Asia Collaborative Research Program grant FY23-Research-Sponsorship-422 "The convergence of assessing human and big model capabilities"; CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN) funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024; National Natural Science Foundation of China (NSFC) (NO. 62476279, NO. 92470205, NO. U2436209); Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China; The Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China No. 24XNKJ18; Fund for building world-class universities (disciplines) of Renmin University of China and Public Computing Cloud, Renmin University of China. L.S. and D.S. gratefully acknowledge financial support from Invesco through their philanthropic donation to Cambridge Judge Business School.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics

  • Sun, Luning
  • Gibbons, Christopher
  • Wang, Xiting
  • Jiang, Liming
  • Stillwell, David
  • Luo, Fang
  • Xie, Xing
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption., LS gratefully acknowledges financial support from Invesco through their philanthropic donation to Cambridge Judge Business School. XW is funded by the National Natural Science Foundation of China (Grant 62476279) . FL is funded by the National Natural Science Foundation of China (Grant 62377003) .