Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 1
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

What Should an AI Assessor Optimise for?

  • Romero-Alvarado, Daniel
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] An AI assessor is an external, ideally independent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can leverage information from the test results of many other AI systems and have the flexibility of being trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Using twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and nonmonotonic mappings and find that, contrary to intuition, optimising for more informative metrics (i.e., yielding a better-conditioned supervision signal) is not universally preferred. Surprisingly, some monotonic transformations are promising. For example, logistic loss is useful for minimising absolute or quadratic errors in regression, and logarithmic score helps maximise quadratic or spherical scores in classification., This work has been supported by CIPROM/2022/6 (FASSLOW), funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN), funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe; and by the VANTAGE Project, Grant Agreement No. 101249800, funded by the European Union through ECCC; and by EUR2024-153548 (PREDAIT) Towards Predictable AI" from Spanish Europe Excelencia" 2024; and the HIDDEN project is funded by the European Union, under grant agreement No 101202228. JHO s research is supported by OpenAI s grant to the AI Progress through the Lens of Predictable AI Ecosystems" programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.