Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 7
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

General scales unlock AI evaluation with explanatory and predictive power

  • Zhou, Lexin
  • Pacchiardi, Lorenzo
  • Collins, Katherine M.
  • Zhang, Seraphina
  • Zhao, Qinlin
  • Huang, Yitian
  • Sun, Luning
  • Prunty, Jonathan E.
  • Li, Zongqian
  • Sanchez-Garcia, Pablo
  • Casares, Pablo A. M.
  • Zu, Jiyun
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Moros-Daval, Yael
  • Jiang-Chen, Ke-Xin
  • Burden, John
  • Mehrbakhsh, Behzad|||0000-0001-9017-989X
  • Stillwell, David
  • Cebrian, Manuel
  • Wang, Jindong
  • Henderson, Peter
  • Wu, Sherry Tongshuang
  • Kyllonen, Patrick C.
  • Cheke, Lucy
  • Xie, Xing
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Ensuring safe and effective use of artificial intelligence (AI) requires understanding and anticipating its performance on new tasks, from advanced scientific challenges to transformed workplace activities1, 2-3. So far, benchmarking has guided progress in AI but has offered limited explanatory and predictive power for general-purpose AI systems4, 5, 6, 7-8, attributed to limited transferability across specific tasks9, 10-11. Here we introduce general scales for AI evaluation that elicit demand profiles explaining what capabilities common AI benchmarks truly measure, extract ability profiles quantifying the general strengths and limits of AI systems and robustly predict AI performance for new task instances. Our fully automated methodology builds on 18 rubrics, capturing a broad range of cognitive and intellectual demands, which place different task instances on the same general scales, illustrated on 15 large language models (LLMs) and 63 tasks. Both the demand and the ability profiles on these scales bring new insights such as construct validity through benchmark sensitivity and specificity and explain conflicting claims about whether AI has reasoning capabilities. Ultimately, high predictive power at the instance level becomes possible using the general scales, providing superior estimates over strong black-box baseline predictors, especially in out-of-distribution settings (new tasks and benchmarks). The scales, rubrics, battery, techniques and results presented here constitute a solid foundation for a science of AI evaluation, underpinning the reliable deployment of AI in the years ahead., We thank the POLARIS Lab and the Computational Cognitive Science Lab at Princeton University, the European AI Office, the UK AISI, the Future of Life Institute, the OECD Future of Skills team, S. Chang, M. Zilka, J. Lian, R. Burnell, W. Schellaert, A. Gomez, M. Tesic, C. Li and P. Romero for their valuable help and feedback at certain stages of the project. We thank OpenAI for granting us research access to several LLMs to conduct the experiments in this paper and DeepSeek and Meta for giving us access to the weights of their models. We acknowledge support from the following institutions: Microsoft Accelerate Foundation Models Research (AFMR) grant programme, Long-Term Future Scholarship financed by Coefficient Giving (formerly Open Philanthropy) and the Spanish Government's Knowledge Generation Projects (PID2023-150271NB-C21). This work was also supported by CIPROM/2022/6 (FASSLOW), IDIFEDER/2021/05 (CLUSTERIA), CIACIF/2023/276, financed by Generalitat Valenciana, the EC H2020-EU grant agreement no. 952215 (TAILOR), Spanish grants PID2021-122830OB-C42 (SFERA), PID2023-150271NB-C21 and PID2024-162030OB-100 (ROBIN) financed by MCIN/AEI/10.13039/501100011033 and 'ERDF A way of making Europe', Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, INCIBE's Chair financed by the EU's NextGenerationEU, EUR2024-153548 (PREDAIT) 'Towards Predictable AI' from 'Spanish Europe Excelencia' 2024, Spanish National Research Council (CSIC) Special Intramural Projects programme and the Cambridge Trust. M.C. declares support from Google.org through the Silicon Valley Community Foundation by means of a grant to Fundacion General CSIC. The research of J.H.-O. is supported by OpenAI's grant to the 'AI Progress through the Lens of Predictable AI Ecosystems' programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Cognitive Agents in Urban Mobility: Integrating LLM Reasoning into Multi-Agent Simulations

  • Calderón-Orellana, Christián Vladimir|||0009-0004-9199-6911
  • Martí, Pasqual|||0000-0001-6406-0641
  • Jordán, Jaume|||0000-0003-0400-9136
  • Palanca Cámara, Javier|||0000-0002-6209-9603
  • Julian, Vicente|||0000-0002-2743-6037
[EN] Urban mobility systems face escalating challenges associated with sustainability, equity, and resilience, further compounded by environmental pressures. Traditional agent-based models (ABMs) often fail to capture cognitively rich, adaptive behaviors, limiting their ability to simulate realistic user responses to disruptions. In this work, we propose a cognitive agent architecture based on Large Language Models (LLMs), featuring multi-horizon memory-driven planning, reflection, and adaptation. Integrated into the SimFleet agent-based simulator with realistic sociodemographic profiles, the agents dynamically generate, adjust, and reflect upon travel plans across a 20-day simulation involving over 320 individuals. Experimental results reveal emergent adaptation patterns under both stable and disrupted transport conditions, and an ablation study under severe service disruption quantifies the contributions of short-term and long-term memory modules to memory-driven reasoning, demonstrating the potential of LLM-driven agents to enhance the realism, flexibility, and interpretability of urban mobility simulations., This work was partially supported by the Spanish Government through grant PID2021-123673OB-C31 (MCIN/AEI/10.13039/501100011033) and the European Regional Development Fund (ERDF, "A way of making Europe"). Additional support was provided by grant TSI-100930-2023-9 from the "Secretaria de Estado de Digitalizacion e Inteligencia Artificial, Catedra ENIA-UPV", and by the PAID-01-22 programme for the recruitment of doctors within the Spanish Science and Technology System at the Universitat Politecnica de Valencia.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

From formal to informal: how register shift confuses LLMs in Spanish dialect recognition

  • Zander-Marzano, Samuel Axel
  • Amores-Giner, Pau
  • Hernández-Orallo, José|||0000-0001-9746-7632
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Mehrbakhsh, Behzad|||0000-0001-9017-989X
[EN] Language dialects and registers represent complex data shifts. The use of LLMs can take advantage of more abstract semantic features compared to traditional approaches based on word frequencies. We construct a register-aware benchmark spanning five countries (Argentina, Chile, Colombia, Mexico, Spain) and three registers (formal, informal, mixed). Four open small LLMs (LLaMA, Gemma, Mistral, DeepSeek) are fine-tuned for 5-way classification, evaluated with all train-test register pairings, and compared with an inexpensive classical TF-IDF + Na & iuml;ve Bayes setting. Several main findings emerge: (i) We find a specialization-generalization trade-off and asymmetric cross-register transfer: informal ->\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ ightarrow $$\end{document} formal degrades far less than formal ->\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ ightarrow $$\end{document} informal; (ii) misclassifications are structured: confusions concentrate in linguistically proximate pairs (e.g., Mexico-Colombia) and align with a low-dimensional geometry from Jensen-Shannon distances; (iii) errors stem from two main sources: over-reliance on toponyms/proper nouns (often defaulting to Spain), and interference from shared slang/morphology., This work has been supported by Open Philanthropy Long-term Future Fund, CIPROM/2022/6 (FASSLOW), funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN), funded by MCIN/AEI/ 10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024, and by the VANTAGE Project, Grant Agreement No. 101249800, funded by the EU through the ECCC under the Digital Europe Programme.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Robustness under noise: assessing the impact of perturbed key attributes on machine learning models

  • Padró-Ferragut, Cristina
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Ramírez Quintana, María José|||0000-0002-0559-3568
[EN] Machine learning models often struggle when confronted with the complexities of real-world data, such as noise and incomplete information, which pose significant challenges for practical applications. This paper presents a systematic framework for assessing the robustness of machine learning models against these adversities. In particular, we investigate how noise affecting the most critical attribute of a dataset impacts on model predictions. Our methodology uses a variety of datasets, a wide range of machine learning models, and a customizable noise induction function, allowing for comprehensive evaluations tailored to specific application needs. Through an extensive experimental analysis, we identify the vulnerabilities of models when their critical attributes are perturbed. In addition, by applying hierarchical clustering, we categorise machine learning families based on their robustness, establishing a taxonomy that highlights the susceptibility of different models to noise. Our findings provide important insights into the interplay between model performance and noise levels and improve the understanding of model behaviour under adverse conditions., We acknowledge support from: CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana; and Spanish grant PID2024-162030OB-I00 (ROBIN) funded by MCIN/AEI/10.13039/ 501100011033 and ERDF A way of making Europe", Cátedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE s Chair funded by the EU-NextGenerationEU through the Spanish government s Plan de Recuperación, Transformación y Resiliencia. CPF is supported by UPV under FPI grant PAID-01-22. Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Funding for open access charge: CRUE-Universitat Politecnica de Valencia.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Predictable artificial intelligence

  • Zhou, Lexin
  • Casares, Pablo A. M.
  • Burden, John
  • Burnell, Ryan
  • Cheke, Lucy
  • Marcoci, Alexandru
  • Heigeartaigh, Sean O.
  • Rutar, Danaja
  • Schellaert, Wout Willy M.
  • Voudouris, Konstantinos
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Ferri Ramírez, César|||0000-0002-8975-1120
  • Mehrbakhsh, Behzad|||0000-0001-9017-989X
  • Moros-Daval, Yael
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Many areas of artificial intelligence, and machine learning in particular, aim at being probably correct, i.e., valid on average, rather than pursuing the idealistic goal of being provably valid for all inputs. However, AI systems could still be predictably valid, such as an imperfect robot deliverer for which we can reliably and precisely predict the task instances for which it is correct and safe, its valid operating range. "Predictable AI" is a nascent research area that explores ways of anticipating key validity indicators (e.g., performance, safety) of present and future AI ecosystems. We argue that achieving predictability is crucial for fostering trust, liability, control, alignment and safety of AI, and thus should be prioritised over performance. We formally characterise predictability, explore its most relevant components, illustrate what can be predicted, describe alternative candidates for predictors, as well as the trade-offs between maximising validity and predictability. To illustrate these concepts, we bring an array of illustrative examples covering diverse ecosystem configurations. "Predictable AI" is related to other areas of technical and non-technical AI research, but have distinctive questions, hypotheses, techniques and challenges. This paper aims to elucidate them, calls for identifying paths towards a landscape of predictably valid AI systems and outlines the potential impact of this emergent field., This work has been supported by Open Philanthropy Long-term Future Fund, CIPROM/2022/6 (FASSLOW) , funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN) , funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024.JHO's research is supported by OpenAI's grant to the 'AI Progress through the Lens of Predictable AI Ecosystems' programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Follow the leader: a deep reinforcement learning framework for safe and efficient autonomous car-following

  • Marin, Andres L.
  • Makridis, Michail A.
  • Tansini, Alessandro
  • Pulvirenti, Luca
  • Corujo, Jaime Suarez
  • Komnos, Dimitrios
  • Fontaras, Georgios
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Ramírez Quintana, María José|||0000-0002-0559-3568
  • C. Monserrat|||0000-0003-1790-8085
[EN] Autonomous and semi-autonomous driving technologies present a promising path to achieving safer and more efficient transportation, potentially reducing energy use and emissions. However, these advantages might not be realized without carefully targeted optimization and ad-hoc designed dataset to evaluate the efficacy of the optimization performance. Through the analysis of both driver actions and vehicle dynamics using a real-world dataset, we present an innovative reward function based on the Deep Deterministic Policy Gradient (DDPG) method. The reward function aims to minimize energy consumption and the increased travel time while keeping acceleration within physical limits. The approach results in an average of 7% energy savings in various vehicle powertrains, including Internal Combustion Engine Vehicles (ICEV), Plug-in Hybrid Electric Vehicles (PHEV), and Battery Electric Vehicles (BEV), while keeping acceleration within realistic ranges, maintaining safe following distances, and keeping trip time increases below 1%. The dataset can be accessed at https://data.jrc.ec.europa.eu/collection/id-00437., We acknowledge support from: MICIU/AEI/10.13039/501100011033 and by ERDF, EU; CIPROM/2022/6 (FASSLOW) and IDIFEDER/2021/05 (CLUSTERIA) funded by Generalitat Valenciana; and Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/501100011033 and "ERDF A way of making Europe," Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia. The authors would also like to acknowledge the support of the Joint Research Centre.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Evaluating General-Purpose AI with Psychometrics

  • Wang, Xiting
  • Jiang, Liming
  • Stillwell, David
  • Chen, Shiqiang
  • Sun, Luning
  • Luo, Fang
  • Xie, Xing
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] IN THE RAPIDLY evolving field of artificial intelligence (AI), rigorous evaluation is critical. Evaluation helps prevent disastrous outcomes in high-stakes applications such as autonomous driving and medical diagnosis,7 and helps stop the spread of racist, sexist, ableist, extremist, and other harmful ideologies.4 It also plays a key role in understanding how intelligent AI is, preventing the misallocation of resources, and guiding improvements in future model development. Despite its importance, rigorous evaluation of versatile general-purpose AI systems such as ChatGPT and Gemini is challenging. The versatility of these systems, which can handle diverse tasks specified, This work was supervised by Luning Sun, Fang Luo, and Xing Xie. The authors would like to thank Jinyan Fan, Marija Slavkovik, Clemens Stachl, Alina A von Davier, Xiangen Hu, Yu Lu, Bryan Maddox, Mengxiao Zhu, Meng Li, Li Dong, Jindong Wang, Igor Sterner, Greg Serapio-Garcia, Peter Romero, Fengli Xu, Fernando Martinez Plumed, and Lidong Zhou for the discussions and comments on early versions of the manuscript. This research was funded by Beijing Natural Science Foundation L247011; Microsoft Research Asia Collaborative Research Program grant FY23-Research-Sponsorship-422 "The convergence of assessing human and big model capabilities"; CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN) funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe, Catedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperacion, Transformacion y Resiliencia, and EUR2024-153548 (PREDAIT) "Towards Predictable AI" from "Spanish Europe Excelencia" 2024; National Natural Science Foundation of China (NSFC) (NO. 62476279, NO. 92470205, NO. U2436209); Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China; The Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China No. 24XNKJ18; Fund for building world-class universities (disciplines) of Renmin University of China and Public Computing Cloud, Renmin University of China. L.S. and D.S. gratefully acknowledge financial support from Invesco through their philanthropic donation to Cambridge Judge Business School.