Pasar al contenido principal

ADECUACION DE LOS SISTEMAS TRIBUTARIOS A LA CUARTA REVOLUCION INDUSTRIAL: INTELIGENCIA ARTIFICIAL, ROBOTICA Y NUEVAS REALIDADES TECNOLOGICAS

PID2022-140110OA-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2022
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT DE VALÈNCIA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 5
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Analysing the Predictability of Language Model Performance

  • Schellaert, Wout Willy M.
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Can a language model predict for which questions another language model will answer successfully? We investigate the extent to which performance prediction is possible and dissect various factors that influence it. Our experimental setting fine-tunes DeBERTa models, which we call assessors, on the evaluation results of generative language models with up to 128 billion parameters, which we refer to as subject systems. Our analysis spans more than 100 tasks from BIG-bench. We find that the assessors can match and even exceed the subjects' confidence in both refinement and calibration, anticipating failures at near perfect levels for some tasks. We also find that for performance prediction it can be beneficial to learn from the scores on multiple tasks or to learn from the scores of multiple subjects, but both depend on the task at hand. Lastly, we find that large and small subject systems are equally predictable, showing promise for the scalability of the predictability problem., This work was funded by valgrAI, the Future of Life Institute (FLI) under grant RFP2-152, the EU (FEDER) and Spanish grant RTI2018-094403-B-C32 funded by MCIN/AEI/10.13039/501100011033 and by CIPROM/2022/6 funded by Generalitat Valenciana, FISCALTICS (I+D+i PID2022-140110OA-I00) granted by MICIU/AEI/10.13039/501100011033, the Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/501100011033 and 'ERDF A way of making Europe, as well as the European Union under agreement INNEST/2021/317 (Neurocalcat) and by the Vic. Inv. of the Universitat Politecnica de Valencia under DOCEMPR21, DOCEMPR22, and DOCEMPR23'.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Larger and more instructable language models become less reliable

  • Zhou, Lexin
  • Schellaert, Wout Willy M.
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Moros-Daval, Yael
  • Ferri Ramírez, César|||0000-0002-8975-1120
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] The prevailing methods to make large language models more powerful and amenable have been based on continuous scaling up (that is, increasing their size, data volume and computational resources1 ) and bespoke shaping up (including post-fltering2,3 , fne tuning or use of human feedback4,5 ). However, larger and more instructable large language models may have become less reliable. By studying the relationship between difculty concordance, task avoidance and prompting stability of several language model families, here we show that easy instances for human participants are also easy for the models, but scaled-up, shaped-up models do not secure areas of low difculty in which either the model does not err or human supervision can spot the errors. We also fnd that early models often avoid user questions but scaled-up, shaped-up models tend to give an apparently sensible yet wrong answer much more often, including errors on difcult questions that human supervisors frequently overlook. Moreover, we observe that stability to diferent natural phrasings of the same question is improved by scaling-up and shaping-up interventions, but pockets of variability persist across difculty levels. These fndings highlight the need for a fundamental shift in the design and development of general-purpose artifcial intelligence, particularly in high-stakes areas for which a predictable distribution of errors is paramount., We acknowledge support from the following institutions: Long-Term Future Scholarship funded by Open Philanthropy; Grant for Master Studies funded by ValgrAI Valencian Graduate School and Research Network for Artificial Intelligence and Generalitat Valenciana; FISCALTICS (I+D+i PID2022-140110OA-I00), funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU; CIPROM/2022/6 (FASSLOW) and IDIFEDER/2021/05 (CLUSTERIA) funded by Generalitat Valenciana; the EC H2020-EU grant agreement no. 952215 (TAILOR); US DARPA HR00112120007 (RECoG-AI) and Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe ; and the Vic. Inv. of the Universitat Politècnica de Valencia under DOCEMPR21, DOCEMPR22 and DOCEMPR23.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

A general supply-inspect cost framework to regulate the reliability-usability trade-offs for few-shot inference

  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Ramírez Quintana, María José|||0000-0002-0559-3568
  • Hernández-Orallo, José|||0000-0001-9746-7632
  • Jaimovitch-López, Gonzalo Eduardo
  • Ferri, C.
[EN] Language models and other recent machine learning paradigms blur the distinction between generative and discriminative tasks, in a continuum that is regulated by the degree of pre- and post-supervision that is required from users, as well as the tolerated level of error. In few-shot inference, we need to find a trade-off between the number and cost of the solved examples that have to be supplied, those that have to be inspected (some of them accurate but others needing correction) and those that are wrong but pass undetected. In this paper, we define a new Supply-Inspect Cost Framework, associated graphical representations and comprehensive metrics that consider all these elements. To optimise few-shot inference under specific operating conditions, we introduce novel algorithms that go beyond the concept of rejection rules in both static and dynamic contexts. We illustrate the effectiveness of all these elements for a transformative domain, data wrangling, for which language models can have a huge impact if we are able to properly regulate the reliability-usability trade-off, as we do in this paper., The funding has been received from ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence; the Norwegian Research Council with Grant no. 329745 (Machine Teaching for Explainable AI); Generalitat Valenciana with Grant nos. CIPROM/2022/6 (FASSLOW) and IDIFEDER/2021/05 (CLUSTERIA); the European Commission under H2020-EU with Grant no. 952215 (TAILOR); US DARPA with Grant no. HR00112120007 (RECoG-AI); the Future of Life Institute with Grant no. RFP2-152; the Spanish Ministry of Science and Innovation (MCIN/AEI/10.13039/ 501100011033) with Grant no. PID2021-122830OB-C42 (SFERA) and ERDF A way of making Europe ; and the Spanish Ministry of Universities with Grant no. PID2022-140110OA-I00 (FISCALTICS) funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Comparative analysis of public and expert perceptions of electrified vehicles in the European Union

  • Mourtzouchou, Andromachi
  • Marin, Andres L.
  • Laveneziana, Lorenzo
  • Tansini, Alessandro
  • Suarez, Jaime
  • Garus, Ada
  • Grosso, Monica
  • Cordera, Rubén
  • Fontaras, Georgios
  • Ciuffo, Biagio
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
[EN] The transition to electric vehicles is widely regarded as an effective measure to combat climate change, and its success will utterly depend on the willingness of consumers to shift to this technology, making it essential to examine public perception when designing supportive policies. Nonetheless, public perception can be biased by multiple factors such as fake news, prejudices and misgivings, leading to a significant divergence from the views of experts in the transportation field. This paper examines the differences between the views of these two groups on electric vehicles, with a specific focus on the current perception in the European Union. The research adopts a two-fold approach, starting with an empirical driving study, followed by a series of focus group discussions with transportation experts and an analysis of Twitter (formally X) data. The data for this study was collected over five years, from June 2019 to June 2024. The findings indicate that both groups generally maintain a neutral stance, with neutrality appearing in 50% of their responses. However, experts tend to prefer battery electric vehicles, positively influenced by their driving experiences. In 40% of their comments, they report favourable sentiments toward these vehicles. Conversely, plug-in hybrid electric vehicles are favoured by the general public, with 35% of tweets indicating a positive attitude, as they are viewed as a more established technology. Differing perspectives on vehicle charging infrastructure are evident: specialists regard it as a hindrance to widespread adoption, with 30% expressing negative views, whereas the general population appears less concerned and more focused on the feelings and the experience of driving., This work was partly funded by CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana, FISCALTICS (I+D+i PID2022-140110OA-I00) granted by MICIU/AEI/https://doi.org/10.13039/501100011033, and Spanis h grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/https://doi.org/10.13039/501100011033 and ERDF A way of making Europe .
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Language Task Difficulty Prediction through LLM-Annotated Meta-Features

  • Moros-Daval, Yael
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Assessing the capabilities of large language models (LLMs) is increasingly challenging due to their generality and uneven task performance. Often, we do not know how much of the success or failure on a particular task is due to the `loading¿ of the language elements in the task, such as narrative understanding, or some other intrinsic (non-linguistic) components, such as domain-specific common sense or reasoning capabilities. Understanding what tasks are most loaded on language and determine the predictability of LLMs on these tasks is crucial for improving benchmarks, designing better LLMs, and ensuring their safe deployment. We present an innovative methodology that uses LLMs to annotate linguistic metafeatures , allowing us to predict task difficulty and understand linguistic loadings more accurately than traditional readability scores. Using GPT-4 for automated annotation, we show strong predictability for a variety of tasks and language models (e.g., MMLU with R2 from 0.68 to 0.83), but observe limited predictability for other tasks (e.g., LSAT with R2 of -0.07)., We acknowledge support from: Grant for Master Studies funded by ValgrAI Valencian Graduate School and Research Network for Artificial Intelligence and Generalitat Valenciana; FISCALTICS (I+D+i PID2022-140110OA-I00), granted by MICIU/AEI/10.13039/ 501100011033 and by ERDF, EU; CIPROM/2022/6 (FASSLOW) and IDIFEDER/2021/05 (CLUSTERIA) funded by Generalitat Valenciana; the EC H2020-EU grant agreement No. 952215 (TAILOR); US DARPA HR00112120007 (RECoG-AI) and Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/501100011033 and ERDF.