Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 3
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Analysing the Predictability of Language Model Performance

  • Schellaert, Wout Willy M.
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] Can a language model predict for which questions another language model will answer successfully? We investigate the extent to which performance prediction is possible and dissect various factors that influence it. Our experimental setting fine-tunes DeBERTa models, which we call assessors, on the evaluation results of generative language models with up to 128 billion parameters, which we refer to as subject systems. Our analysis spans more than 100 tasks from BIG-bench. We find that the assessors can match and even exceed the subjects' confidence in both refinement and calibration, anticipating failures at near perfect levels for some tasks. We also find that for performance prediction it can be beneficial to learn from the scores on multiple tasks or to learn from the scores of multiple subjects, but both depend on the task at hand. Lastly, we find that large and small subject systems are equally predictable, showing promise for the scalability of the predictability problem., This work was funded by valgrAI, the Future of Life Institute (FLI) under grant RFP2-152, the EU (FEDER) and Spanish grant RTI2018-094403-B-C32 funded by MCIN/AEI/10.13039/501100011033 and by CIPROM/2022/6 funded by Generalitat Valenciana, FISCALTICS (I+D+i PID2022-140110OA-I00) granted by MICIU/AEI/10.13039/501100011033, the Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/501100011033 and 'ERDF A way of making Europe, as well as the European Union under agreement INNEST/2021/317 (Neurocalcat) and by the Vic. Inv. of the Universitat Politecnica de Valencia under DOCEMPR21, DOCEMPR22, and DOCEMPR23'.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Larger and more instructable language models become less reliable

  • Zhou, Lexin
  • Schellaert, Wout Willy M.
  • Martínez-Plumed, Fernando|||0000-0003-2902-6477
  • Moros-Daval, Yael
  • Ferri Ramírez, César|||0000-0002-8975-1120
  • Hernández-Orallo, José|||0000-0001-9746-7632
[EN] The prevailing methods to make large language models more powerful and amenable have been based on continuous scaling up (that is, increasing their size, data volume and computational resources1 ) and bespoke shaping up (including post-fltering2,3 , fne tuning or use of human feedback4,5 ). However, larger and more instructable large language models may have become less reliable. By studying the relationship between difculty concordance, task avoidance and prompting stability of several language model families, here we show that easy instances for human participants are also easy for the models, but scaled-up, shaped-up models do not secure areas of low difculty in which either the model does not err or human supervision can spot the errors. We also fnd that early models often avoid user questions but scaled-up, shaped-up models tend to give an apparently sensible yet wrong answer much more often, including errors on difcult questions that human supervisors frequently overlook. Moreover, we observe that stability to diferent natural phrasings of the same question is improved by scaling-up and shaping-up interventions, but pockets of variability persist across difculty levels. These fndings highlight the need for a fundamental shift in the design and development of general-purpose artifcial intelligence, particularly in high-stakes areas for which a predictable distribution of errors is paramount., We acknowledge support from the following institutions: Long-Term Future Scholarship funded by Open Philanthropy; Grant for Master Studies funded by ValgrAI Valencian Graduate School and Research Network for Artificial Intelligence and Generalitat Valenciana; FISCALTICS (I+D+i PID2022-140110OA-I00), funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU; CIPROM/2022/6 (FASSLOW) and IDIFEDER/2021/05 (CLUSTERIA) funded by Generalitat Valenciana; the EC H2020-EU grant agreement no. 952215 (TAILOR); US DARPA HR00112120007 (RECoG-AI) and Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe ; and the Vic. Inv. of the Universitat Politècnica de Valencia under DOCEMPR21, DOCEMPR22 and DOCEMPR23.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Two Optimized Methods for Efficient, Stable and Transient Transformation of Broccoli (Brassica oleracea Var. Italica)

  • Coronado-Martín, Alberto
  • Atarés Huerta, Alejandro|||0000-0003-1728-5471
  • Porcel, R.|||0000-0001-8068-741X
  • Yenush, Lynne|||0000-0001-8589-7002
  • Mulet, José Miguel|||0000-0002-9087-3838
[EN] Broccoli (Brassica oleracea var. italica) is an important crop valued for its nutritional and health-promoting properties, yet its biotechnological improvement is limited by low effectivity and genotype-dependent transformation protocols. The absence of reliable transient expression systems further constrains functional genomics and genome-editing applications. Here, we optimized regeneration and transformation protocols for different broccoli genotypes. Endoreduplication patterns in young tissues were analyzed by flow cytometry to identify suitable explants, and combinations of plant growth regulators were tested to develop an efficient organogenic medium. Stable transformation was achieved via Agrobacterium tumefaciens using nptII and eGFP markers. Cotyledons and hypocotyls up to day 7 showed similar endoreduplication patterns, with abundant 2n cells, but hypocotyls exhibited higher regeneration capacity. The optimized medium supported efficient organogenesis while maintaining diploidy. Transformation efficiency reached 10.4% in `S1¿ and 2.8% in `Naxos¿, highlighting genotype dependence. In parallel, a transient expression system was established using cotyledon-derived protoplasts and electroporation-mediated DNA delivery. GFP expression was confirmed through fluorescence microscopy, confocal imaging, and Western blotting. These protocols provide a robust toolkit for broccoli genetic manipulation, facilitating molecular biology studies in the native plant, functional genomics and genome-editing strategies, including CRISPR-based approaches., This work was supported by the convocatoria de ayudas a Proyectos de Colaboración público-privada 2022: CPP2022-009860, financiado por el MCIN/AEI/10.13039/501100011033 y por la Unión Europea NextGenerationEU/PRTR. Alberto Coronado was supported by a predoctoral contract from the Generalitat Valenciana, co-financed by the Fondo Social Europeo (FSE+) from the program Contratos predoctorales para la formación de doctores en colaboración con empresas (20210128, DOCEMPR21).