RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación
Contribución de congreso (ConferenceOutput)
Language Task Difficulty Prediction through LLM-Annotated Meta-Features
[EN] Assessing the capabilities of large language models
(LLMs) is increasingly challenging due to their generality and uneven task performance. Often, we do not know how much of the success or failure on a particular task is due to the `loading¿ of the language elements in the task, such as narrative understanding, or some
other intrinsic (non-linguistic) components, such as domain-specific
common sense or reasoning capabilities. Understanding what tasks
are most loaded on language and determine the predictability of
LLMs on these tasks is crucial for improving benchmarks, designing better LLMs, and ensuring their safe deployment. We present an
innovative methodology that uses LLMs to annotate linguistic metafeatures , allowing us to predict task difficulty and understand linguistic loadings more accurately than traditional readability scores.
Using GPT-4 for automated annotation, we show strong predictability for a variety of tasks and language models (e.g., MMLU with R2
from 0.68 to 0.83), but observe limited predictability for other tasks
(e.g., LSAT with R2 of -0.07)., We acknowledge support from: Grant for Master Studies funded by
ValgrAI Valencian Graduate School and Research Network for Artificial Intelligence and Generalitat Valenciana; FISCALTICS (I+D+i
PID2022-140110OA-I00), granted by MICIU/AEI/10.13039/
501100011033 and by ERDF, EU; CIPROM/2022/6 (FASSLOW)
and IDIFEDER/2021/05 (CLUSTERIA) funded by Generalitat
Valenciana; the EC H2020-EU grant agreement No. 952215
(TAILOR); US DARPA HR00112120007 (RECoG-AI) and
Spanish grant PID2021-122830OB-C42 (SFERA) funded by
MCIN/AEI/10.13039/501100011033 and ERDF.
Proyecto:
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2021-122830OB-C42
,
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2022-140110OA-I00
,
European Commission / H2020 952215
,
Generalitat Valenciana CIPROM/2022/6
,
Generalitat Valenciana IDIFEDER/2021/05
,
U.S. Department of Defense HR00112120007