RUC. Repositorio da Universidade da Coruña
Publicación
Contribución de congreso (ConferenceOutput)
Assessment of Pre-Trained Models Across Languages and Grammars
Bali, Indonesia. November, 1-4 2023., [Absctract]: We present an approach for assessing how
multilingual large language models (LLMs)
learn syntax in terms of multi-formalism syntactic structures. We aim to recover constituent
and dependency structures by casting parsing
as sequence labeling. To do so, we select a
few LLMs and study them on 13 diverse UD
treebanks for dependency parsing and 10 treebanks for constituent parsing. Our results show
that: (i) the framework is consistent across encodings, (ii) pre-trained word vectors do not
favor constituency representations of syntax
over dependencies, (iii) sub-word tokenization
is needed to represent syntax, in contrast to
character-based models, and (iv) occurrence
of a language in the pretraining data is more
important than the amount of task data when
recovering syntax from the word vectors., We acknowledge the European Research
Council (ERC), which has funded this research under the Horizon Europe research and
innovation programme (SALSA, grant agreement No 101100615), ERDF/MICINN-AEI
(SCANNER-UDC, PID2020-113230RB-C21),
Xunta de Galicia (ED431C 2020/11), grant
FPI 2021 (PID2020-113230RB-C21) funded by
MCIN/AEI/10.13039/501100011033, and Centro
de Investigación de Galicia “CITIC”, funded by
the Xunta de Galicia through the collaboration
agreement between the Consellería de Cultura,
Educación, Formación Profesional e Universidades
and the Galician universities for the reinforcement
of the research centres of the Galician University
System (CIGUS)., Xunta de Galicia; ED431C2020/11
Proyecto:
EC / HE 101100615
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2020-113230RB-C21
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2022-1393080A-100
HANDLE: http://hdl.handle.net/2183/36572
Ver en: http://hdl.handle.net/2183/36572