Repositorio Digital de la UPF
Publicación
Contribución de congreso (ConferenceOutput)
Not a nuisance but a useful heuristic: outlier dimensions favor frequent tokens in language models
We study last-layer outlier dimensions, i.e. dimensions that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern language models, and trace their function back to the heuristic of constantly predicting frequent words. We further show how a model can block this heuristic when it is not contextually appropriate, by assigning a counterbalancing weight mass to the remaining dimensions, and we investigate which model parameters boost outlier dimensions and when they arise during training. We conclude that outlier dimensions are a specialized mechanism discovered by many distinct models to implement a useful token prediction heuristic., We thank Beatrix Miranda Ginn Nielsen, Santiago Acevedo, Diego Doimo, Javier Ferrando, Alessandro Laio and the members of the UPF COLT group for feedback. Our work was funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 101019291). We also received funding from the Catalan government (AGAUR grant SGR 2021 00470). NG also received the support of a fellowship from Fundación Ramón Areces. GB also received the support of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación (Spain). This paper reflects the authors’ view only, and the funding agencies are not responsible for any use that may be made of the information it contains.