Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 12
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Not a nuisance but a useful heuristic: outlier dimensions favor frequent tokens in language models

  • Macocco, Iuri
  • Graichen, Nora
  • Boleda, Gemma
  • Baroni, Marco
We study last-layer outlier dimensions, i.e. dimensions that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern language models, and trace their function back to the heuristic of constantly predicting frequent words. We further show how a model can block this heuristic when it is not contextually appropriate, by assigning a counterbalancing weight mass to the remaining dimensions, and we investigate which model parameters boost outlier dimensions and when they arise during training. We conclude that outlier dimensions are a specialized mechanism discovered by many distinct models to implement a useful token prediction heuristic., We thank Beatrix Miranda Ginn Nielsen, Santiago Acevedo, Diego Doimo, Javier Ferrando, Alessandro Laio and the members of the UPF COLT group for feedback. Our work was funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 101019291). We also received funding from the Catalan government (AGAUR grant SGR 2021 00470). NG also received the support of a fellowship from Fundación Ramón Areces. GB also received the support of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación (Spain). This paper reflects the authors’ view only, and the funding agencies are not responsible for any use that may be made of the information it contains.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Cross-domain image captioning with discriminative finetuning

  • Dessì, Roberto
  • Bevilacqua, Michele
  • Gualdoni, Eleonora
  • Rakotonirina, Nathanael Carraz
  • Franzon, Francesca
  • Baroni, Marco
Comunicació presentada a: CVPR 2023. IEEE/CVF Conference on Computer Vision and Pattern Recognition celebrat del 18 al 22 de juny a Vancouver, Canada., Neural captioners are typically trained to mimic humangenerated references without optimizing for any specific communication goal, leading to problems such as the generation of vague captions. In this paper, we show that fine-tuning an out-of-the-box neural captioner with a selfsupervised discriminative communication objective helps to recover a plain, visually descriptive language that is more informative about image contents. Given a target image, the system must learn to produce a description that enables an out-of-the-box text-conditioned image retriever to identify such image among a set of candidates. We experiment with the popular ClipCap captioner, also replicating the main results with BLIP. In terms of similarity to groundtruth human descriptions, the captions emerging from discriminative finetuning lag slightly behind those generated by the non-finetuned model, when the latter is trained and tested on the same caption dataset. However, when the model is used without further tuning to generate captions for out-of-domain datasets, our discriminatively-finetuned captioner generates descriptions that resemble human references more than those produced by the same captioner without finetuning. We further show that, on the Conceptual Captions dataset, discriminatively finetuned captions are more helpful than either vanilla ClipCap captions or ground-truth captions for human annotators tasked with an image discrimination task, EG, NCR, FF and MB received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements No. 715154 and No. 101019291) and the Spanish Research Agency (ref. PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Rhetorically-based scalar-additivity: the view from Italian addirittura

  • Pistoia-Reda, Salvatore
  • McNally, Louise, 1965-
Even-like particles have widely been analyzed as inducing scalar and additive presuppositions (cf. Horn 1969; Karttunen & Peters 1979; Rooth 1992; Gast & van der Auwera 2011). However, the additivity of even has been controversial since at least Rullmann 1997 and increasingly called into question (see Greenberg & Umbach 2021 for references); Greenberg specifically argues that scalar even-like particles can vary in additivity. This claim is surprising in light of the typological study in Gast & van der Auwera 2011, which subsumes even and similar expressions under a larger class of additive particles. Against this background, we present an analysis of Italian addirittura, which with perfino has been described as scalaradditive (Visconti 2005) – but only optionally so – and is chosen preferentially over perfino precisely in those contexts that Greenberg takes to challenge the additivity of even. We argue, drawing on observations in Atayan 2017, that addirittura contrasts with perfino in deriving its scalar alternatives from rhetorical structure rather than focus structure. Once this is recognized we can view addirittura as additive, after all, in a rhetorical sense we describe below., This work was supported by AGAUR Beatriu de Pinós grant 2018BP00070 and grant PID2020-112602GB-I00/MICIN/AEI/10.13039/501100011033 financed by the Spanish Ministerio de Ciencia e Innovación and Agencia Estatal de Investigación.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Run Like a Girl! Sport-Related Gender Bias in Language and Vision

  • Harrison, Sophie
  • Gualdoni, Eleonora
  • Boleda, Gemma
Gender bias in Language and Vision datasets and models has the potential to perpetuate harmful stereotypes and discrimination. We analyze gender bias in two Language and Vision datasets. Consistent with prior work, we find that both datasets underrepresent women, which promotes their invisibilization. Moreover, we hypothesize and find that a bias affects human naming choices for people playing sports: speakers produce names indicating the sport (e.g. ‘tennis player’ or ‘surfer’) more often when it is a man or a boy participating in the sport than when it is a woman or a girl, with an average of 46% vs. 35% of sports-related names for each gender. A computational model trained on these naming data reproduces the bias. We argue that both the data and the model result in representational harm against women., This research is an output of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación (Spain) and of grant agreement No. 715154 funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Communication breakdown: on the low mutual intelligibility between human and neural captioning

  • Dessì, Roberto
  • Gualdoni, Eleonora
  • Franzon, Francesca
  • Boleda, Gemma
  • Baroni, Marco
Comunicació presentada a la 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), celebrada del 7 a l'11 de desembre de 2022 a Abu Dhabi, Emirats Àrabs Units., We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced IMAGECODE data-set (Krojer et al., 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval performance is almost at chance level. Our results thus add to the growing body of evidence that, even when the “language” of neural models resembles English, this superficial resemblance might be deeply misleading., This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements No. 715154 and No. 101019291) and the Spanish Research Agency (ref. PID2020-112602GB-I00).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Quantifying informativeness of names in visual space

  • Gualdoni, Eleonora
  • Kemp, Charles
  • Xu, Yang
  • Boleda, Gemma
Comunicació presentada a 45th Annual Conference of the Cognitive Science Society (CogSci 2023), celebrada a Sydney (Austràlia) del 26 al 29 de juliol de 2023., The human lexicon expresses a wide array of concepts with a limited set of words. Previous work has suggested that semantic categories are structured compactly to enable informative communication. Informativeness is typically quantified with respect to an entire semantic domain and not at the level of individual names. We develop a measure of name informativeness using an information-theoretic framework grounded in visual object representations derived from natural images. Our approach uses computer vision models to characterize informativeness of individual names with respect to large-scale data in a naturalistic setting. We show that our informativeness measure predicts degrees of specificity in lexical categories more precisely than alternative measures based on entropy and frequency. We also show that name informativeness jointly captures within-category similarity and distinctiveness across categories. Our analyses suggest how the variability of names from a broad part of the lexicon may be understood through the lens of information theory., This research is an output of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovacion and the Agencia Estatal ´ de Investigacion (Spain), of grant agreement No. 715154 ´ funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, and of NSERC Alliance International-Catalyst Grant ALLRP 576149-22. The second author was supported by grant FT190100200 from the Australian Research Council.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Rhetorically-based scalar-additivity: the view from Italian addirittura

  • Pistoia-Reda, Salvatore
  • McNally, Louise, 1965-
Even-like particles have widely been analyzed as inducing scalar and additive presuppositions (cf. Horn 1969; Karttunen & Peters 1979; Rooth 1992; Gast & van der Auwera 2011). However, the additivity of even has been controversial since at least Rullmann 1997 and increasingly called into question (see Greenberg & Umbach 2021 for references); Greenberg specifically argues that scalar even-like particles can vary in additivity. This claim is surprising in light of the typological study in Gast & van der Auwera 2011, which subsumes even and similar expressions under a larger class of additive particles. Against this background, we present an analysis of Italian addirittura, which with perfino has been described as scalaradditive (Visconti 2005) – but only optionally so – and is chosen preferentially over perfino precisely in those contexts that Greenberg takes to challenge the additivity of even. We argue, drawing on observations in Atayan 2017, that addirittura contrasts with perfino in deriving its scalar alternatives from rhetorical structure rather than focus structure. Once this is recognized we can view addirittura as additive, after all, in a rhetorical sense we describe below., This work was supported by AGAUR Beatriu de Pinós grant 2018BP00070 and grant PID2020-112602GB-I00/MICIN/AEI/10.13039/501100011033 financed by the Spanish Ministerio de Ciencia e Innovación and Agencia Estatal de Investigación.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Quantifying informativeness of names in visual space

  • Gualdoni, Eleonora
  • Kemp, Charles
  • Xu, Yang
  • Boleda, Gemma
Comunicació presentada a 45th Annual Conference of the Cognitive Science Society (CogSci 2023), celebrada a Sydney (Austràlia) del 26 al 29 de juliol de 2023., The human lexicon expresses a wide array of concepts with a limited set of words. Previous work has suggested that semantic categories are structured compactly to enable informative communication. Informativeness is typically quantified with respect to an entire semantic domain and not at the level of individual names. We develop a measure of name informativeness using an information-theoretic framework grounded in visual object representations derived from natural images. Our approach uses computer vision models to characterize informativeness of individual names with respect to large-scale data in a naturalistic setting. We show that our informativeness measure predicts degrees of specificity in lexical categories more precisely than alternative measures based on entropy and frequency. We also show that name informativeness jointly captures within-category similarity and distinctiveness across categories. Our analyses suggest how the variability of names from a broad part of the lexicon may be understood through the lens of information theory., This research is an output of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovacion and the Agencia Estatal ´ de Investigacion (Spain), of grant agreement No. 715154 ´ funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, and of NSERC Alliance International-Catalyst Grant ALLRP 576149-22. The second author was supported by grant FT190100200 from the Australian Research Council.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Cross-domain image captioning with discriminative finetuning

  • Dessì, Roberto
  • Bevilacqua, Michele
  • Gualdoni, Eleonora
  • Rakotonirina, Nathanael Carraz
  • Franzon, Francesca
  • Baroni, Marco
Comunicació presentada a: CVPR 2023. IEEE/CVF Conference on Computer Vision and Pattern Recognition celebrat del 18 al 22 de juny a Vancouver, Canada., Neural captioners are typically trained to mimic humangenerated references without optimizing for any specific communication goal, leading to problems such as the generation of vague captions. In this paper, we show that fine-tuning an out-of-the-box neural captioner with a selfsupervised discriminative communication objective helps to recover a plain, visually descriptive language that is more informative about image contents. Given a target image, the system must learn to produce a description that enables an out-of-the-box text-conditioned image retriever to identify such image among a set of candidates. We experiment with the popular ClipCap captioner, also replicating the main results with BLIP. In terms of similarity to groundtruth human descriptions, the captions emerging from discriminative finetuning lag slightly behind those generated by the non-finetuned model, when the latter is trained and tested on the same caption dataset. However, when the model is used without further tuning to generate captions for out-of-domain datasets, our discriminatively-finetuned captioner generates descriptions that resemble human references more than those produced by the same captioner without finetuning. We further show that, on the Conceptual Captions dataset, discriminatively finetuned captions are more helpful than either vanilla ClipCap captions or ground-truth captions for human annotators tasked with an image discrimination task, EG, NCR, FF and MB received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements No. 715154 and No. 101019291) and the Spanish Research Agency (ref. PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Run Like a Girl! Sport-Related Gender Bias in Language and Vision

  • Harrison, Sophie
  • Gualdoni, Eleonora
  • Boleda, Gemma
Gender bias in Language and Vision datasets and models has the potential to perpetuate harmful stereotypes and discrimination. We analyze gender bias in two Language and Vision datasets. Consistent with prior work, we find that both datasets underrepresent women, which promotes their invisibilization. Moreover, we hypothesize and find that a bias affects human naming choices for people playing sports: speakers produce names indicating the sport (e.g. ‘tennis player’ or ‘surfer’) more often when it is a man or a boy participating in the sport than when it is a woman or a girl, with an average of 46% vs. 35% of sports-related names for each gender. A computational model trained on these naming data reproduces the bias. We argue that both the data and the model result in representational harm against women., This research is an output of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación (Spain) and of grant agreement No. 715154 funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Communication breakdown: on the low mutual intelligibility between human and neural captioning

  • Dessì, Roberto
  • Gualdoni, Eleonora
  • Franzon, Francesca
  • Boleda, Gemma
  • Baroni, Marco
Comunicació presentada a la 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), celebrada del 7 a l'11 de desembre de 2022 a Abu Dhabi, Emirats Àrabs Units., We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced IMAGECODE data-set (Krojer et al., 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval performance is almost at chance level. Our results thus add to the growing body of evidence that, even when the “language” of neural models resembles English, this superficial resemblance might be deeply misleading., This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements No. 715154 and No. 101019291) and the Spanish Research Agency (ref. PID2020-112602GB-I00).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Not a nuisance but a useful heuristic: outlier dimensions favor frequent tokens in language models

  • Macocco, Iuri
  • Graichen, Nora
  • Boleda, Gemma
  • Baroni, Marco
We study last-layer outlier dimensions, i.e. dimensions that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern language models, and trace their function back to the heuristic of constantly predicting frequent words. We further show how a model can block this heuristic when it is not contextually appropriate, by assigning a counterbalancing weight mass to the remaining dimensions, and we investigate which model parameters boost outlier dimensions and when they arise during training. We conclude that outlier dimensions are a specialized mechanism discovered by many distinct models to implement a useful token prediction heuristic., We thank Beatrix Miranda Ginn Nielsen, Santiago Acevedo, Diego Doimo, Javier Ferrando, Alessandro Laio and the members of the UPF COLT group for feedback. Our work was funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 101019291). We also received funding from the Catalan government (AGAUR grant SGR 2021 00470). NG also received the support of a fellowship from Fundación Ramón Areces. GB also received the support of grant PID2020-112602GBI00/MICIN/AEI/10.13039/501100011033, funded by the Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación (Spain). This paper reflects the authors’ view only, and the funding agencies are not responsible for any use that may be made of the information it contains.