Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 26
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

"The red one!": on learning to refer to things based on discriminative properties

  • Lazaridou, Angeliki
  • Pham, Nghia The
  • Baroni, Marco
Comunicació presentada a: 54th Annual Meeting of the Association for Computational Linguistics celebrat del 7 al 12 d'agost de 2016 a Berlín, Alemanya., As a first step towards agents learning to communicate about their visual environment, we propose a system that, given visual representations of a referent (CAT) and a context (SOFA), identifies their discriminative attributes, i.e., properties that distinguish them (has_tail). Moreover, although supervision is only provided in terms of discriminativeness of attributes for pairs, the model learns to assign plausible attributes to specific objects (SOFA-has_cushion). Finally, we present a preliminary experiment confirming the referential success of the predicted discriminative attributes., This work was supported by ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Distributional vectors encode referential attributes

  • Boleda, Gemma
  • Gupta, Abhijeet
  • Baroni, Marco
  • Padó, Sebastian
Distributional methods have proven to excel at capturing fuzzy, graded aspects of meaning (Italy is more similar to Spain than to Germany). In contrast, it is difficult to extract the values of more specific attributes of word referents from distributional representations, attributes of the kind typically found in structured knowledge bases (Italy has 60 million inhabitants). In this paper, we pursue the hypothesis that distributional vectors also implicitly encode referential attributes. We show that a standard supervised regression model is in fact sufficient to retrieve such attributes to a reasonable degree of accuracy: When evaluated on the prediction of both categorical and numeric attributes of countries and cities, the model consistently reduces baseline error by 30%, and is not far from the upper bound. Further analysis suggests that our model is able to “objectify” distributional representations for entities, anchoring them more firmly in the external world in measurable ways., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model

  • Pham, Nghia The
  • Kruszewski, German
  • Lazaridou, Angeliki
  • Baroni, Marco
Comunicació presentada a: 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing celebrat del 26 al 31 de juliol de 2015 a Pequín, Xina., We introduce C-PHRASE, a distributional semantic model that learns word representations by optimizing context prediction for phrases at all levels in a syntactic tree, from single words to full sentences. C-PHRASE outperforms the state-of-theart C-BOW model on a variety of lexical tasks. Moreover, since C-PHRASE word vectors are induced through a compositional learning objective (modeling the contexts of words combined into phrases), when they are summed, they produce sentence representations that rival those generated by ad-hoc compositional models., We thank Gemma Boleda and the anonymous reviewers for useful comments. We acknowledge ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Convolutional neural network language models

  • Boleda, Gemma
  • Pham, Nghia The
  • Kruszewski, German
Convolutional Neural Networks (CNNs) have shown to yield very strong results in several Computer Vision tasks. Their application to language has received much less attention, and it has mainly focused on static classification tasks, such as sentence classification for Sentiment Analysis or relation extraction. In this work, we study the application of CNNs to language modeling, a dynamic, sequential prediction task that needs models to capture local as well as long-range dependency information. Our contribution is twofold. First, we show that CNNs achieve 11-26% better absolute performance than feed-forward neural language models, demonstrating their potential for language representation even in sequential tasks. As for recurrent models, our model outperforms RNNs but is below state of the art LSTM models. Second, we gain some understanding of the behavior of the model, showing that CNNs in language act as feature detectors at a high level of abstraction, like in Computer Vision, and that the model can profitably use information from as far as 16 words before the target., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES) and the Erasmus Mundus Scholarship for Joint Master Programs.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

“Look, some green circles!”: learning to quantify from images

  • Boleda, Gemma
  • Sorodoc, Ionut-Teodor
  • Lazaridou, Angeliki
  • Herbelot, Aurélie
  • Pezzelle, Sandro
  • Bernardi, Raffaella
In this paper, we investigate whether a neural network model can learn the meaning of natural language quantifiers (no,some and all) from their use in visual contexts. We show that memory networks perform well in this task, and that explicit counting is not necessary to the system’s performance, supporting psycholinguistic evidence on the acquisition of quantifiers., This project has received funding from the European Unions Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

“Show me the cup”: reference with continuous representations

  • Boleda, Gemma
  • Baroni, Marco
  • Padó, Sebastian
Comunicació presentada a CICLing 2017, celebrat a Budapest (Hongria) del 17 al 23 d'abril de 2017., One of the most basic functions of language is to refer to objects in a shared scene. Modeling reference with continuous representations is challenging because it requires individuation, i.e., tracking and distinguishing an arbitrary number of referents. We introduce a neural network model that, given a de nite description and a set of objects represented by natural images, points to the intended object if the expression has a unique referent, or indicates a failure, if it does not. The model, directly trained on reference acts, is competitive with a pipeline manually engineered to perform the same task, both when referents are purely visual, and when they are characterized by a combination of visual and linguistic properties., This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe) and ERC grant agreement No 715154 (AMORE); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Distributional semantics in use

  • Boleda, Gemma
  • Bernardi, Raffaella
  • Fernández, Raquel
  • Paperno, Denis
In this position paper we argue that an adequate semantic model must account for language in use, taking into account how discourse context affects the meaning of words and larger linguistic units. Distributional semantic models are very attractive models of meaning mainly because they capture conceptual aspects and are automatically induced from natural language data. However, they need to be extended in order to account for language use in a discourse or dialogue context. We discuss phenomena that the new generation of distributional semantic models should capture, and propose concrete tasks on which they could be tested., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe), FP7 programme ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES), and the Netherlands Organization for Scientific Research (NWO).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Leveraging preposition ambiguity to assess compositional distributional models of semantics

  • Ritter, Samuel
  • Long, Cotie
  • Paperno, Denis
  • Baroni, Marco
  • Botvinick, Matthew
  • Goldberg, Adele
Comunicació presentada a: Fourth Joint Conference on Lexical and Computational Semantics celebrat del 4 al 5 de juny de 2015 a Denver, EUA., Complex interactions among the meanings of words are important factors in the function that maps word meanings to phrase meanings. Recently, compositional distributional semantics models (CDSM) have been designed with the goal of emulating these complex interactions; however, experimental results on the effectiveness of CDSM have been difficult to interpret because the current metrics for assessing them do not control for the confound of lexical information. We present a new method for assessing the degree to which CDSM capture semantic interactions that dissociates the influences of lexical and compositional information. We then provide a dataset for performing this type of assessment and use it to evaluate six compositional models using both co-occurrence based and neural language model input vectors. Results show that neural language input vectors are consistently superior to co-occurrence based vectors, that several CDSM capture substantial compositional information, and that, surprisingly, vector addition matches and is in many cases superior to purpose-built paramaterized models., Denis Paperno and Marco Baroni were supported by ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES). Samuel Ritter and Matthew Botvinick were supported by Intelligence Advanced Research Projects Activity (IARPA) Grant n. 102-01.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

First-order vs. higher-order modification in distributional semantics

  • Boleda, Gemma
  • Vecchi, Eva Maria
  • Cornudella Gaya, Miquel
  • McNally, Louise, 1965-
Comunicació presentada a Empirical Methods in Natural Language Processing and Computational Natural Language Learning, celebrada del 12 al 14 de juliol de 2012, a Jeju Island, Korea., Adjectival modification, particularly by expressions that have been treated as higher-order modifiers in the formal semantics tradition, raises interesting challenges for semantic composition in distributional semantic models. We contrast three types of adjectival modifiers -- intersectively used color terms (as in white towel, clearly first-order), subsectively used color terms (white wine, which have been modeled as both first- and higher-order), and intensional adjectives (former bassist, clearly higher-order) -- and test the ability of different composition strategies to model their behavior. In addition to opening up a new empirical domain for research on distributional semantics, our observations concerning the attested vectors for the different types of adjectives, the nouns they modify, and the resulting noun phrases yield insights into modification that have been little evident in the formal semantics literature to date., This research was funded by the Spanish MICINN (FFI2010-09464-E, FFI2010-15006, TIN2009-14715-C04-04), the Catalan AGAUR (2010BPA00070), and the EU (PASCAL2; FP7-ICT-216886). Eva Maria Vecchi was partially funded by ERC Starting Grant 283554.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

The LAMBADA dataset: word prediction requiring a broad discourse context

  • Boleda, Gemma
  • Paperno, Denis
  • Kruszewski, German
  • Lazaridou, Angeliki
  • Pham, Quan Ngoc
  • Bernardi, Raffaella
  • Pezzelle, Sandro
  • Baroni, Marco
  • Fernandez, Raquel
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-ofthe-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); NWO VIDI grant n. 276-89-008 (Asymmetry in Conversation).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Intensionality was only alleged: on adjective-noun composition in distributional semantics

  • Boleda, Gemma
  • Baroni, Marco
  • Pham, Nghia The
  • McNally, Louise, 1965-
Distributional semantics has very successfully modeled semantic phenomena at the word level, and recently interest has grown in extending it to capture the meaning of phrases via semantic composition. We present experiments in adjective-noun composition which (1) show that adjectival modification can be successfully modeled with distributional semantics, (2) show that composition models inspired by the semantics of higher-order predication fare better than those that perform simple feature union or intersection, (3) contrary to what the theoretical literature might lead one to expect, do not yield a distinction between intensional and non-intensional modification, and (4) suggest that head noun polysemy and whether the adjective corresponds to a typical attribute of the noun are relevant factors in the distributional representation of adjective phrases., We acknowledge the support of Spanish MICINN grant FFI2010-09464-E (McNally, Boleda), the ICREA Foundation (McNally), Catalan AGAUR grant 2010BP-A00070, MICINN grant TIN2009-14715-C04-04, EU grant PASCAL2, FP7-ICT-216886, the DARPA DEFT program under AFRL grant FA8750-13-2-0026 (Boleda) and the ERC under the 2011 Starting Independent Research Grant 283554 to the COMPOSES project (Baroni, Pham). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of DARPA, AFRL or the US government.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

High-risk learning: acquiring new word vectors from tiny data

  • Herbelot, Aurélie
  • Baroni, Marco
Comunicació presentada a: 2017 Conference on Empirical Methods in Natural Language Processing celebrat del 7 al 11 de setembre de 2017 a Copenhaguen, Dinamarca., Distributional semantics models are known to struggle with small data. It is generally accepted that in order to learn ‘a good vector’ for a word, a model must have sufficient examples of its usage. This contradicts the fact that humans can guess the meaning of a word from a few occurrences only. In this paper, we show that a neural language model such as Word2Vec only necessitates minor modifications to its standard architecture to learn new terms from tiny data, using background knowledge from a previously learnt semantic space. We test our model on word definitions and on a nonce task involving 2-6 sentences’ worth of context, showing a large increase in performance over state-of-the-art models on the definitional task., We acknowledge ERC 2011 Starting Independent Research Grant No 283554 (COMPOSES). This project has also received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 751250.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Hubness and pollution: delving into cross-space mapping for zero-shot learning

  • Lazaridou, Angeliki
  • Dinu, Georgiana
  • Baroni, Marco
Comunicació presentada a: 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing celebrat del 26 al 31 de juliol de 2015 a Pequín, Xina., Zero-shot methods in language, vision and other domains rely on a cross-space mapping function that projects vectors from the relevant feature space (e.g., visualfeature-based image representations) to a large semantic word space (induced in an unsupervised way from corpus data), where the entities of interest (e.g., objects images depict) are labeled with the words associated to the nearest neighbours of the mapped vectors. Zero-shot cross-space mapping methods hold great promise as a way to scale up annotation tasks well beyond the labels in the training data (e.g., recognizing objects that were never seen in training). However, the current performance of cross-space mapping functions is still quite low, so that the strategy is not yet usable in practical applications. In this paper, we explore some general properties, both theoretical and empirical, of the cross-space mapping function, and we build on them to propose better methods to estimate it. In this way, we attain large improvements over the state of the art, both in cross-linguistic (word translation) and cross-modal (image labeling) zero-shot experiments., We acknowledge ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

"The red one!": on learning to refer to things based on discriminative properties

  • Lazaridou, Angeliki
  • Pham, Nghia The
  • Baroni, Marco
Comunicació presentada a: 54th Annual Meeting of the Association for Computational Linguistics celebrat del 7 al 12 d'agost de 2016 a Berlín, Alemanya., As a first step towards agents learning to communicate about their visual environment, we propose a system that, given visual representations of a referent (CAT) and a context (SOFA), identifies their discriminative attributes, i.e., properties that distinguish them (has_tail). Moreover, although supervision is only provided in terms of discriminativeness of attributes for pairs, the model learns to assign plausible attributes to specific objects (SOFA-has_cushion). Finally, we present a preliminary experiment confirming the referential success of the predicted discriminative attributes., This work was supported by ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

First-order vs. higher-order modification in distributional semantics

  • Boleda, Gemma
  • Vecchi, Eva Maria
  • Cornudella Gaya, Miquel
  • McNally, Louise, 1965-
Comunicació presentada a Empirical Methods in Natural Language Processing and Computational Natural Language Learning, celebrada del 12 al 14 de juliol de 2012, a Jeju Island, Korea., Adjectival modification, particularly by expressions that have been treated as higher-order modifiers in the formal semantics tradition, raises interesting challenges for semantic composition in distributional semantic models. We contrast three types of adjectival modifiers -- intersectively used color terms (as in white towel, clearly first-order), subsectively used color terms (white wine, which have been modeled as both first- and higher-order), and intensional adjectives (former bassist, clearly higher-order) -- and test the ability of different composition strategies to model their behavior. In addition to opening up a new empirical domain for research on distributional semantics, our observations concerning the attested vectors for the different types of adjectives, the nouns they modify, and the resulting noun phrases yield insights into modification that have been little evident in the formal semantics literature to date., This research was funded by the Spanish MICINN (FFI2010-09464-E, FFI2010-15006, TIN2009-14715-C04-04), the Catalan AGAUR (2010BPA00070), and the EU (PASCAL2; FP7-ICT-216886). Eva Maria Vecchi was partially funded by ERC Starting Grant 283554.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Intensionality was only alleged: on adjective-noun composition in distributional semantics

  • Boleda, Gemma
  • Baroni, Marco
  • Pham, Nghia The
  • McNally, Louise, 1965-
Distributional semantics has very successfully modeled semantic phenomena at the word level, and recently interest has grown in extending it to capture the meaning of phrases via semantic composition. We present experiments in adjective-noun composition which (1) show that adjectival modification can be successfully modeled with distributional semantics, (2) show that composition models inspired by the semantics of higher-order predication fare better than those that perform simple feature union or intersection, (3) contrary to what the theoretical literature might lead one to expect, do not yield a distinction between intensional and non-intensional modification, and (4) suggest that head noun polysemy and whether the adjective corresponds to a typical attribute of the noun are relevant factors in the distributional representation of adjective phrases., We acknowledge the support of Spanish MICINN grant FFI2010-09464-E (McNally, Boleda), the ICREA Foundation (McNally), Catalan AGAUR grant 2010BP-A00070, MICINN grant TIN2009-14715-C04-04, EU grant PASCAL2, FP7-ICT-216886, the DARPA DEFT program under AFRL grant FA8750-13-2-0026 (Boleda) and the ERC under the 2011 Starting Independent Research Grant 283554 to the COMPOSES project (Baroni, Pham). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of DARPA, AFRL or the US government.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

High-risk learning: acquiring new word vectors from tiny data

  • Herbelot, Aurélie
  • Baroni, Marco
Comunicació presentada a: 2017 Conference on Empirical Methods in Natural Language Processing celebrat del 7 al 11 de setembre de 2017 a Copenhaguen, Dinamarca., Distributional semantics models are known to struggle with small data. It is generally accepted that in order to learn ‘a good vector’ for a word, a model must have sufficient examples of its usage. This contradicts the fact that humans can guess the meaning of a word from a few occurrences only. In this paper, we show that a neural language model such as Word2Vec only necessitates minor modifications to its standard architecture to learn new terms from tiny data, using background knowledge from a previously learnt semantic space. We test our model on word definitions and on a nonce task involving 2-6 sentences’ worth of context, showing a large increase in performance over state-of-the-art models on the definitional task., We acknowledge ERC 2011 Starting Independent Research Grant No 283554 (COMPOSES). This project has also received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 751250.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Distributional vectors encode referential attributes

  • Boleda, Gemma
  • Gupta, Abhijeet
  • Baroni, Marco
  • Padó, Sebastian
Distributional methods have proven to excel at capturing fuzzy, graded aspects of meaning (Italy is more similar to Spain than to Germany). In contrast, it is difficult to extract the values of more specific attributes of word referents from distributional representations, attributes of the kind typically found in structured knowledge bases (Italy has 60 million inhabitants). In this paper, we pursue the hypothesis that distributional vectors also implicitly encode referential attributes. We show that a standard supervised regression model is in fact sufficient to retrieve such attributes to a reasonable degree of accuracy: When evaluated on the prediction of both categorical and numeric attributes of countries and cities, the model consistently reduces baseline error by 30%, and is not far from the upper bound. Further analysis suggests that our model is able to “objectify” distributional representations for entities, anchoring them more firmly in the external world in measurable ways., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

“Look, some green circles!”: learning to quantify from images

  • Boleda, Gemma
  • Sorodoc, Ionut-Teodor
  • Lazaridou, Angeliki
  • Herbelot, Aurélie
  • Pezzelle, Sandro
  • Bernardi, Raffaella
In this paper, we investigate whether a neural network model can learn the meaning of natural language quantifiers (no,some and all) from their use in visual contexts. We show that memory networks perform well in this task, and that explicit counting is not necessary to the system’s performance, supporting psycholinguistic evidence on the acquisition of quantifiers., This project has received funding from the European Unions Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

“Show me the cup”: reference with continuous representations

  • Boleda, Gemma
  • Baroni, Marco
  • Padó, Sebastian
Comunicació presentada a CICLing 2017, celebrat a Budapest (Hongria) del 17 al 23 d'abril de 2017., One of the most basic functions of language is to refer to objects in a shared scene. Modeling reference with continuous representations is challenging because it requires individuation, i.e., tracking and distinguishing an arbitrary number of referents. We introduce a neural network model that, given a de nite description and a set of objects represented by natural images, points to the intended object if the expression has a unique referent, or indicates a failure, if it does not. The model, directly trained on reference acts, is competitive with a pipeline manually engineered to perform the same task, both when referents are purely visual, and when they are characterized by a combination of visual and linguistic properties., This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe) and ERC grant agreement No 715154 (AMORE); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Hubness and pollution: delving into cross-space mapping for zero-shot learning

  • Lazaridou, Angeliki
  • Dinu, Georgiana
  • Baroni, Marco
Comunicació presentada a: 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing celebrat del 26 al 31 de juliol de 2015 a Pequín, Xina., Zero-shot methods in language, vision and other domains rely on a cross-space mapping function that projects vectors from the relevant feature space (e.g., visualfeature-based image representations) to a large semantic word space (induced in an unsupervised way from corpus data), where the entities of interest (e.g., objects images depict) are labeled with the words associated to the nearest neighbours of the mapped vectors. Zero-shot cross-space mapping methods hold great promise as a way to scale up annotation tasks well beyond the labels in the training data (e.g., recognizing objects that were never seen in training). However, the current performance of cross-space mapping functions is still quite low, so that the strategy is not yet usable in practical applications. In this paper, we explore some general properties, both theoretical and empirical, of the cross-space mapping function, and we build on them to propose better methods to estimate it. In this way, we attain large improvements over the state of the art, both in cross-linguistic (word translation) and cross-modal (image labeling) zero-shot experiments., We acknowledge ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model

  • Pham, Nghia The
  • Kruszewski, German
  • Lazaridou, Angeliki
  • Baroni, Marco
Comunicació presentada a: 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing celebrat del 26 al 31 de juliol de 2015 a Pequín, Xina., We introduce C-PHRASE, a distributional semantic model that learns word representations by optimizing context prediction for phrases at all levels in a syntactic tree, from single words to full sentences. C-PHRASE outperforms the state-of-theart C-BOW model on a variety of lexical tasks. Moreover, since C-PHRASE word vectors are induced through a compositional learning objective (modeling the contexts of words combined into phrases), when they are summed, they produce sentence representations that rival those generated by ad-hoc compositional models., We thank Gemma Boleda and the anonymous reviewers for useful comments. We acknowledge ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Distributional semantics in use

  • Boleda, Gemma
  • Bernardi, Raffaella
  • Fernández, Raquel
  • Paperno, Denis
In this position paper we argue that an adequate semantic model must account for language in use, taking into account how discourse context affects the meaning of words and larger linguistic units. Distributional semantic models are very attractive models of meaning mainly because they capture conceptual aspects and are automatically induced from natural language data. However, they need to be extended in order to account for language use in a discourse or dialogue context. We discuss phenomena that the new generation of distributional semantic models should capture, and propose concrete tasks on which they could be tested., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe), FP7 programme ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES), and the Netherlands Organization for Scientific Research (NWO).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Leveraging preposition ambiguity to assess compositional distributional models of semantics

  • Ritter, Samuel
  • Long, Cotie
  • Paperno, Denis
  • Baroni, Marco
  • Botvinick, Matthew
  • Goldberg, Adele
Comunicació presentada a: Fourth Joint Conference on Lexical and Computational Semantics celebrat del 4 al 5 de juny de 2015 a Denver, EUA., Complex interactions among the meanings of words are important factors in the function that maps word meanings to phrase meanings. Recently, compositional distributional semantics models (CDSM) have been designed with the goal of emulating these complex interactions; however, experimental results on the effectiveness of CDSM have been difficult to interpret because the current metrics for assessing them do not control for the confound of lexical information. We present a new method for assessing the degree to which CDSM capture semantic interactions that dissociates the influences of lexical and compositional information. We then provide a dataset for performing this type of assessment and use it to evaluate six compositional models using both co-occurrence based and neural language model input vectors. Results show that neural language input vectors are consistently superior to co-occurrence based vectors, that several CDSM capture substantial compositional information, and that, surprisingly, vector addition matches and is in many cases superior to purpose-built paramaterized models., Denis Paperno and Marco Baroni were supported by ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES). Samuel Ritter and Matthew Botvinick were supported by Intelligence Advanced Research Projects Activity (IARPA) Grant n. 102-01.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The LAMBADA dataset: word prediction requiring a broad discourse context

  • Boleda, Gemma
  • Paperno, Denis
  • Kruszewski, German
  • Lazaridou, Angeliki
  • Pham, Quan Ngoc
  • Bernardi, Raffaella
  • Pezzelle, Sandro
  • Baroni, Marco
  • Fernandez, Raquel
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-ofthe-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); NWO VIDI grant n. 276-89-008 (Asymmetry in Conversation).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Convolutional neural network language models

  • Boleda, Gemma
  • Pham, Nghia The
  • Kruszewski, German
Convolutional Neural Networks (CNNs) have shown to yield very strong results in several Computer Vision tasks. Their application to language has received much less attention, and it has mainly focused on static classification tasks, such as sentence classification for Sentiment Analysis or relation extraction. In this work, we study the application of CNNs to language modeling, a dynamic, sequential prediction task that needs models to capture local as well as long-range dependency information. Our contribution is twofold. First, we show that CNNs achieve 11-26% better absolute performance than feed-forward neural language models, demonstrating their potential for language representation even in sequential tasks. As for recurrent models, our model outperforms RNNs but is below state of the art LSTM models. Second, we gain some understanding of the behavior of the model, showing that CNNs in language act as feature detectors at a high level of abstraction, like in Computer Vision, and that the model can profitably use information from as far as 16 words before the target., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES) and the Erasmus Mundus Scholarship for Joint Master Programs.