Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 22
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Living a discrete life in a continuous world: reference with distributed representations

  • Boleda, Gemma
  • Padó, Sebastian
  • Pham, Nghia The
  • Baroni, Marco
Comunicació presentada a IWCS 2017, celebrat a Montpellier (França) del 19 al 22 de setembre de 2017., Reference is the crucial property of language that allows us to connect linguistic expressions to the world. Modeling it requires handling both continuous and discrete aspects of meaning. Data-driven models excel at the former, but struggle with the latter, and the reverse is true for symbolic models. We propose a fully data-driven, end-toend trainable model that, while operating on continuous multimodal representations, learns to organize them into a discrete-like entity library. We also introduce a referential task to test it, cross-modal tracking. Our model beats standard neural network architectures, but is outperformed by some parametrizations of Memory Networks, another model with external memory., This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154; AMORE); Marie Sklodowska-Curie grant agreement No 655577 (LOVe); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Formal Distributional Semantics: Introduction to the Special Issue

  • Boleda, Gemma
  • Herbelot, Aurélie
Formal Semantics and Distributional Semantics are two very influential semantic frameworks in Computational Linguistics. Formal Semantics is based on a symbolic tradition and centered around the inferential properties of language. Distributional Semantics is statistical and data-driven, and focuses on aspects of meaning related to descriptive content. The two frameworks are complementary in their strengths, and this has motivated interest in combining them into an overarching semantic framework: a “Formal Distributional Semantics.” Given the fundamentally different natures of the two paradigms, however, building an integrative framework poses significant theoretical and engineering challenges. The present issue of Computational Linguistics advances the state of the art in Formal Distributional Semantics; this introductory article explains the motivation behind it and summarizes the contributions of previous work on the topic, providing the necessary background for the articles that follow., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no 655577 (LOVe) as well as the ERC 2011 Starting Independent Research Grant no. 283554 (COMPOSES).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Distributional vectors encode referential attributes

  • Boleda, Gemma
  • Gupta, Abhijeet
  • Baroni, Marco
  • Padó, Sebastian
Distributional methods have proven to excel at capturing fuzzy, graded aspects of meaning (Italy is more similar to Spain than to Germany). In contrast, it is difficult to extract the values of more specific attributes of word referents from distributional representations, attributes of the kind typically found in structured knowledge bases (Italy has 60 million inhabitants). In this paper, we pursue the hypothesis that distributional vectors also implicitly encode referential attributes. We show that a standard supervised regression model is in fact sufficient to retrieve such attributes to a reasonable degree of accuracy: When evaluated on the prediction of both categorical and numeric attributes of countries and cities, the model consistently reduces baseline error by 30%, and is not far from the upper bound. Further analysis suggests that our model is able to “objectify” distributional representations for entities, anchoring them more firmly in the external world in measurable ways., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Convolutional neural network language models

  • Boleda, Gemma
  • Pham, Nghia The
  • Kruszewski, German
Convolutional Neural Networks (CNNs) have shown to yield very strong results in several Computer Vision tasks. Their application to language has received much less attention, and it has mainly focused on static classification tasks, such as sentence classification for Sentiment Analysis or relation extraction. In this work, we study the application of CNNs to language modeling, a dynamic, sequential prediction task that needs models to capture local as well as long-range dependency information. Our contribution is twofold. First, we show that CNNs achieve 11-26% better absolute performance than feed-forward neural language models, demonstrating their potential for language representation even in sequential tasks. As for recurrent models, our model outperforms RNNs but is below state of the art LSTM models. Second, we gain some understanding of the behavior of the model, showing that CNNs in language act as feature detectors at a high level of abstraction, like in Computer Vision, and that the model can profitably use information from as far as 16 words before the target., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES) and the Erasmus Mundus Scholarship for Joint Master Programs.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

“Look, some green circles!”: learning to quantify from images

  • Boleda, Gemma
  • Sorodoc, Ionut-Teodor
  • Lazaridou, Angeliki
  • Herbelot, Aurélie
  • Pezzelle, Sandro
  • Bernardi, Raffaella
In this paper, we investigate whether a neural network model can learn the meaning of natural language quantifiers (no,some and all) from their use in visual contexts. We show that memory networks perform well in this task, and that explicit counting is not necessary to the system’s performance, supporting psycholinguistic evidence on the acquisition of quantifiers., This project has received funding from the European Unions Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

“Show me the cup”: reference with continuous representations

  • Boleda, Gemma
  • Baroni, Marco
  • Padó, Sebastian
Comunicació presentada a CICLing 2017, celebrat a Budapest (Hongria) del 17 al 23 d'abril de 2017., One of the most basic functions of language is to refer to objects in a shared scene. Modeling reference with continuous representations is challenging because it requires individuation, i.e., tracking and distinguishing an arbitrary number of referents. We introduce a neural network model that, given a de nite description and a set of objects represented by natural images, points to the intended object if the expression has a unique referent, or indicates a failure, if it does not. The model, directly trained on reference acts, is competitive with a pipeline manually engineered to perform the same task, both when referents are purely visual, and when they are characterized by a combination of visual and linguistic properties., This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe) and ERC grant agreement No 715154 (AMORE); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Distributional semantics in use

  • Boleda, Gemma
  • Bernardi, Raffaella
  • Fernández, Raquel
  • Paperno, Denis
In this position paper we argue that an adequate semantic model must account for language in use, taking into account how discourse context affects the meaning of words and larger linguistic units. Distributional semantic models are very attractive models of meaning mainly because they capture conceptual aspects and are automatically induced from natural language data. However, they need to be extended in order to account for language use in a discourse or dialogue context. We discuss phenomena that the new generation of distributional semantic models should capture, and propose concrete tasks on which they could be tested., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe), FP7 programme ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES), and the Netherlands Organization for Scientific Research (NWO).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

The LAMBADA dataset: word prediction requiring a broad discourse context

  • Boleda, Gemma
  • Paperno, Denis
  • Kruszewski, German
  • Lazaridou, Angeliki
  • Pham, Quan Ngoc
  • Bernardi, Raffaella
  • Pezzelle, Sandro
  • Baroni, Marco
  • Fernandez, Raquel
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-ofthe-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); NWO VIDI grant n. 276-89-008 (Asymmetry in Conversation).
Repositorio Digital de la UPF
Publicación Capítulo o parte de un libro (BookPart)

Conceptual versus referential affordance in concept composition

  • McNally, Louise, 1965-
  • Boleda, Gemma
One of the defining traits of language is its capacity to mediate between concepts in our mind, which encapsulate generalizations, and the things they refer to in a given communicative act, with all their idiosyncratic properties. This article examines precisely this interplay between conceptual and referential aspects of meaning, and proposes that concept composition (or concept combination, a term more commonly used in Psychology) exploits both: Conceptually afforded composition is at play when a modifier and its head fit as could be expected given the properties of the two concepts involved, whereas in referentially afforded composition the result of the composition depends on specific, independently available properties of the referent. For instance, red box tends to be applied to boxes whose surface is red, but, given the appropriate context, it can also be applied to e.g. a brown box that contains red objects. We support our proposal with data from nominal modification, and explore a way to formally distinguish the two kinds of composition and integrate them into a more general framework for semantic analysis. Along the way, we recover the classically Fregean notion of sense as including conceptual information, and show the potential of distributional semantics, a framework that has become very influential in Cognitive Science and Computational Linguistics, to address research questions from a theoretical linguistic perspective., This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe) and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154, AMORE); Spanish MINECO grants FFI2010-15006, FFI2010-09464-E, and FFI2013-41301, as well as an ICREA Academia award to the first author.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Instances and concepts in distributional space

  • Boleda, Gemma
  • Gupta, Abhijeet
  • Padó, Sebastian
Comunicació presentada a l'European Chapter of the Association for Computational Linguistics (EACL) celebrat a València, Espanya, del 2 al 6 de juny de 2017., Instances (“Mozart”) are ontologically distinct from concepts or classes (“composer”). Natural language encompasses both, but instances have received comparatively little attention in distributional semantics. Our results show that instances and concepts differ in their distributional properties. We also establish that instantiation detection (“Mozart – composer”) is generally easier than hypernymy detection (“chemist – scientist”), and that results on the influence of input representation do not transfer from hyponymy to instantiation., The authors have received funding from DFG (SFB 732, project B9). and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154; AMORE), as well as under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe).
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Zipf's law for word frequencies: Word forms versus lemmas in long texts

  • Corral, Alvaro
  • Boleda Torrent, Gemma
  • Ferrer Cancho, Ramon|||0000-0002-7820-923X
Zipf's law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf's law should hold in the most natural way, studying its validity for plain word forms and for the corresponding lemma forms. We analyze several long literary texts comprising four languages, with different levels of morphological complexity. In all cases Zipf's law is fulfilled, in the sense that a power-law distribution of word or lemma frequencies is valid for several orders of magnitude. We investigate the extent to which the word-lemma transformation preserves two parameters of Zipf's law: the exponent and the low-frequency cut-off. We are not able to demonstrate a strict invariance of the tail, as for a few texts both exponents deviate significantly, but we conclude that the exponents are very similar, despite the remarkavble transformation that going from words to lemmas represents, considerably affecting all ranges of frequencies. In contrast, the low-frequency cut-offs are less stable, tending to increase substantially after the transformation., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Living a discrete life in a continuous world: reference with distributed representations

  • Boleda, Gemma
  • Padó, Sebastian
  • Pham, Nghia The
  • Baroni, Marco
Comunicació presentada a IWCS 2017, celebrat a Montpellier (França) del 19 al 22 de setembre de 2017., Reference is the crucial property of language that allows us to connect linguistic expressions to the world. Modeling it requires handling both continuous and discrete aspects of meaning. Data-driven models excel at the former, but struggle with the latter, and the reverse is true for symbolic models. We propose a fully data-driven, end-toend trainable model that, while operating on continuous multimodal representations, learns to organize them into a discrete-like entity library. We also introduce a referential task to test it, cross-modal tracking. Our model beats standard neural network architectures, but is outperformed by some parametrizations of Memory Networks, another model with external memory., This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154; AMORE); Marie Sklodowska-Curie grant agreement No 655577 (LOVe); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

Conceptual versus referential affordance in concept composition

  • McNally, Louise, 1965-
  • Boleda, Gemma
One of the defining traits of language is its capacity to mediate between concepts in our mind, which encapsulate generalizations, and the things they refer to in a given communicative act, with all their idiosyncratic properties. This article examines precisely this interplay between conceptual and referential aspects of meaning, and proposes that concept composition (or concept combination, a term more commonly used in Psychology) exploits both: Conceptually afforded composition is at play when a modifier and its head fit as could be expected given the properties of the two concepts involved, whereas in referentially afforded composition the result of the composition depends on specific, independently available properties of the referent. For instance, red box tends to be applied to boxes whose surface is red, but, given the appropriate context, it can also be applied to e.g. a brown box that contains red objects. We support our proposal with data from nominal modification, and explore a way to formally distinguish the two kinds of composition and integrate them into a more general framework for semantic analysis. Along the way, we recover the classically Fregean notion of sense as including conceptual information, and show the potential of distributional semantics, a framework that has become very influential in Cognitive Science and Computational Linguistics, to address research questions from a theoretical linguistic perspective., This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe) and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154, AMORE); Spanish MINECO grants FFI2010-15006, FFI2010-09464-E, and FFI2013-41301, as well as an ICREA Academia award to the first author.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Formal Distributional Semantics: Introduction to the Special Issue

  • Boleda, Gemma
  • Herbelot, Aurélie
Formal Semantics and Distributional Semantics are two very influential semantic frameworks in Computational Linguistics. Formal Semantics is based on a symbolic tradition and centered around the inferential properties of language. Distributional Semantics is statistical and data-driven, and focuses on aspects of meaning related to descriptive content. The two frameworks are complementary in their strengths, and this has motivated interest in combining them into an overarching semantic framework: a “Formal Distributional Semantics.” Given the fundamentally different natures of the two paradigms, however, building an integrative framework poses significant theoretical and engineering challenges. The present issue of Computational Linguistics advances the state of the art in Formal Distributional Semantics; this introductory article explains the motivation behind it and summarizes the contributions of previous work on the topic, providing the necessary background for the articles that follow., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no 655577 (LOVe) as well as the ERC 2011 Starting Independent Research Grant no. 283554 (COMPOSES).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Instances and concepts in distributional space

  • Boleda, Gemma
  • Gupta, Abhijeet
  • Padó, Sebastian
Comunicació presentada a l'European Chapter of the Association for Computational Linguistics (EACL) celebrat a València, Espanya, del 2 al 6 de juny de 2017., Instances (“Mozart”) are ontologically distinct from concepts or classes (“composer”). Natural language encompasses both, but instances have received comparatively little attention in distributional semantics. Our results show that instances and concepts differ in their distributional properties. We also establish that instantiation detection (“Mozart – composer”) is generally easier than hypernymy detection (“chemist – scientist”), and that results on the influence of input representation do not transfer from hyponymy to instantiation., The authors have received funding from DFG (SFB 732, project B9). and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154; AMORE), as well as under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Distributional vectors encode referential attributes

  • Boleda, Gemma
  • Gupta, Abhijeet
  • Baroni, Marco
  • Padó, Sebastian
Distributional methods have proven to excel at capturing fuzzy, graded aspects of meaning (Italy is more similar to Spain than to Germany). In contrast, it is difficult to extract the values of more specific attributes of word referents from distributional representations, attributes of the kind typically found in structured knowledge bases (Italy has 60 million inhabitants). In this paper, we pursue the hypothesis that distributional vectors also implicitly encode referential attributes. We show that a standard supervised regression model is in fact sufficient to retrieve such attributes to a reasonable degree of accuracy: When evaluated on the prediction of both categorical and numeric attributes of countries and cities, the model consistently reduces baseline error by 30%, and is not far from the upper bound. Further analysis suggests that our model is able to “objectify” distributional representations for entities, anchoring them more firmly in the external world in measurable ways., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

“Look, some green circles!”: learning to quantify from images

  • Boleda, Gemma
  • Sorodoc, Ionut-Teodor
  • Lazaridou, Angeliki
  • Herbelot, Aurélie
  • Pezzelle, Sandro
  • Bernardi, Raffaella
In this paper, we investigate whether a neural network model can learn the meaning of natural language quantifiers (no,some and all) from their use in visual contexts. We show that memory networks perform well in this task, and that explicit counting is not necessary to the system’s performance, supporting psycholinguistic evidence on the acquisition of quantifiers., This project has received funding from the European Unions Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

“Show me the cup”: reference with continuous representations

  • Boleda, Gemma
  • Baroni, Marco
  • Padó, Sebastian
Comunicació presentada a CICLing 2017, celebrat a Budapest (Hongria) del 17 al 23 d'abril de 2017., One of the most basic functions of language is to refer to objects in a shared scene. Modeling reference with continuous representations is challenging because it requires individuation, i.e., tracking and distinguishing an arbitrary number of referents. We introduce a neural network model that, given a de nite description and a set of objects represented by natural images, points to the intended object if the expression has a unique referent, or indicates a failure, if it does not. The model, directly trained on reference acts, is competitive with a pipeline manually engineered to perform the same task, both when referents are purely visual, and when they are characterized by a combination of visual and linguistic properties., This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe) and ERC grant agreement No 715154 (AMORE); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); DFG (SFB 732, Project D10); and Spanish MINECO (grant FFI2013-41301-P).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Distributional semantics in use

  • Boleda, Gemma
  • Bernardi, Raffaella
  • Fernández, Raquel
  • Paperno, Denis
In this position paper we argue that an adequate semantic model must account for language in use, taking into account how discourse context affects the meaning of words and larger linguistic units. Distributional semantic models are very attractive models of meaning mainly because they capture conceptual aspects and are automatically induced from natural language data. However, they need to be extended in order to account for language use in a discourse or dialogue context. We discuss phenomena that the new generation of distributional semantic models should capture, and propose concrete tasks on which they could be tested., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe), FP7 programme ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES), and the Netherlands Organization for Scientific Research (NWO).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Zipf's law for word frequencies: Word forms versus lemmas in long texts

  • Corral, Alvaro
  • Boleda Torrent, Gemma
  • Ferrer Cancho, Ramon
Zipf's law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf's law should hold in the most natural way, studying its validity for plain word forms and for the corresponding lemma forms. We analyze several long literary texts comprising four languages, with different levels of morphological complexity. In all cases Zipf's law is fulfilled, in the sense that a power-law distribution of word or lemma frequencies is valid for several orders of magnitude. We investigate the extent to which the word-lemma transformation preserves two parameters of Zipf's law: the exponent and the low-frequency cut-off. We are not able to demonstrate a strict invariance of the tail, as for a few texts both exponents deviate significantly, but we conclude that the exponents are very similar, despite the remarkavble transformation that going from words to lemmas represents, considerably affecting all ranges of frequencies. In contrast, the low-frequency cut-offs are less stable, tending to increase substantially after the transformation., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

The LAMBADA dataset: word prediction requiring a broad discourse context

  • Boleda, Gemma
  • Paperno, Denis
  • Kruszewski, German
  • Lazaridou, Angeliki
  • Pham, Quan Ngoc
  • Bernardi, Raffaella
  • Pezzelle, Sandro
  • Baroni, Marco
  • Fernandez, Raquel
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-ofthe-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES); NWO VIDI grant n. 276-89-008 (Asymmetry in Conversation).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Convolutional neural network language models

  • Boleda, Gemma
  • Pham, Nghia The
  • Kruszewski, German
Convolutional Neural Networks (CNNs) have shown to yield very strong results in several Computer Vision tasks. Their application to language has received much less attention, and it has mainly focused on static classification tasks, such as sentence classification for Sentiment Analysis or relation extraction. In this work, we study the application of CNNs to language modeling, a dynamic, sequential prediction task that needs models to capture local as well as long-range dependency information. Our contribution is twofold. First, we show that CNNs achieve 11-26% better absolute performance than feed-forward neural language models, demonstrating their potential for language representation even in sequential tasks. As for recurrent models, our model outperforms RNNs but is below state of the art LSTM models. Second, we gain some understanding of the behavior of the model, showing that CNNs in language act as feature detectors at a high level of abstraction, like in Computer Vision, and that the model can profitably use information from as far as 16 words before the target., This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES) and the Erasmus Mundus Scholarship for Joint Master Programs.