Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 170
Encontrada(s) 2 página(s)

Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

  • Gialampoukidis, Ilias
  • Moumtzidou, Anastasia
  • Liparas, Dimitris
  • Tsikrika, Theodora
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Heterogeneous sources of information, such as images, videos, text and metadata are often used to describe di erent or complementary views of the same multimedia object, especially in the online news domain and in large annotated image collections. The retrieval of multimedia objects, given a mul- timodal query, requires the combination of several sources of information in an e cient and scalable way. Towards this direction, we provide a novel unsuper- vised framework for multimodal fusion of visual and textual similarities, which are based on visual features, visual concepts and textual metadata, integrating non-linear graph-based fusion and Partial Least Squares Regression. The fu- sion strategy is based on the construction of a multimodal contextual similarity matrix and the non-linear combination of relevance scores from query-based similarity vectors. Our framework can employ more than two modalities and high-level information, without increase in memory complexity, when com- pared to state-of-the-art baseline methods. The experimental comparison is done in three public multimedia collections in the multimedia retrieval task. The results have shown that the proposed method outperforms the baseline methods, in terms of Mean Average Precision and Precision@20., This work was partially supported by the European Commission by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Neural architectures for named entity recognition

  • Lample, Guillaume
  • Ballesteros, Miguel
  • Subramanian, Sandeep
  • Kawakami, Kazuya
  • Dyer, Chris
Comunicació presentada a la 2016 Conference of the North American Chapter of the Association for Computational Linguistics, celebrada a San Diego (CA, EUA) els dies 12 a 17 de juny 2016., State-of-the-art named entity recognition systems/nrely heavily on hand-crafted features and/ndomain-specific knowledge in order to learn/neffectively from the small, supervised training/ncorpora that are available. In this paper, we/nintroduce two new neural architectures—one/nbased on bidirectional LSTMs and conditional/nrandom fields, and the other that constructs/nand labels segments using a transition-based/napproach inspired by shift-reduce parsers./nOur models rely on two sources of information/nabout words: character-based word/nrepresentations learned from the supervised/ncorpus and unsupervised word representations/nlearned from unannotated corpora. Our/nmodels obtain state-of-the-art performance in/nNER in four languages without resorting to/nany language-specific knowledge or resources/nsuch as gazetteers., This work was sponsored in part by the Defense/nAdvanced Research Projects Agency (DARPA)/nInformation Innovation Office (I2O) under the/nLow Resource Languages for Emergent Incidents/n(LORELEI) program issued by DARPA/I2O under/nContract No. HR0011-15-C-0114. Miguel Ballesteros/nis supported by the European Commission under/nthe contract numbers FP7-ICT-610411 (project/nMULTISENSOR) and H2020-RIA-645012 (project/nKRISTINA).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Design of knowledge-based agent as a social companion

  • Wanner, Leo
  • André, Elisabeth
  • Blat, Josep
  • Dasiopoulou, Stamatia
  • Farrús, Mireia
  • Fraga, Thiago
  • Kamateri, Eleni
  • Lingenfelser, Florian
  • Llorach, Gerard
  • Martínez, Oriol
  • Meditskos, Georgios
  • Mille, Simon
  • Minker, Wolfgang
  • Pragst, Louisa
  • Schiller, Dominik
  • Stam, Andries
  • Stellingwerff, Ludo
  • Sukno, Federico Mateo
  • Vieru, Bianca
  • Vrochidis, Stefanos
Article de la comunicació presentada a International Conference on Health and Social Care Information Systems and Technologies (HCist 2017), celebrat a Barcelona el novembre de 2017., We present work in progress on an intelligent embodied conversation agent that is supposed to act as a social companion with linguistic and emotional competence in the context of basic and health care. The core of the agent is an ontology-based knowledge model that supports flexible reasoning-driven conversation planning strategies. A dedicated search engine ensures the provision of background information from the web, necessary for conducting a conversation on a specific topic. Multimodal communication analysis and generation modules analyze respectively generate facial expressions, gestures and multilingual speech. The assessment of the prototypical implementation of the agent shows that users accept it as a natural and trustworthy conversation counterpart. For the final release, all involved technologies will be further improved and matured., The presented work is funded by the European Commission as part of the H2020 Program under the contract number 645012. We would like to thank our colleagues Chiara Baudracco, Jutta Mohr, Eylem Ög, Valérie Sarholz, and Benjamin Schäfer for organizing and carrying out the evaluation trials and for their patience with the numerous inadequacies of KRISTINA.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Unsupervised keyword extraction using the GoW model and centrality scores

  • Batziou, Elissavet
  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: The 4th International Conference, INSCI 2017, celebrat a Thessaloniki, Grècia, del 22 al 24 de novembre de 2017., Nowadays, a large amount of text documents are produced on a daily basis, so we need e cient and e ective access to their con- tent. News articles, blogs and technical reports are often lengthy, so the reader needs a quick overview of the underlying content. To that end we present graph-based models for keyword extraction, in order to compare the Bag of Words model with the Graph of Words model in the key- word extraction problem. We compare their performance in two publicly available datasets using the evaluation measures Precision@10, mean Av- erage Precision and Jaccard coe cient. The methods we have selected for comparison are grouped into two main categories. On the one hand, centrality measures on the formulated Graph-of-Words (GoW) are able to rank all words in a document from the most central to the less central, according to their score in the GoW representation. On the other hand, community detection algorithms on the GoW provide the largest commu- nity that contains the key nodes (words) in the GoW. We selected these methods as the most prominent methods to identify central nodes in a GoW model. We conclude that term-frequency scores (BoW model) are useful only in the case of less structured text, while in more structured text documents, the order of words plays a key role and graph-based models are superior to the term-frequency scores per document., This work was supported by the projects H2020-645012 (KRISTINA) and H2020-700024 (TENSOR), funded by the European Commission.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

UPF at EPE 2017: transduction-based deep analysis

  • Mille, Simon
  • Latorre, Iván
  • Wanner, Leo
Comunicació presentada a: Shared Task on Extrinsic Parser Evaluation (EPE 2017), celebrada a Pisa, Itàlia, el 20 de setembre de 2017., This paper describes the runs submitted to EPE 2017 by Universitat Pompeu Fabra. The three outputs correspond to three different levels of linguistic abstraction: (i) a surface-syntactic tree, (ii) a deep-syntactic tree, and (iii) a predicateargument graph. The surface-syntactic tree is obtained with an off-the-shelf parser trained on the CoNLL’09 Penn Treebank, and the deeper representations by running a sequence of graph transduction grammars on the output of the parser.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Exploiting visual similarities for ontology alignment

  • Doulaverakis, Charalampos
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Ontology alignment is the process where two different ontologies that usually describe similar domains are ’aligned’, i.e. a set of correspondences between their entities, regarding semantic equivalence, is determined. In order to identify these correspondences several methods and metrics that measure semantic equivalence have been proposed in literature. The most common features that these metrics employ are string-, lexical-, structure- and semantic-based similarities for which several approaches have been developed. However, what hasn’t been investigated is the usage of visual-based features for determining entity similarity in cases where images are associated with concepts. Nowadays the existence of several resources (e.g. ImageNet) that map lexical concepts onto images allows for exploiting visual similarities for this purpose. In this paper, a novel approach for ontology matching based on visual similarity is presented. Each ontological entity is associated with sets of images, retrieved through ImageNet or web-based search, and state of the art visual feature extraction, clustering and indexing for computing the similarity between entities is employed. An adaptation of a popular Wordnet-based matching algorithm to exploit the visual similarity is also proposed. Our method is compared with traditional metrics against a standard ontology alignment benchmark dataset and demonstrates promising results., This work was supported by MULTISENSOR (contract no. FP7-610411) and KRISTINA (contract no./nH2020-645012) projects, partially funded by the European Commission.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

VERGE in VBS 2017

  • Moumtzidou, Anastasia
  • Mironidis, Theodoros
  • Markatopoulou, Fotini
  • Andreadis, Stelios
  • Gialampoukidis, Ilias
  • Galanopoulos, Damianos
  • Ioannidou, Anastasia
  • Vrochidis, Stefanos
  • Mezaris, Vasileios
  • Kompatsiaris, Ioannis
  • Patras, Ioannis
Comunicació presentada a Video Browser Showdown (VBS'17), a 23rd International Conference on MultiMedia Modeling (MMM'17), celebrat el 4 de gener de 2017 a Reykjavik, Islàndia., This paper presents VERGE interactive video retrieval engine, which is capable of browsing and searching into video content. The system integrates several content-based analysis and retrieval modules including concept detec-tion, clustering, visual similarity search, object-based search, query analysis and multimodal and temporal fusion., This work was supported by the EU’s Horizon 2020 research and innovation programme under grant agreements H2020-687786 InVID, H2020-693092 MOVING, H2020-645012 KRISTINA and H2020-700024 TENSOR.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Automatic extraction of parallel speech corpora from dubbed movies

  • Öktem, Alp
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: 10th Workshop on Building and Using Comparable Corpora (BUCC), celebrat a Vancouver, Canadà, del 30 de juliol al 4 d'agost de 2017., This paper presents a methodology to extract parallel speech corpora based on any language pair from dubbed movies, together with an application framework in which some corresponding prosodic parameters are extracted. The obtained parallel corpora are especially suitable for speech-to-speech translation applications when a prosody transfer between source and target languages is desired., This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number 645012. The second author is partially funded by the Spanish Ministry of Economy, Industry and Competitiveness through the Ram´on y Cajal program.
Repositorio Digital de la UPF
Publication Capítulo o parte de un libro (BookPart)

Web-based embodied conversational agents and older people

  • Llorach, Gerard
  • Agenjo, Javier
  • Blat, Josep
  • Sayago Barrantes, Sergio
Within Human-Computer Interaction, there has recently been an important turn to embodied and voice-based interaction. In this chapter, we discuss our ongoing research on building online Embodied Conversational Agents (ECAs), specifically, their interactive 3D web graphics aspects. We present ECAs based on our technological pipeline, which integrates a number of free online editors, such as Adobe Fuse CC or MakeHuman, and standards, mainly BML (Behaviour Markup Language). We claim that making embodiment available for online ECAs is attainable, and advantageous over current alternatives, mostly desktop-based. In this chapter we also report on initial results of activities aimed to explore the physical appearance of ECAs for older people. A group of them (N = 14) designed female ECAs. Designing them was easy and great fun. The perspective on older-adult HCI introduced in this chapter is mostly technological, allowing for rapid online experimentations to address key issues, such as anthropomorphic aspects, in the design of ECAs with, and for, older people., This work was funded by the EU’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 675324 (ENRICH) and under the contract number H2020-645012-RIA (KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Retrieval of multimedia objects by fusing multiple modalities

  • Gialampoukidis, Ilias
  • Moumtzidou, Anastasia
  • Tsikrika, Theodora
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: ICMR'16. International Conference on Multimedia Retrieval 2016, celebrat a Nova York del 6 al 9 de juny de 2016, Searching for multimedia objects with heterogeneous modalities is critical for the construction of effective multimedia retrieval systems. Towards this direction, we propose a framework for the multimodal fusion of visual and textual similarities, based on visual features, visual concepts and textual concepts. Our method is compared to the baseline method that only fuses two modalities but integrates all early, late, linearly weighted, diffusion and graph-based models in one unifying framework. Our framework integrates more than two modalities and high-level information, so as to retrieve multimedia objects enriched with high-level textual and visual concepts, in response to a multimodal query. The experimental comparison is done under the same memory complexity, in two multimedia collections in the multimedia retrieval task. The results have shown that we outperform the baseline method, in terms of Mean Average Precision., This work was partially supported by the European Commission by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Automatic modification of communication style in dialogue management

  • Pragst, Louisa
  • Miehle, Juliana
  • Ultes, Stefan
  • Minker, Wolfgang
Comunicació presentada a: INLG 2016 Workshop on Computational Creativity and Natural Language Generation, celebrat a Edinburgh, Escòcia, del 5 al 8 de setembre de 2016., In task-oriented dialogues, there is often only one right answer the system can give. However, a lack of variation can seem repetitive and unnatural. Humans change the way they express something, e.g. by being more or less concise. We aim to approximate this ability by automatically varying the level of verbosity and directness of a given system action. In this work, we illustrate how verbosity and directness may be utilised in adaptive dialogue management and present different approaches to automatically generate varying levels of verbosity and directness for given system actions. Thereby, new and unforeseen system actions can be created dynamically., This paper is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Towards a multimedia knowledge-based agent with social competence and human interaction capabilities

  • Wanner, Leo
  • Blat, Josep
  • Dasiopoulou, Stamatia
  • Domínguez Bajo, Mónica
  • Llorach, Gerard
  • Mille, Simon
  • Sukno, Federico Mateo
  • Kamateri, Eleni
  • Kompatsiaris, Ioannis
  • Vrochidis, Stefanos
  • André, Elisabeth
  • Lingenfelser, Florian
  • Mehlmann, Gregor
  • Stam, Andries
  • Stellingwerff, Ludo
  • Lamel, Lori
  • Vieru, Bianca
  • Minker, Wolfgang
  • Pragst, Louisa
  • Ultes, Stefan
We present work in progress on an intelligent embodied conversation agent in the basic care and healthcare domain. In contrast to most of the existing agents, the presented agent is aimed to have linguistic cultural, social and emotional competence needed to interact with elderly and migrants. It is composed of an ontology-based and reasoning-driven dialogue manager, multimodal communication analysis and generation modules and a search engine for the retrieval of multimedia background content from the web needed for conducting a conversation on a given topic., The presented work is funded by the European Commission under the contract number H2020-645012-RIA.
Repositorio Digital de la UPF
Publication Artículo preliminar (WorkingPaper)

FORGe at E2E 2017

  • Mille, Simon
  • Dasiopoulou, Stamatia
This paper describes the FORGe generator at WebNLG. The input DBpedia triples are mapped onto sentences by applying a series of rule-based graph-transducers and aggregation grammars to template predicate-argument structures associated to each property.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Using hierarchical information structure for prosody prediction in content-to-speech applications

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Burga Díaz, Alicia
  • Wanner, Leo
Paper presented at Speech Prosody 8, 2016 May 31 - Jun 3; Boston, United States., State-of-the-art prosody modelling in content-to-speech (CTS) applications still uses the same methodology to predict intonation cues as text-to-speech (TTS) applications, namely the analysis of the generated surface sentences with respect to part of speech, syntactic dependency relations and word order. On the other side, several theoretical studies argue that morphology, syntax, and information (or communicative) structure that organizes/na given content (semantic or deep-syntactic structure) with respect to the intention of the speaker show a strong correlation with intonation. However, little empirical work based on sufficiently large corpora has been carried out so far to buttress this argumentation. We present empirical evidence for the Information Structure–Prosody correlation using the Wall Street Journal Penn Treebank corpus recorded by native American English speakers. Our experiments reach a prosody prediction accuracy of 80% using the hierarchical information structure from the Meaning-Text Theory, compared to 59% of the baseline., This work is part of a project that has received funding from the European Union’s Horizon 2020 Research and Innovation/nProgramme under the Grant Agreement number H2020-RIA-645012. The second author is partially funded by a grant from/nthe Spanish Ministry of Economy and Competitivity in the framework of the Juan de la Cierva fellowship program.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Praat on the Web: an upgrade of Praat for semi-automatic speech annotation

  • Domínguez Bajo, Mónica
  • Latorre, Iván
  • Farrús, Mireia
  • Codina Filbà, Joan
  • Wanner, Leo
Paper presented at COLING 2016, the 26th International Conference on Computational Linguistics, 2016 Dec 11-17; Osaka, Japan., This paper presents an implementation of the widely used speech analysis tool Praat as a web application with an extended functionality for feature annotation. In particular, Praat on the Web addresses some of the central limitations of the original Praat tool and provides (i) enhanced visualization of annotations in a dedicated window for feature annotation at interval and point segments, (ii) a dynamic scripting composition exemplified with a modular prosody tagger, and (iii) portability and an operational web interface. Speech annotation tools with such a functionality are key for exploring large corpora and designing modular pipelines., This work is part of the KRISTINA project, which has received funding from the European Unions Horizon 2020 Research and Innovation Programme under the Grant Agreement number H2020-RIA-645012. It has been also partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502). The third author is partially funded by the Spanish Ministry of Economy and Competitivity through the Juan de la Cierva program.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Hybrid graph-based and non-linear late fusion approach for multimedia retrieval

  • Gialampoukidis, Ilias
  • Moumtzidou, Anastasia
  • Liparas, Dimitris
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: 14th International Workshop on Content-Based Multimedia Indexing (CBMI 2016) celebrat del 15 al 17 de juny de 2016 a Bucarest, Romania., Nowadays, multimedia retrieval has become a task of high importance, due to the need for efficient and fast access to very large and heterogeneous multimedia collections. An interesting challenge within the aforementioned task is the efficient combination of different modalities in a multimedia object and especially the fusion between textual and visual information. The fusion of multiple modalities for retrieval in an unsupervised way has been mostly based on early, weighted linear, graph-based and diffusion-based techniques. In contrast, we present a strategy for fusing textual and visual modalities, through the combination of a non-linear fusion model and a graph-based late fusion approach. The fusion strategy is based on the construction of a uniform multimodal contextual similarity matrix and the non-linear combination of relevance scores from query-based similarity vectors. The proposed late fusion approach is evaluated in the multimedia retrieval task, by applying it to two multimedia collections, namely the WIKI11 and IAPR-TC12. The experimental results indicate its superiority over the baseline method in terms of Mean Average Precision for both considered datasets., This work was supported by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012), funded by the European Commission.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Web-based live speech-driven lip-sync

  • Llorach, Gerard
  • Evans, Alun Thomas
  • Blat, Josep
  • Grimm, Giso
  • Hohmann, Volker
Comunicació presentada a: 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-Games), celebrada a Barcelona del 7 al 9 de setembre de 2016., Virtual characters are an integral part of many games and virtual worlds. The ability to accurately synchronize lip movement to audio speech is an important aspect in the believability of the character. In this paper we propose a simple rule-based lip-syncing algorithm for virtual agents using the web browser. It works in real-time with live input, unlike most current lip-syncing proposals, which may require considerable amounts of computation, expertise and time to set up. Our method generates reliable speech animation based on live speech using three blend shapes and no training, and it only needs manual adjustment of three parameters for each speaker (sensitivity, smoothness and vocal tract length). Our proposal is based on the limited real-time audio processing functions of the client web browser (thus, the algorithm needs to be simple), but this facilitates the use of web based embodied conversational agents., This research has been partially funded by the Spanish Ministry of Economy and Competitiveness (RESET TIN2014-53199-C3-3-R), by the DFG research grant FOR173 and by the European Commission under the contract number H2020-645012-RIA (KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Prosograph: a tool for prosody visualisation of large speech corpora

  • Öktem, Alp
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: The 18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), celebrada a Estocolm, Suència, del 20 al 24 d'agost de 2017., This paper presents an open-source tool that has been developed to visualize a speech corpus with its transcript and prosodic features aligned at word level. In particular, the tool is aimed at providing a simple and clear way to visualize prosodic patterns along large segments of speech corpora, and can be applied in any research that involves prosody analysis., This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number 645012. The second author is partially funded by the Spanish Ministry of Economy, Industry and Competitiveness through the Ram´on y Cajal program.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

The Information structure–prosody interface in text-to-speech technologies: an empirical perspective

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogues. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel’ˇcuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-–prosody interface, This work has been partially funded by the European Commission in the context of its H2020 Programme under the contract numbers H2020-645012-RIA (KRISTINA) and H2020-870930-IA (WELCOME). The second author has been funded by the Agencia Estatal de Investigaci´on (AEI), Ministerio de Ciencia, Innovaci´on y Universidades and the Fondo Social Europeo (FSE), grant RYC2015-17239 (AEI/FSE, UE).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A neural network architecture for multilingual punctuation generation

  • Ballesteros, Miguel
  • Wanner, Leo
Even syntactically correct sentences are perceived as awkward if they do not contain correct punctuation. Still, the problem of automatic generation of punctuation marks has been largely neglected for a long time. We/npresent a novel model that introduces punctuation marks into raw text material with transition-based algorithm using LSTMs. Unlike the state-of-the-art approaches, our model is language-independent and also neutral with respect to the intended use of the punctuation. Multilingual experiments show that it achieves high accuracy on the full range of punctuation marks across languages., This work was supported by the European Commission under the contract numbers FP7-ICT-/n610411 (MULTISENSOR) and H2020-RIA-645012 (KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Visualizing deep-syntactic parser output

  • Soler Company, Juan
  • Ballesteros, Miguel
  • Bohnet, Bernd
  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a la 2015 Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT 2015), celebrada del 31 de maig al 5 de juny 2015 a Denver (CO, EUA)., “Deep-syntactic” dependency structures bridge the gap between the surface-syntactic structures as produced by state-of-the-art dependency parsers and semantic logical forms in that they abstract away from surfacesyntactic idiosyncrasies, but still keep the linguistic structure of a sentence. They have thus a great potential for such downstream applications as machine translation and summarization. In this demo paper, we propose an online version of a deep-syntactic parser that outputs deep-syntactic structures from plain sentences and visualizes them using the Brat tool. Along with the deep syntactic structures, the user can also inspect the visual presentation of the surface-syntactic structures that serve as input to the deep-syntactic parser and that are produced by the joint tagger and syntactic transition-based parser ran in the pipeline before deep-syntactic parsing takes place., This work has been partially funded by the European Union’s Seventh Framework and Horizon 2020 Research and Innovation Programmes under the Grant Agreement numbers FP7-ICT-610411, FP7-SME- 606163, and H2020-RIA-645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Combining acoustic and linguistic features in phrase-oriented prosody prediction

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
Paper presented at Speech Prosody 8, 2016 May 31 - Jun 3; Boston, United States., Intonation is traditionally considered to be the most important prosodic feature, whereupon an important research effort has been devoted to automatic segmentation and labeling of speech samples to grasp intonation cues. A number of studies also show that when duration or intensity are incorporated, automatic prosody labeling is further improved. However, the combination of word level acoustic features still attains poor results when machine learning techniques are applied on annotated corpora to derive intonation for speech synthesis applications. To address this problem, we present an experimental set-up for the development of a hierarchical prosodic structure model which combines linguistic features, including information structure, and three acoustic elements (intensity, pitch and duration). We show empirically that this combination leads to a considerably more accurate representation of prosody and, consequently, a more reliable automatic labeling of speech corpora for machine learning., This work is part of a project that has received funding from the European Union’s Horizon 2020 Research and Innovation/nProgramme under the Grant Agreement number H2020-RIA-645012. The second author is partially funded by a grant from/nthe Spanish Ministry of Economy and Competitivity in the framework of the Juan de la Cierva fellowship program.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Towards an ontology-driven adaptive dialogue framework

  • Meditskos, Georgios
  • Dasiopoulou, Stamatia
  • Pragst, Louisa
  • Ultes, Stefan
  • Vrochidis, Stefanos
  • Wanner, Leo
Comunicació presentada a MARMI'16. 1st International Workshop on Multimedia Analysis and Retrieval for Multimodal Interaction, celebrat del 6 al 9 de juny de 2016 June 6-9 a Nova York, Estats Units d'Amèrica., In this paper, we describe the principles and technologies that underpin the development of an adaptive dialogue manager framework, tailored to carrying out human-agent conversations in a natural, robust and exible manner. Our research focus is twofold. First, the investigation of dialogue strategies that can handle dynamically created user and system actions, while still enabling the agent to adapt its actions to various and possibly changing contexts. Second, the utilisation of rich semantic annotations for capturing background knowledge, as well as conversation topics and semantics of user utterances extracted through language analysis. The resulting annotations comprise the situational descriptions upon which reasoning takes place to recognise the conversation context and compile appropriate responses., This work has been supported by the H2020-ICT-645012 project KRISTINA: A Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Transition-based dependency parsing with stack long short-term memory

  • Dyer, Chris
  • Ballesteros, Miguel
  • Ling, W
  • Matthews, A
  • Smith, Noah A.
We propose a technique for learning representations of parser states in transitionbased dependency parsers. Our primary innovation is a new control structure for sequence-to-sequence neural networks— the stack LSTM. Like the conventional stack data structures used in transitionbased parsing, elements can be pushed to/nor popped from the top of the stack in constant time, but, in addition, an LSTM maintains a continuous space embedding of the stack contents. This lets us formulate an efficient parsing model that captures three facets of a parser’s state: (i) unbounded look-ahead into the buffer of incoming words, (ii) the complete history of actions taken by the parser, and (iii) the complete contents of the stack of partially built tree fragments, including their internal/nstructures. Standard backpropagation techniques are used for training and yield state-of-the-art parsing performance., This work was sponsored in part by the U. S. Army Research Laboratory and the U. S. Army Research Office/nunder contract/grant number W911NF-10-1-0533, and in part by NSF CAREER grant IIS-1054319./nMiguel Ballesteros is supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

FORGe at SemEval-2017 Task 9: deep sentence generation based on a sequence of graph transducers

  • Mille, Simon
  • Carlini, Roberto
  • Burga Díaz, Alicia
  • Wanner, Leo
Comunicació presentada a: The 11th International Workshop on Semantic Evaluations (SemEval-2017), celebrada del 3 al 4 d'agost de 2017 a Vancouver, Canadà., We present the contribution of Universitat Pompeu Fabra’s NLP group to the Sem-Eval Task 9.2 (AMR-to-English Generation). The proposed generation pipeline comprises: (i) a series of rule-based graphtransducers for the syntacticization of the input graphs and the resolution of morphological agreements, and (ii) an off-theshelf statistical linearization component., The work described in this paper has been partially funded by the European Commission under the contract numbers FP7-ICT-610411, H2020- 645012-RIA, H2020-700024-RIA, and H2020-DRS-01-2015.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Fusion of valence and arousal annotations through dynamic subjective ordinal modelling

  • Ruiz, Adrià
  • Martinez, Oriol
  • Binefa i Valls, Xavier
  • Sukno, Federico Mateo
Comunicació presentada a: FG 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition, celebrada del 30 de maig al 3 de juny de 2017 a Washington, Estats Units d'Amèrica., An essential issue when training and validating computer vision systems for affect analysis is how to obtain reliable ground-truth labels from a pool of subjective annotations. In this paper, we address this problem when labels are given in an ordinal scale and annotated items are structured as temporal sequences. This problem is of special importance in affective computing, where collected data is typically formed by videos of human interactions annotated according to the Valence and Arousal (V-A) dimensions. Moreover, recent works have shown that inter-observer agreement of V-A annotations can be considerably improved if these are given in a discrete ordinal scale. In this context, we propose a novel framework which explicitly introduces ordinal constraints to model the subjective perception of annotators. We also incorporate dynamic information to take into account temporal correlations between ground-truth labels. In our experiments over synthetic and real data with V-A annotations, we show that the proposed method outperforms alternative approaches which do not take into account either the ordinal structure of labels or their temporal correlation., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Ramon y Cajal fellowships and the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), and the Kristina project funded by the European Union Horizon 2020 research and innovation programme under grant agreement No 645012. Adria Ruiz would also like to acknowledge Spanish Government to provide support under grant FPU13/01740.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

KRISTINA, un asistente virtual para superar las barreras lingüísticas en la atención sanitaria

  • Dafa, Lehdia Mohamed
  • Baudracco, Chiara
  • Moragas, Ana
  • Llor Vila, Carles
  • Molero Trujillo, David
  • Vilanova Estrada, Pere
Las tecnologías de la información y comunicación (TIC) están posibilitando grandes beneficios en el campo de la salud. En este sentido, KRISTINA (Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities) es un proyecto que forma parte del programa marco de investigación e innovación de la Unión Europea HORIZON 2020, a través del cual el proyecto ha sido financiado por la Comisión Europea con tres millones de euros. Su objetivo es desarrollar mediante tecnología digital un agente multilingüe con funcionalidades de comunicación en entornos sanitarios y de bienestar. El proyecto está orientado a las personas migradas, mayores de edad, de origen turco, así como a cuidadores polacos que trabajan en Alemania, y a la población magrebí residente en España. Actualmente se están definiendo las nuevas funcionalidades del segundo prototipo de avatar y se está evaluando la interacción humana entre el mismo y los usuarios., Information and communication technologies are bringing about major breakthroughs and benefits in the health field. The KRISTINA (Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities) project is part of the EU Framework Programme for Research and Innovation HORIZON 2020 and has received funding of three million euros from the European Commission. The main objective is to develop, through digital technologies, a multilingual agent with communicative functions to help immigrants overcome language barriers when seeking medical information on the network and contacting the health system. The targeted users are elderly people coming from Turkey to Germany, Polish caregivers working in Germany and Maghrebi migrants in Spain. The main functionalities of the first avatar prototype and its human interaction are currently being evaluated and tested by the main partners of the project consortium.
Repositorio Digital de la UPF
Publication Reseña (Review)

La interfaz estructura informativa-prosodia: el rol de la tematicidad jerárquica basado en un modelo empírico

  • Domínguez Bajo, Mónica
Tesis doctoral en tecnologías de la información y las comunicaciones realizada por Mónica Domínguez Bajo en la Universidad Pompeu Fabra bajo la dirección del Dr. Leo Wanner y la Dra. Mireia Farrús Cabecerán. El acto de defensa tuvo lugar el viernes 17 de noviembre de 2017 ante el tribunal formado por los doctores Bernd Möbius (Universidad de Saarland), Pilar Prieto Vives (Universidad Pompeu Fabra) y Catherine Lai (Universidad de Edimburgo). Obtuvo la califcación de Sobresaliente., PhD thesis in communication and information tecnologies written by Mónica Domínguez Bajo at the University Pompeu Fabra under the supervision of Dr. Leo Wanner and la Dra. Mireia Farrús Cabecerán. The author was examined on Friday, 17th November 2017 by a committee formed by the doctors Bernd Möbius (University of Saarland), Pilar Prieto Vives (University Pompeu Fabra) y Catherine Lai (University of Edinburgh). It obtained the grade of Excellent.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Data-driven sentence generation with non-isomorphic trees

  • Ballesteros, Miguel
  • Bohnet, Bernd
  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a la 2015 Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT 2015), celebrada del 31 de maig al 5 de juny 2015 a Denver (CO, EUA)., Abstract structures from which the generation naturally starts often do not contain any functional nodes, while surface-syntactic structures or a chain of tokens in a linearized tree contain all of them. Therefore, data-driven linguistic generation needs to be able to cope with the projection between non-isomorphic structures that differ in their topology and number of nodes. So far, such a projection has been a challenge in data-driven generation/nand was largely avoided. We present a fully stochastic generator that is able to cope with projection between non-isomorphic structures. The generator, which starts from PropBank-like structures, consists of a cascade/nof SVM-classifier based submodules that map in a series of transitions the input structures onto sentences. The generator has been evaluated for English on the Penn-Treebank and for Spanish on the multi-layered AncoraUPF corpus., Our work on deep stochastic sentence generation is partially supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Revising the METU-Sabancı Turkish treebank: an exercise in surface-syntactic annotation of agglutinative languages

  • Burga Díaz, Alicia
  • Öktem, Alp
  • Wanner, Leo
Comunicació presentada a: the Fourth International Conference on Dependency Linguistics (Depling 2017), celebrada a Pisa, Itàlia, del 18 al 20 de setembre 2017., In this paper, we present a revision of the training set of the METU-Sabancı Turkish syntactic dependency treebank composed of 4997 sentences in accordance with the principles of the Meaning-Text Theory (MTT). MTT reflects the multilayered nature of language by a linguistic model in which each linguistic phenomenon is treated at its corresponding level(s). Our analysis of the METU-Sabancı syntactic relation tagset reveals that it encodes deepmorphological and surface-syntactic phenomena, which should be separated according to the MTT model. We propose an alternative surface-syntactic relation annotation schema and show that this schema also allows for a sound projection of the obtained surface annotation onto a deepsyntactic annotation, as needed for the implementation of down-stream language understanding applications., The presented work has been funded by the European Commission as part of the H2020 Programme, under the contract numbers 645012–RIA and 700024-RIA.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A topic detection and visualisation system on social media posts

  • Andreadis, Stelios
  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: Internet Science. 4th International Conference, INSCI 2017 celebrada del 22 al 24 de novembre de 2017 a Thessaloniki, Grècia., Large amounts of social media posts are produced on a daily basis and monitoring all of them is a challenging task. In this direction we demonstrate a topic detection and visualisation tool in Twitter data, which filters Twitter posts by topic or keyword, in two different languages; German and Turkish. The system is based on state-of-the-art news clustering methods and the tool has been created to handle streams of recent news information in a fast and user-friendly way. The user interface and user-system interaction examples are presented in detail., This work was supported by the EC-funded projects H2020-645012 (KRISTINA) and H2020-700475 (beAWARE).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

A pipeline for the creation of progressively rendered web 3D scenes

  • Evans, Alun Thomas
  • Agenjo, Javier
  • Blat, Josep
We present an end-to-end pipeline for the export of 3D scenes from content creaton tools to a real-tme rendering engine in an embeddable web-page, including a novel system for compression/decompression of textured polygonal meshes. We show that the compression/decompression outperforms the best state-of-the-art non-progressive alternatve, especially as bandwidth increases, showing that web-specifc techniques should consider the whole user pipeline. Our pipeline also includes progressivity, which is paramount for a good interactve user experience, and permits full user interacton with lower resoluton versions of the 3D scenes, while progressively higher resoluton data is downloaded. Finally, we discuss how our method may be used in the future to facilitate the transfer of animated meshes.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

On the role of communicative structure in read aloud applications for elderly

  • Domínguez Bajo, Mónica
  • Burga Díaz, Alicia
  • Farrús, Mireia
  • Wanner, Leo
Presentat a: the Workshop on Intelligent Conversation Agents in Home and Geriatric Care Applications (AAMAS workshop), celebrat el 15 de juliol de 2018 a Estocolm, Suècia. Conjuntament amb the Federated AI Meeting (FAIM) i the 2nd Workshop on AI for Aging, Rehabilitation, and Independent Assisted Living., Conversational technologies that assist elderly people need to adapt to common disabilities in old age. Visual, hearing and even more so cognitive impairments pose serious difficulties for our seniors to handle a standard conversation with a human. Understanding a virtual agent may be ever harder. In this case, communicative strategies are key to adapt the virtual agent to the needs of elderly users. This paper addresses the role of the communicative structure for expressive speech prosody, which is known to be crucial for better speech comprehension. It reports on efforts to improve prosody within a text-to-speech system based on one aspect of the communicative structure, namely thematicity. The work has been implemented as an application in a social virtual agent, KRISTINA, which reads aloud news articles upon request for elderly users in German.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Attentional parallel RNNs for generating punctuation in transcribed speech

  • Öktem, Alp
  • Farrús, Mireia
  • Wanner, Leo
Until very recently, the generation of punctuation marks for automatic speech recognition (ASR) output has been mostly done by looking at the syntactic structure of the recognized utterances. Prosodic cues such as breaks, speech rate, pitch intonation that influence placing of punctuation marks on speech transcripts have been seldom used. We propose a method that uses recurrent neural networks, taking prosodic and lexical information into account in order to predict punctuation marks for raw ASR output. Our experiments show that an attention mechanism over parallel sequences of prosodic cues aligned with transcribed speech improves accuracy of punctuation generation., We would like to thank Francesco Barbieri for offering his technical insights throughout this work. This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number H2020-RIA-645012. The second author is partially funded by the Spanish Ministry of Economy, Industry and Competitiveness through the Ramón y Cajal program.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Paragraph prosodic patterns to enhance text-to-speech naturalness

  • Peiró Lilja, Àlex
  • Farrús, Mireia
Comunicació presentada a: the 9th International Conference on Speech Prosody 2018, celebrat del 13 al 16 de juny a Poznań, Polònia., Speech synthesis has reached a reasonable high quality in recent years. However, there is still room for improvement in terms of naturalness and expressiveness when dealing with large multisentential discourse, since most text-to-speech synthesizers do not fully take into account the prosodic differences that have been observed in discourse units such as paragraphs. This work presents an implementation of paragraph-based prosodic patterns into the open-source MARYTTS platform, enriching its prosody output by means of intra- and inter-paragraph prosodic features. The set of characteristics include pitch decay, pitch range and speech rate variation (as intra-paragraph features), as well as paragraph break pauses and speech rate variation (as inter-paragraph features), previously analyzed in a large set of TED Talks and read-speech sections of the Spoken Wikipedia Corpus. The perception tests, performed both in English and German parametric voices, suggest that paragraph-based features should be further studied and taken into account on future implementations to synthesize large discourse speech., The authors would like to thank all listeners that took part in the experiments. This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number H2020-RIA-645012. The second author is partially funded by the Spanish Ministry of Economy, Industry and Competitiveness through the Ram´on y Cajal program.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Integrating lexical and prosodic features for automatic paragraph segmentation

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Spoken documents, such as podcasts or lectures, are a growing presence in everyday life. Being able to automatically identify their discourse structure is an important step to understanding what a spoken document is about. Moreover, finer-grained units, such as paragraphs, are highly desirable for presenting and analyzing spoken content. However, little work has been done on discourse based speech segmentation below the level of broad topics. In order to examine how discourse transitions are cued in speech, we investigate automatic paragraph segmentation of TED talks using lexical and prosodic features. Experiments using Support Vector Machines, AdaBoost, and Neural Networks show that models using supra-sentential prosodic features and induced cue words perform better than those based on the type of lexical cohesion measures often used in broad topic segmentation. Moreover, combining a wide range of individually weak lexical and prosodic predictors improves performance, and modelling contextual information using recurrent neural networks outperforms other approaches by a large margin. Our best results come from using late fusion methods that integrate representations generated by separate lexical and prosodic models while allowing interactions between these features streams rather than treating them as independent information sources. Application to ASR outputs shows that adding prosodic features, particularly using late fusion, can significantly ameliorate decreases in performance due to transcription errors., The second author was funded from the EU’s Horizon 2020 Research and Innovation Programme under the GA H2020-RIA-645012 and the Spanish Ministry of Economy and Competitivity Juan de la Cierva program. The other authors were funded by the University of Edinburgh.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Description logics and rules for multimodal situational awareness in healthcare

  • Meditskos, Georgios
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: Video Browser Showdown (VBS'17) dins 23rd International Conference on MultiMedia Modeling (MMM'17); celebrats el 4 de gener de 2017 a Reykjavik, Islàndia., We present a framework for semantic situation understand- ing and interpretation of multimodal data using Description Logics (DL) and rules. More precisely, we use DL models to formally describe contex- tualised dependencies among verbal and non-verbal descriptors in multi- modal natural language interfaces, while context aggregation, fusion and interpretation is supported by SPARQL rules. Both background knowl- edge and multimodal data, e.g. language analysis results, facial expres- sions and gestures recognized from multimedia streams, are captured in terms of OWL 2 ontology axioms, the de facto standard formalism of DL models on the Web, fostering reusability, adaptability and interoperabil- ity of the framework. The framework has been applied in the eminent eld of healthcare, providing the models for the semantic enrichment and fusion of verbal and non-verbal descriptors in dialogue-based systems., This work has been partially supported by the H2020- 645012 project \KRISTINA: A Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities".
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

An automatic prosody tagger for spontaneous speech

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
Speech prosody is known to be central in advanced communication technologies. However, despite the advances of theoretical studies in speech prosody, so far, no large scale prosody annotated resources that would facilitate empirical research and the development of empirical computational approaches are available. This is to a large extent due to the fact that current common prosody annotation conventions offer a descriptive framework of intonation contours and phrasing based on labels. This makes it difficult to reach a satisfactory inter-annotator agreement during the annotation of gold standard annotations and, subsequently, to create consistent large scale annotations. To address this problem, we present an annotation schema for prominence and boundary labeling of prosodic phrases based upon acoustic parameters and a tagger for prosody annotation at the prosodic phrase level. Evaluation proves that interannotator agreement reaches satisfactory values, from 0.60 to 0.80 Cohen’s kappa, while the prosody tagger achieves acceptable recall and f-measure figures for five spontaneous samples used in the evaluation of monologue and dialogue formats in English and Spanish. The work presented in this paper is a first step towards a semi-automatic acquisition of large corpora for empirical prosodic analysis., This work is part of the KRISTINA project, which has received funding from the European Unions Horizon 2020 Research and Innovation Programme under the Grant Agreement number H2020-RIA-645012. It has been also partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502). The second author is partially funded by the Spanish Ministry of Economy and Competitivity through the Juan de la Cierva program., Paper presented at COLING 2016, the 26th International Conference on Computational Linguistics, 2016 Dec 11-17; Osaka, Japan.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Survey on automatic lip-reading in the era of deep learning

  • Fernandez-Lopez, Adriana
  • Sukno, Federico Mateo
In the last few years, there has been an increasing interest in developing systems for Automatic Lip-Reading (ALR). Similarly to other computer vision applications, methods based on Deep Learning (DL) have become very popular and have permitted to substantially push forward the achievable performance. In this survey, we review ALR research during the last decade, highlighting the progression from approaches previous to DL (which we refer to as traditional) toward end-to-end DL architectures. We provide a comprehensive list of the audio-visual databases available for lip-reading, describing what tasks they can be used for, their popularity and their most important characteristics, such as the number of speakers, vocabulary size, recording settings and total duration. In correspondence with the shift toward DL, we show that there is a clear tendency toward large-scale datasets targeting realistic application settings and large numbers of samples per class. On the other hand, we summarize, discuss and compare the different ALR systems proposed in the last decade, separately considering traditional and DL approaches. We address a quantitative analysis of the different systems by organizing them in terms of the task that they target (e.g. recognition of letters or digits and words or sentences) and comparing their reported performance in the most commonly used datasets. As a result, we find that DL architectures perform similarly to traditional ones for simpler tasks but report significant improvements in more complex tasks, such as word or sentence recognition, with up to 40% improvement in word recognition rates. Hence, we provide a detailed description of the available ALR systems based on end-to-end DL architectures and identify a tendency to focus on the modeling of temporal context as the key to advance the field. Such modeling is dominated by recurrent neural networks due to their ability to retain context at multiple scales (e.g. short- and long-term information). In this sense, current efforts tend toward techniques that allow a more comprehensive modeling and interpretability of the retained context., This work is partly supported by the Spanish Ministry of Economy and Competitiveness, Spain under project grant TIN2017-90124-P, the Ramon y Cajal Programme, the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), and the Kristina project funded by the European Union Horizon 2020 - Research and Innovation Framework Programme under grant agreement No. 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Cultural communication idiosyncrasies in human-computer interaction

  • Yoshino, Koichiro
  • Miehle, Juliana
  • Pragst, Louisa
  • Ultes, Stefan
  • Nakamura, Satoshi
  • Minker, Wolfgang
Comunicació presentada a: 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue; celebrada del 13 al 15 de setembre de 2016 a Los Angeles, USA, In this work, we investigate whether the cultural idiosyncrasies found in human human interaction may be transferred to human-computer interaction. With the aim of designing a culture-sensitive dialogue system, we designed a user study creating a dialogue in a domain that has the potential capacity to reveal cultural differences. The dialogue contains different options for the system output according to cultural differences. We conducted a survey among Germans and Japanese to investigate whether the supposed differences may be applied in human-computer interaction. Our results show that there are indeed differences, but not all results are consistent with the cultural models., This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645012. This research and development work was also supported by the MIC/SCOPE #152307004.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A social companion and conversation partner for elderly

  • Miehle, Juliana
  • Bagci, Ilker
  • Minker, Wolfgang
  • Ultes, Stefan
Comunicació presentada a: 8th International Workshop On Spoken Dialogue Systems (IWSDS), celebrat del 6 al 9 de juny a Farmington, Estats Units., In this work, we present the development and evaluation of a social companion and conversation partner for the special user group of elderly.With the aim of designing a user-adaptive system, we responded to the desires of the elderly which have been identified during various interviews and created a companion which talks and listens to the elderly users. Moreover, we conducted a user study with a small group of retired seniors living at home or in a nursing home. The results show that our companion and its dialogue were perceived very positively and that a social companion and conversation partner is indeed in demand by lonely seniors., This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Question answering over pattern-based user models

  • Meditskos, Georgios
  • Dasiopoulou, Stamatia
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a la 12th International Conference on Semantic Systems celebrada a Leipzig, Alemanya, del 13-14 de setembre de 2016., In this paper we present an ontology-driven framework for natural language question analysis and answering over user models (e.g. preferences, habits and health problems of individuals) that are formally captured using ontology design patterns. Pattern-based modelling is extremely useful for capturing n-ary relations in a well-defined and axiomatised manner, but it introduces additional challenges in building NL interfaces for accessing the underlying content. This is mainly due to the encapsulation of domain semantics inside conceptual layers of abstraction (e.g. using reification or container classes) that demand flexible, context-aware approaches for query analysis and interpretation. We describe the coupling of a frame-based formalisation of natural language user utterances with a context-aware query interpretation towards question answering over pattern-based RDF knowledge bases. The proposed framework is part of a human-like socially communicative agent that acts as an intermediate between elderly migrants and care personnel, assisting the latter to solicit personal information about care recipients (e.g. medical history, care needs, preferences, routines, habits, etc.)., This work has been partially supported by the H2020-645012 project /KRISTINA: A Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities".
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Multi-instance dynamic ordinal random fields for weakly-supervised pain intensity estimation

  • Ruiz Ovejero, Adrià
  • Rudovic, Ognjen
  • Binefa i Valls, Xavier
  • Pantic, Maja
Comunicació presentada a: Computer Vision – ACCV 2016, 13th Asian Conference on Computer Vision, celebrat a Taipei, Taiwan, del 20 al 24 de novembre de 2016., In this paper, we address the Multi-Instance-Learning (MIL) problem when bag labels are naturally represented as ordinal variables (Multi-Instance-Ordinal Regression). Moreover, we consider the case where bags are temporal sequences of ordinal instances. To model this, we propose the novel Multi-Instance Dynamic Ordinal Random Fields (MI-DORF). In this model, we treat instance-labels inside the bag as latent ordinal states. The MIL assumption is modelled by incorporating a high-order cardinality potential relating bag and instance-labels, into the energy function. We show the benefits of the proposed approach on the task of weakly-supervised pain intensity estimation from the UNBC Shoulder-Pain Database. In our experiments, the proposed approach significantly outperforms alternative non-ordinal methods that either ignore the MIL assumption, or do not model dynamic information in target data., This paper is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grants agreement no. 645012 (KRISTINA), no. 645094 (SEWA) and no. 688835 (DE-ENIGMA). Adria Ruiz would also like to acknowledge Spanish Government to provide support under grant FPU13/01740.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Extending WordNet with fine-grained collocational information via supervised distributional learning

  • Espinosa-Anke, Luis
  • Camacho-Collados, Jose
  • Rodríguez Fernández, Sara
  • Saggion, Horacio
  • Wanner, Leo
Comunicació presentada a COLING 2016: the 26th International Conference on Computational Linguistics; 2016 Dec. 11-16; Osaka (Japan), WordNet is probably the best known lexical resource in Natural Language Processing. While it/nis widely regarded as a high quality repository of concepts and semantic relations, updating and/nextending it manually is costly. One important type of relation which could potentially add enormous/nvalue toWordNet is the inclusion of collocational information, which is paramount in tasks/nsuch as Machine Translation, Natural Language Generation and Second Language Learning. In/nthis paper, we present ColWordNet (CWN), an extended WordNet version with fine-grained collocational/ninformation, automatically introduced thanks to a method exploiting linear relations/nbetween analogous sense-level embeddings spaces. We perform both intrinsic and extrinsic evaluations,/nand release CWN for the use and scrutiny of the community., This work was partially funded by the following projects: MULTISENSOR (FP7-ICT-610411),/nKRISTINA (H2020-645012-RIA), HARENES (FFI201130219-CO2-02), through a predoctoral grant/n(BES-2012-057036), the Mar´ıa de Maeztu Units of Excellence Program (MDM-2015-0502) and TUNER/n(TIN2015-65308-C5-5-R, MINECO/FEDER, UE). Jose Camacho-Collados is supported by a Google/nDoctoral Fellowship in Natural Language Processing.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Topic detection using the DBSCAN-Martingale and the time operator

  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
  • Antoniou, Ioannis
Comunicació presentada a: The 17th Conference of the Applied Stochastic Models and Data Analysis (ASMDA), celebrada del 6 al 9 de juny de 2017 a Londres, Regne Unit., Topic detection is usually considered as a decision process implemented in some relevant context, for example clustering. In this case, clusters correspond to topics that should be identifed. Density-based clustering, for example, uses only a density level E and a lower bound for the number of points in a cluster. As the density level is hard to be estimated, a stochastic process, called the DBSCANMartingale, is constructed for the combination of several outputs of DBSCAN for various randomly selected values of E in a predefned closed interval [0; Emax] from the uniform distribution. We have observed that most of the clusters are extracted in the interval [0; Emax=2], and moreover in the interval [Emax=2; Emax] the DBSCANMartingale stochastic process is less innovative, i.e. extracts only a few or no clusters. Therefore, non-symmetric skewed distributions are needed to generate density levels for the extraction of all clusters in a fast way. In this work we show that skewed distributions may be used instead of the uniform, so as to extract all clusters as quickly as possible. Experiments on real datasets show that the average innovation time of the DBSCAN-Martingale stochastic process is reduced when skewed distributions are employed, so less time is needed to extract all clusters., The first author would like to thank the Research Committee of the Aristo- tle University of Thessaloniki for awarding him the \Aristeia" postdoctoral scholarship 2016. Moreover, this work has been partially supported by the EC-funded project KRISTINA (H2020-645012).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Paragraph-based prosodic cues for speech synthesis applications

  • Farrús, Mireia
  • Lai, Catherine
  • Moore, Johanna D.
Paper presented at: Speech Prosody 2016; 2016 May 31-June 3; Boston (MA, USA), Speech synthesis has improved in both expressiveness and voice quality in recent years. However, obtaining full expressiveness when dealing with large multi-sentential synthesized discourse is still a challenge, since speech synthesizers do not take into account the prosodic differences that have been observed in discourse units such as paragraphs. The current study validates and extends previous work by analyzing the prosody of paragraph units in a large and diverse corpus of TED Talks using automatically extracted F0, intensity and timing features. In addition, a series of classification experiments was performed in order to identify which features are consistently used to distinguish paragraph breaks. The results show significant differences in prosody related to paragraph position. Moreover, the classification experiments show that boundary features such as pause duration and differences in F0 and intensity levels are the most consistent cues in marking paragraph boundaries. This suggests that these features should be taken into account when generating spoken discourse in order to improve naturalness and expressiveness., Part of this work has received funding from the EU’s Horizon 2020 Research and Innovation Programme under the GA H2020-RIA-645012. The first author is partially funded by the Spanish Ministry of Economy and Competitivity through the Juan de la Cierva program and a Jos´e Castillejo mobility grant
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Recurrent neural network grammars

  • Dyer, Chris
  • Kuncoro, Adhiguna
  • Ballesteros, Miguel
  • Smith, Noah A.
Comunicació presentada a la 2016 Conference of the North American Chapter of the Association for Computational Linguistics, celebrada a San Diego (CA, EUA) els dies 12 a 17 de juny 2016., We introduce recurrent neural network grammars,/nprobabilistic models of sentences with/nexplicit phrase structure. We explain efficient/ninference procedures that allow application to/nboth parsing and language modeling. Experiments/nshow that they provide better parsing in/nEnglish than any single previously published/nsupervised generative model and better language/nmodeling than state-of-the-art sequential/nRNNs in English and Chinese., This work was sponsored in part by the Defense/nAdvanced Research Projects Agency (DARPA)/nInformation Innovation Office (I2O) under the/nLow Resource Languages for Emergent Incidents/n(LORELEI) program issued by DARPA/I2O under/nContract No. HR0011-15-C-0114; it was also supported/nin part by Contract No. W911NF-15-1-0543/nwith the DARPA and the Army Research Office/n(ARO). Approved for public release, distribution/nunlimited. The views expressed are those of the authors/nand do not reflect the official policy or position/nof the Department of Defense or the U.S. Government./nMiguel Ballesteros was supported by the/nEuropean Commission under the contract numbers/nFP7-ICT-610411 (project MULTISENSOR) and/nH2020-RIA-645012 (project KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Dem@Home: ambient intelligence for clinical support of people living with dementia

  • Andreadis, Stelios
  • Stavropoulos, Thanos G.
  • Meditskos, Georgios
  • Kompatsiaris, Ioannis
Paper presented at the 1st Workshop on Semantic Web Technologies for Mobile and Pervasive Environments co-located with the 13th Extended Semantic Web Conference (ESWC 2016); 2016 May 29; Heraklion (Greece)., With the ever-growing prevalence of dementia, nursing costs are in-creasing, while the ability to live independently vanishes. Dem@Home is an ambient assisted living framework to support independent living while receiv-ing intelligent clinical care. Dem@Home integrates a variety of ambient and wearable sensors together with sophisticated, interdisciplinary methods of im-age and semantic analysis. Semantic Web technologies, such as OWL 2, are ex-tensively employed to represent sensor observations and application domain specifics as well as to implement hybrid activity recognition and problem detec-tion. Complete with tailored user interfaces, clinicians are provided with accu-rate monitoring of multiple life aspects, such as physical activity, sleep, com-plex daily tasks and clinical problems, leading to adaptive non-pharmaceutical interventions. The method has been already validated for both recognition per-formance and improvement on a clinical level, in four home pilots., This work has been supported by the H2020-ICT-645012 project KRISTINA: A Knowledge-Based Information Agent with Social Competence and Human Interac-tion Capabilities.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Towards expressive prosody generation in TTS for reading aloud applications

  • Domínguez Bajo, Mónica
  • Burga Díaz, Alicia
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: IberSpeech 2018, celebrat a Barcelona del 21 al 23 de novembre de 2018., Conversational technologies that assist elderly people need to adapt to common disabilities in old age. Visual, hearing and even more so cognitive impairments pose serious difficulties for our seniors to handle a standard conversation with a human. Understanding a virtual agent may be ever harder. In this case, communicative strategies are key to adapt the virtual agent to the needs of elderly users. This paper addresses the role of the communicative structure for expressive speech prosody, which is known to be crucial for better speech comprehension. It reports on efforts to improve prosody within a text-to-speech system based on one aspect of the communicative structure, namely thematicity. The work has been implemented as an application in a social virtual agent, KRISTINA, which reads aloud news articles upon request for elderly users in German.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Query-based topic detection using concepts and named entities

  • Gialampoukidis, Ilias
  • Liparas, Dimitris
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: 1st International Workshop on Multimodal Media Data Analytics, celebrat juntament amb 22nd European Conference on Artificial Intelligence (ECAI 2016), el 30 d'agost de 2016 a La Haia, Holanda., In this paper, we present a framework for topic detection in news articles. The framework receives as input the results retrieved from a query-based search and clusters them by topic. To this end, the recently introduced “DBSCAN-Martingale” method for automatically estimating the number of topics and the well-established Latent Dirichlet Allocation topic modelling approach for the assignment of news articles into topics of interest, are utilized. Furthermore, the proposed query-based topic detection framework works on high-level textual features (such as concepts and named entities) that are extracted from news articles. Our topic detection approach is tackled as a text clustering task, without knowing the number of clusters and compares favorably to several text clustering approaches, in a public dataset of retrieved results, with respect to four representative queries., This work was supported by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012), funded by the European Commission.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

From emotions to action units with hidden and semi-hidden-task learning

  • Ruiz, Adrià
  • Van de Weijer, Joost
  • Binefa i Valls, Xavier
Comunicació presentada a: IEEE International Conference on Computer Vision (ICCV 2015) celebrat a Santiago, Xile, de l'11 al 18 de desembre de 2015., Limited annotated training data is a challenging problem in Action Unit recognition. In this paper, we investigate how the use of large databases labelled according to the 6 universal facial expressions can increase the generalization ability of Action Unit classifiers. For this purpose, we propose a novel learning framework: Hidden-Task Learning. HTL aims to learn a set of Hidden-Tasks (Action Units) for which samples are not available but, in contrast, training data is easier to obtain from a set of related Visible- Tasks (Facial Expressions). To that end, HTL is able to exploit prior knowledge about the relation between Hidden and Visible-Tasks. In our case, we base this prior knowledge on empirical psychological studies providing statistical correlations between Action Units and universal facial expressions. Additionally, we extend HTL to Semi-Hidden Task Learning (SHTL) assuming that Action Unit training samples are also provided. Performing exhaustive experiments over four different datasets, we show that HTL and SHTL improve the generalization ability of AU classifiers by training them with additional facial expression data. Additionally, we show that SHTL achieves competitive performance compared with state-of-the-art Transductive Learning approaches which face the problem of limited training data by using unlabelled test samples during training., This paper is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645012. Adria Ruiz and Xavier Binefa would also like to acknowledge Spanish Government to provide support under grants CICYT TIN2012-39203 and FPU13/01740. Joost van de Weijer acknowledges Project TIN2013-41751 of the Spanish Ministry of Science and the Generalitat de Catalunya Project under Grant 2014-SGR-221.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Automatic viseme vocabulary construction to enhance continuous lip-reading

  • Fernandez-Lopez, Adriana
  • Sukno, Federico Mateo
Comunicació presentada a: 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), celebrat del 27 de febrer a l'1 de març de 2017 a Porto, Portugal., Speech is the most common communication method between humans and involves the perception of both auditory and visual channels. Automatic speech recognition focuses on interpreting the audio signals, but it has been demonstrated that video can provide information that is complementary to the audio. Thus, the study of automatic lip-reading is important and is still an open problem. One of the key challenges is the definition of the visual elementary units (the visemes) and their vocabulary. Many researchers have analyzed the importance of the phoneme to viseme mapping and have proposed viseme vocabularies with lengths between 11 and 15 visemes. These viseme vocabularies have usually been manually defined by their linguistic properties and in some cases using decision trees or clustering techniques. In this work, we focus on the automatic construction of an optimal viseme vocabulary based on the association of phonemes with similar appearance. To this end, we construct an automatic system that uses local appearance descriptors to extract the main characteristics of the mouth region and HMMs to model the statistic relations of both viseme and phoneme sequences. To compare the performance of the system different descriptors (PCA, DCT and SIFT) are analyzed. We test our system in a Spanish corpus of continuous speech. Our results indicate that we are able to recognize approximately 58% of the visemes, 47% of the phonemes and 23% of the words in a continuous speech scenario and that the optimal viseme vocabulary for Spanish is composed by 20 visemes., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Ramon y Cajal fellowships and the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), and the Kristina project funded by the European Union Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Data-driven deep-syntactic dependency parsing

  • Ballesteros, Miguel
  • Bohnet, Bernd
  • Mille, Simon
  • Wanner, Leo
‘Deep-syntactic’ dependency structures that capture the argumentative, attributive and co-/nordinative relations between full words of a sentence have a great potential for a number/nof NLP-applications. The abstraction degree of these structures is in between the output/nof a syntactic dependency parser (connected trees defined over all words of a sentence and/nlanguage-specific grammatical functions) and the output of a semantic parser (forests of trees/ndefined over individual lexemes or phrasal chunks and abstract semantic role labels which/ncapture the frame structures of predicative elements and drop all attributive and coordinative/ndependencies). We propose a parser that provides deep-syntactic structures. The parser has/nbeen tested on Spanish, English and Chinese, The work reported on in this paper has been partially funded by the European Commission under the contract numbers FP7-ICT-610411 (MULTISENSOR) and H2020-645012-RIA (KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Hybrid approach for biomedical relation extraction using finite state automata and random forest-weighted fusion

  • Mavropoulos, Thanassis
  • Liparas, Dimitris
  • Symeonidis, Spyridon
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: The 18th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2017), celebrada a Budapest, Hungria, del 17 al 23 d'abril de 2017., The automatic extraction of relations between medical entities found in related texts is considered to be a very important task, due to the multitude of applications that it can support, from question answering systems to the devel-opment of medical ontologies. Many different methodologies have been pre-sented and applied to this task over the years. Of particular interest are hybrid approaches, in which different techniques are combined in order to improve the individual performance of either one of them. In this study, we extend a previ-ously established hybrid framework for medical relation extraction, which we modify by enhancing the pattern-based part of the framework and by applying a more sophisticated weighting method. Most notably, we replace the use of regu-lar expressions with finite state automata for the pattern-building part, while the fusion part is replaced by a weighting strategy that is based on the operational capabilities of the Random Forests algorithm. The experimental results indicate the superiority of the proposed approach against the aforementioned well-established hybrid methodology and other state-of-the-art approaches., This work was supported by the project KRISTINA (H2020-645012), funded by the European Commission. Deidentified clinical records used in this research were provided by the i2b2 National Center for Biomedical Computing funded by U54LM008748 and were originally prepared for the Shared Tasks for Challenges in NLP for Clinical Data organized by Dr. Ozlem Uzuner, i2b2 and SUNY.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Adaptive dialogue management in the KRISTINA project for multicultural health care applications

  • Pragst, Louisa
  • Ultes, Stefan
  • Kraus, Matthias
  • Minker, Wolfgang
Pòster presentat a: The 19thWorkshop on the Semantics and Pragmatics of Dialogue, dins SEMDIAL 2015, celebrat del 24 al 26 d'agost de 2015 a Gothenburg, Suècia., The goal of the EU-funded KRISTINA project is to help migrants in European countries get information about their resident country’s health care system by the means of a socially competent dialogue system. This system has to be able to handle a considerably large dialogue domain as well as hold a natural conversation whilst taking into account the cultural background as well as the current emotional state of its dialogue partner. Dialogue management, as core component responsible for the course of the conversation, therefore needs to be able to meet these challenges. Our research is focused on adaptiveness of dialogue management to the cultural background and emotional state of the user as well as the generation of appropriate emotional responses. Furthermore the benefits of the integration of a reasoner will be investigated., This work is part of a project that has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Gradient-based steering for vision-based crowd simulation algorithms

  • Dutra, T. B.
  • Marques, Ricardo
  • Cavalcante‐Neto, J.B.
  • Vidal, C. A.
  • Pettré, J.
Most recent crowd simulation algorithms equip agents with a synthetic vision component for steering. They offer promising perspectives through a more realistic simulation of the way humans navigate according to their perception of the surrounding environment. In this paper, we propose a new perception/motion loop to steering agents along collision free trajectories that significantly improves the quality of vision-based crowd simulators. In contrast with solutions where agents avoid collisions in a purely reactive (binary) way, we suggest exploring the full range of possible adaptations and retaining the locally optimal one. To this end, we introduce a cost function, based on perceptual variables, which estimates an agent’s situation considering both the risks of future collision and a desired destination. We then compute the partial derivatives of that function with respect to all possible motion adaptations. The agent then adapts its motion by following the gradient. This paper has thus two main contributions: the definition of a general purpose control scheme for steering synthetic vision-based agents; and the proposition of cost functions for evaluating the perceived danger of the current situation. We demonstrate improvements in several cases., T. B. Dutra acknowl1edges CAPES for the fellowship (PDSE Proc. 0130/13-3) and J. Ondˇrej for helping in the implementation of his model. R. Marques acknowledges the projects Percolation (ANR- 13-JS02-0008), and Kristina (H2020-RIA-645012) for partially financing this research.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Using prosody to classify discourse relations

  • Kleinhans, Janine
  • Farrús, Mireia
  • Gravano, Agustín
  • Pérez, Juan Manuel
  • Lai, Catherine
  • Wanner, Leo
Comunicació presentada a: The 18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), celebrada a Estocolm, Suència, del 20 al 24 d'agost de 2017., This work aims to explore the correlation between the discourse structure of a spoken monologue and its prosody by predicting discourse relations from different prosodic attributes. For this purpose, a corpus of semi-spontaneous monologues in English has been automatically annotated according to the Rhetorical Structure Theory, which models coherence in text via rhetorical relations. From corresponding audio files, prosodic features such as pitch, intensity, and speech rate have been extracted from different contexts of a relation. Supervised classification tasks using Support Vector Machines have been performed to find relationships between prosodic features and rhetorical relations. Preliminary results show that intensity combined with other features extracted from intra- and intersegmental environments is the feature with the highest predictability for a discourse relation. The prediction of rhetorical relations from prosodic features and their combinations is straightforwardly applicable to several tasks such as speech understanding or generation. Moreover, the knowledge of how rhetorical relations should be marked in terms of prosody will serve as a basis to improve speech synthesis applications and make voices sound more natural and expressive., This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number 645012. The second author is partially funded by the Spanish Ministry of Economy, Industry and Competitiveness through the Ramón y Cajal program. The third and fourth authors are partially funded by ANPCYT PICT 2014-1561, and the Air Force Office of Scientific Research, Air Force Material Command, USAF under Award No. FA9550-15-1-0055.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Example-based acquisition of fine-grained collocational resources

  • Rodríguez Fernández, Sara
  • Carlini, Roberto
  • Espinosa-Anke, Luis
  • Wanner, Leo
Comunicació presentada a la Tenth International Conference on Language Resources and Evaluation, celebrada a Portoroz (Eslovènia) els dies 23 a 28 de maig 2016., Collocations such as heavy rain or make [a] decision, are combinations of two elements where one (the base) is freely chosen, while the/nchoice of the other (collocate) is restricted, depending on the base. Collocations present difficulties even to advanced language learners,/nwho usually struggle to find the right collocate to express a particular meaning, e.g., both heavy and strong express the meaning ‘intense’,/nbut while rain selects heavy, wind selects strong. Lexical Functions (LFs) describe the meanings that hold between the elements of/ncollocations, such as ‘intense’, ‘perform’, ‘create’, ‘increase’, etc. Language resources with semantically classified collocations would/nbe of great help for students, however they are expensive to build, since they are manually constructed, and scarce. We present an unsupervised/napproach to the acquisition and semantic classification of collocations according to LFs, based on word embeddings in which,/ngiven an example of a collocation for each of the target LFs and a set of bases, the system retrieves a list of collocates for each base and LF., The present work has been partially funded by the Spanish Ministry of Economy and Competitiveness (MINECO), through a predoctoral grant (BES-2012-057036) in the framework of the project HARenES (FFI2011-30219-C02-02), by the European Commission under the contract number H2020-RIA-645012, and by the ICT PhD program of Universitat Pompeu Fabra through a travel grant.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Ontology-driven context interpretation and conflict resolution for dialogue-based home care assistance

  • Meditskos, Georgios
  • Kontopoulos, Efstratios
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: The 9th International Conference Semantic Web Applications and Tools for Life Sciences (SWAT4LS), celebrat a Amsterdam, Holanda, del 5 al 8 de desembre de 2016, In this paper we present a framework for conversational awareness and conflict resolution in spoken dialogue systems for home care assistance. Conver-sational awareness is supported through OWL ontologies for capturing conver-sational modalities, while interpretation and incremental context enrichment is facilitated through Description Logics reasoning. Conflict resolution further as-sists the interaction with end users, facilitating exception handling and context prioritisation by coupling defeasible logics with medical and profile information., This work has been partially supported by the H2020-645012 project “KRISTINA: A Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities”.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Transition-based spinal parsing

  • Ballesteros, Miguel
  • Carreras, Xavier
Comunicació presentada a la 19th Conference on Computational Language Learning, celebrada els dies 30 i 31 de juliol 2015 a Beijing (China) i organitzada per l'ACL Special Interest Group on Natural Language Learning., We present a transition-based arc-eager model to parse spinal trees, a dependencybased representation that includes phrasestructure information in the form of constituent spines assigned to tokens. As a main advantage, the arc-eager model can use a rich set of features combining dependency and constituent information, while parsing in linear time. We describe a set of conditions for the arc-eager system to produce valid spinal structures. In experiments using beam search we show that the model obtains a good trade-off between speed and accuracy, and yields state of the art performance for both dependency and constituent parsing measures., Miguel Ballesteros is supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Crater monitoring through social media observations

  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Pòster presentat a: European Planetary Science Congress 2017, celebrat a Riga, Letònia, del 17 al 22 de setembre de 2017., Lunar craters have attracted the attention of not only scientists but also citizens. Modern high-resolution cameras with zoom capabilities allow citizens to capture and share pictures of the Moon in Social Media platforms, such as Twitter. We have collected 69 pictures of the Moon, from 01-01-2017 to 17-04-2017, that have been uploaded on Twitter and have been associated with the keyword #crater. The lunar pictures are indexed using SIFT descriptors and are then clustered using density-based approaches to group them into the automatically detected levels of zoom., This work has supported by the EC-funded project KRISTINA (H2020-645012).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Towards reasoned modality selection in an embodied conversation agent

  • Ten-Ventura, Carla
  • Carlini, Roberto
  • Dasiopoulou, Stamatia
  • Llorach, Gerard
  • Wanner, Leo
Comunicació presentada a: the 17th International Conference, IVA 2017, celebrat del 27 al 30 d'agost de 2017 a Estocolm, Suècia., We present work in progress on (verbal, facial, and gestural) modality selection in an embodied multilingual and multicultural conversation agent. In contrast to most of the recent proposals, which consider non-verbal behavior as being superimposed on and/or derived from the verbal modality, we argue for a holistic model that assigns modalities to individual content elements in accordance with semantic and contextual constraints as well as with cultural and personal characteristics of the addressee. Our model is thus in line with the SAIBA framework, although methodological differences become apparent at a more fine-grained level of realization., The presented work is funded by the European Commission as part of the H2020 Programme, under the contract number 645012{RIA. Many thanks to the three reviewers for their very helpful comments and suggestions.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Multi-instance dynamic ordinal random fields for weakly supervised dacial behavior analysis

  • Ruiz, Adrià
  • Rudovic, Ognjen
  • Binefa i Valls, Xavier
  • Pantic, Maja
We propose a multi-instance-learning (MIL) approach for weakly supervised learning problems, where a training set is formed by bags (sets of feature vectors or instances) and only labels at bag-level are provided. Specifically, we consider the multi-instance dynamic-ordinal-regression (MI-DOR) setting, where the instance labels are naturally represented as ordinal variables and bags are structured as temporal sequences. To this end, we propose MI dynamic ordinal random fields (MI-DORF). In this paper, we treat instance-labels as temporally dependent latent variables in an undirected graphical model. Different MIL assumptions are modelled via newly introduced high-order potentials relating bag and instance-labels within the energy function of the model. We also extend our framework to address the partially observed MI-DOR problem, where a subset of instance labels is also available during training. We show on the tasks of weakly supervised facial action unit and pain intensity estimation, that the proposed framework outperforms alternative learning approaches. Furthermore, we show that MI-DORF can be employed to reduce the data annotation efforts in this context by large-scale., This work was supported by the European Union’s Horizon 2020 Research and Innovation Programme under Grant 645012. The work of O. Rudovic was supported by the H2020 Research Program through the Marie Skodowska-Curie Grant Agreement under Grant 701236 (EngageME). The work of X. Binefa was supported in part by Spanish Government under Grant MINECO TIN2017-90124-P and in part by the Generalitat de Catalunya under Grant MINECO TIN2017-90124-P and Grant AGAUR 2017-SGR-1311
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Demo of FORGe: the Pompeu Fabra open rule-based generator

  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a: The 10th International Natural Language Generation conference, celebrat del 3 al 4 de setembre de 2017 a Santiago de Compostela, This demo paper presents the multilingual deep sentence generator developed by the TALN group at Universitat Pompeu Fabra, implemented as a series of rule-based graphtransducers., The work described in this paper has been partially funded by the European Commission under the contract numbers FP7-ICT-610411, H2020-645012-RIA, H2020-700024-RIA, and H2020-700475-RIA.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

KRISTINA: a knowledge-based virtual conversation agent

  • Wanner, Leo
  • André, Elisabeth
  • Blat, Josep
  • Dasiopoulou, Stamatia
  • Farrús, Mireia
  • Kamateri, Eleni
  • Lingenfelser, Florian
  • Llorach, Gerard
  • Martínez, Oriol
  • Meditskos, Georgios
  • Mille, Simon
  • Minker, Wolfgang
  • Pragst, Louisa
  • Schiller, Dominik
  • Stam, Andries
  • Stellingwerff, Ludo
  • Sukno, Federico Mateo
  • Vieru, Bianca
  • Vrochidis, Stefanos
Comunicació presentada a: The 15th International Conference, PAAMS 2017, celebrada a Porto, Portugal, del 21 al 23 de juny de 2017, We present an intelligent embodied conversation agent with linguistic, social and emotional competence. Unlike the vast majority of the state-of-the-art conversation agents, the proposed agent is constructed around an ontology-based knowledge model that allows for exible reasoning-driven dialogue planning, instead of using prede ned dialogue scripts. It is further complemented by multimodal communication analysis and generation modules and a search engine for the retrieval of multimedia background content from the web needed for conducting a conversation on a given topic. The evaluation of the 1st prototype of the agent shows a high degree of acceptance of the agent by the users with respect to its trustworthiness, naturalness, etc. The individual technologies are being further improved in the 2nd prototype., The presented work is funded by the European Commission as part of the H2020 Programme, under the contract number 645012-RIA.
Repositorio Digital de la UPF
Publication Capítulo o parte de un libro (BookPart)

Recurrent neural network interaction quality estimation

  • Pragst, Louisa
  • Ultes, Stefan
  • Minker, Wolfgang
Getting a good estimation of the Interaction Quality (IQ) of a spoken dialogue helps to increase the user satisfaction as the dialogue strategy may be adapted accordingly. Therefore, some research has already been conducted in order to automatically estimate the Interaction Quality. This paper adds to this by describing how Recurrent Neural Networks may be used to estimate the Interaction Quality for each dialogue turn and by evaluating their performance on this task. Here, we will show that RNNs may outperform non-recurrent neural networks., This paper is part of a project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Fast visual vocabulary construction for image retrieval using skewed-split k-d trees

  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a la 22nd International Conference on MultiMedia Modeling (MMM16), celebrada els dies 4 a 6 de gener 2016 a Miami (FL, EUA)., Most of the image retrieval approaches nowadays are based on the Bag-of-Words (BoW) model, which allows for representing an image efficiently and quickly. The efficiency of the BoW model is related to the efficiency of the visual vocabulary. In general, visual vocabularies are created by clustering all available visual features, formulating specific patterns. Clustering techniques are k-means oriented and they are replaced by approximate k-means methods for very large datasets. In this work, we propose a faster construction of visual vocabularies compared to the existing method in the case of SIFT descriptors, based on our observation that the values of the 128-dimensional SIFT descriptors follow the exponential distribution. The application of our method to image retrieval in specific image datasets showed that the mean Average Precision is not reduced by our approximation, despite that the visual vocabulary has been constructed significantly faster compared to the state of the art methods., This work was supported by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012), funded by the European Commission.
Repositorio Digital de la UPF
Publication Artículo preliminar (WorkingPaper)

FORGe at WebNLG 2017

  • Mille, Simon
  • Dasiopoulou, Stamatia
This paper describes the FORGe generator at E2E. The input triples are mapped onto sentences by applying a series of rule-based graph-transducers and aggregation grammars to template predicate-argument structures associated to each property. We submitted two primary systems to the task, one based on the grammars, and one based on templates, and one secondary system, which is a variation of the grammar-based one.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Compilation of Corpora for the study of the Information Structure-Prosody interface

  • Domínguez Bajo, Mónica
  • Burga Díaz, Alicia
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: The Language Resources and Evaluation Conferece, celebrada del 7 al 12 de maig de 2018 a Miyazaki, Japó., Theoretical studies on the Information Structure–prosody interface argue that the content packaged in terms of theme and rheme correlates with the intonation of the corresponding sentence. However, there are few empirical studies that support this argument and even fewer resources that promote reproducibility and scalability of experiments. In this paper, we introduce a methodology for the compilation of annotated corpora to study the correspondence between Information Structure and prosody. The application of this methodology is exemplified on a corpus of read speech in English annotated with hierarchical thematicity and automatically extracted prosodic parameters., This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number H2020-RIA-645012. It has been also partly supported by the Spanish Ministry of Economy and Competitiveness under the Mar´ıa de Maeztu Unit of Excellence Programme (MDM-2015-0502), and the third author is partially funded by the Ramón y Cajal program.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Distilling an ensemble of greedy dependency parsers into one MST parser

  • Kuncoro, Adhiguna
  • Ballesteros, Miguel
  • Kong, Lingpeng
  • Dyer, Chris
  • Smith, Noah A.
Comunicació presentada a la 2016 Conference on Empirical Methods in Natural Language Processing, celebrada de l'1 al 5 de novembre de 2016 a Austin, Texas., We introduce two first-order graph-based dependency parsers achieving a new state of the art. The first is a consensus parser built from an ensemble of independently trained greedy LSTM transition-based parsers with different random initializations. We cast this approach as minimum Bayes risk decoding (under the Hamming cost) and argue that weaker consensus within the ensemble is a useful signal of difficulty or ambiguity. The second parser is a “distillation” of the ensemble into a single model. We train the distillation parser using a structured hinge loss objective with a novel cost that incorporates ensemble uncertainty estimates for each possible attachment, thereby avoiding the intractable crossentropy computations required by applying standard distillation objectives to problems with structured outputs. The first-order distillation parser matches or surpasses the state of the art on English, Chinese, and German., We thank Swabha Swayamdipta, Sam Thomson, Jesse Dodge, Dallas Card, Yuichiro Sawai, Graham Neubig, and the anonymous reviewers for useful feedback. We also thank Juntao Yu and Bernd Bohnet for re-running the parser of Bohnet and Nivre (2012) on Chinese with gold tags. This work was sponsored in part by the Defense Advanced Research Projects Agency (DARPA) Information Innovation Office (I2O) under the Low Resource Languages for Emergent Incidents (LORELEI) program issued by DARPA/I2O under Contract No. HR0011-15-C-0114; it was also supported in part by Contract No. W911NF-15-1-0543 with the DARPA and the Army Research Office (ARO). Approved for public release, distribution unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. Miguel Ballesteros was supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Interaction quality estimation using long short-term memories

  • Rach, Niklas
  • Minker, Wolfgang
  • Ultes, Stefan
Comunicació presentada a SIGDIAL 2017 Conference, the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, celebrada del 15 al 17 d'agost a Saarbrucken, Alemanya., For estimating the Interaction Quality (IQ) in Spoken Dialogue Systems (SDS), the dialogue history is of significant importance. Previous works included this information manually in the form of precomputed temporal features into the classification process. Here, we employ a deep learning architecture based on Long Short-Term Memories (LSTM) to extract this information automatically from the data, thus estimating IQ solely by using current exchange features. We show that it is thereby possible to achieve competitive results as in a scenario where manually optimized temporal features have been included., This work is part of a project that has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Automatic paragraph segmentation with lexical and prosodic features

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Comunicació presentada a la Interspeech 2016, celebrada per la International Speech Communication Association (ISCA) els dies 8 a 12 de septembre de 2016 a San Francisco (EUA)., As long-form spoken documents become more ubiquitous in everyday life, so does the need for automatic discourse segmentation in spoken language processing tasks. Although previous work has focused on broad topic segmentation, detection of finer-grained discourse units, such as paragraphs, is highly desirable for presenting and analyzing spoken content. To better understand how different aspects of speech cue these subtle discourse transitions, we investigate automatic paragraph segmentation of TED talks. We build lexical and prosodic paragraph segmenters using Support Vector Machines, AdaBoost, and Long Short Term Memory (LSTM) recurrent neural networks. In general, we find that induced cue words and supra-sentential prosodic features outperform features based on topical coherence, syntactic form and complexity. However, our best performance is achieved by combining a wide range of individually weak lexical and prosodic features, with the sequence modelling LSTM generally outperforming the other classifiers by a large margin. Moreover, we find that models that allow lower level interactions between different feature types produce better results than treating lexical and prosodic contributions as separate, independent information sources., The second author is funded from the EU’s Horizon 2020 Research and Innovation Programme under the GA H2020-RIA-645012 and the Spanish Ministry of Economy and Competitivity Juan de la Cierva program.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Semantics-driven recognition of collocations using word embeddings

  • Rodríguez Fernández, Sara
  • Espinosa-Anke, Luis
  • Carlini, Roberto
  • Wanner, Leo
L2 learners often produce “ungrammatical” word combinations such as, e.g., *give a suggestion or *make a walk. This is because of the “collocationality” of one of their items (the base) that limits the acceptance of collocates to express a specific meaning (‘perform’ above). We propose an algorithm that delivers, for a given base and the intended meaning of a collocate, the actual collocate lexeme(s) (make / take above). The algorithm exploits the/nlinear mapping between bases and collocates from examples and generates a collocation transformation matrix which is then applied to novel unseen cases. The evaluation shows a promising line of research in collocation discovery., The present work has been partially funded by the Spanish Ministry of Economy and Competitiveness/n(MINECO), through a predoctoral grant (BES-2012-057036) in the framework of the project HARenES (FFI2011-30219-C02-02), and by the European Commission under the grant number H2020–645012–RIA. We also acknowledge support from the Maria de Maeztu Excellence Program (MDM-2015-0502).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Multimedia interactive search engine based on graph-based and non-linear multimodal fusion

  • Moumtzidou, Anastasia
  • Gialampoukidis, Ilias
  • Mironidis, Theodoros
  • Liparas, Dimitris
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: 14th International Workshop on Content-Based Multimedia Indexing (CBMI 2016) celebrat del 15 al 17 de juny de 2016 a Bucarest, Romania., This paper presents an interactive multimedia search engine, which is capable of searching into multimedia collections by fusing textual and visual information. Apart from multimedia search, the engine is able to perform text search and image retrieval independently using both high-level and lowlevel information. The images of the multimedia collection are organized by color, offering fast browsing in the image collection., This work was partially supported by the European Commission by the projects MULTISENSOR (FP7-610411), HOMER (FP7-312883) and KRISTINA (H2020-645012).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Towards estimating the upper bound of visual-speech recognition: the visual lip-reading feasibility databas

  • Fernandez-Lopez, Adriana
  • Martinez, Oriol
  • Sukno, Federico Mateo
Comunicació presentada a: FG 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition, celebrada del 30 de maig al 3 de juny de 2017 a Washington, Estats Units d'Amèrica., Speech is the most used communication method between humans and it involves the perception of auditory and visual channels. Automatic speech recognition focuses on interpreting the audio signals, although the video can provide information that is complementary to the audio. Exploiting the visual information, however, has proven challenging. On one hand, researchers have reported that the mapping between phonemes and visemes (visual units) is one-to-many because there are phonemes which are visually similar and indistinguishable between them. On the other hand, it is known that some people are very good lip-readers (e.g: deaf people). We study the limit of visual only speech recognition in controlled conditions. With this goal, we designed a new database in which the speakers are aware of being read and aim to facilitate lip-reading. In the literature, there are discrepancies on whether hearingimpaired people are better lip-readers than normal-hearing people. Then, we analyze if there are differences between the lip-reading abilities of 9 hearing-impaired and 15 normalhearing people. Finally, human abilities are compared with the performance of a visual automatic speech recognition system. In our tests, hearing-impaired participants outperformed the normal-hearing participants but without reaching statistical significance. Human observers were able to decode 44% of the spoken message. In contrast, the visual only automatic system achieved 20% of word recognition rate. However, if we repeat the comparison in terms of phonemes both obtained very similar recognition rates, just above 50%. This suggests that the gap between human lip-reading and automatic speechreading might be more related to the use of context than to the ability to interpret mouth appearance., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Ramon y Cajal fellowships and the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), and the Kristina project funded by the European Union Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

VERGE in VBS 2018

  • Moumtzidou, Anastasia
  • Andreadis, Stelios
  • Markatopoulou, Fotini
  • Galanopoulos, Damianos
  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Mezaris, Vasileios
  • Kompatsiaris, Ioannis
  • Patras, Ioannis
Comunicació presentada a: MultiMedia Modeling 24th International Conference, MMM 2018 celebrada a Bangkok, Tailàndia, del 5 al 7 de febrer de 2018., This paper presents VERGE interactive video retrieval engine, which is capable of browsing and searching into video content. The system integrates several content-based analysis and retrieval mod- ules including concept detection, clustering, visual and textual similarity search, query analysis and reranking, as well as multimodal fusion., This work was supported by the EU's Horizon 2020 research and innovation programme under grant agreements H2020-645012 KRISTINA, H2020-779962 V4Design, H2020-732665 EMMA, and H2020-687786 InVID.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Improved transition-based parsing by modeling characters instead of words with LSTMs

  • Ballesteros, Miguel
  • Dyer, Chris
  • Smith, Noah A.
Comunicació presentada a la 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), organitzada pel SIGDAT i celebrada els dies 17 a 21 de setembre 2015 a Lisboa (Portugal)., We present extensions to a continuousstate dependency parsing method that makes it applicable to morphologically rich languages. Starting with a highperformance transition-based parser that uses long short-term memory (LSTM) recurrent neural networks to learn representations of the parser state, we replace lookup-based word representations with representations constructed from the orthographic representations of the words, also using LSTMs. This allows statistical sharing across word forms that are similar on the surface. Experiments for morphologically rich languages show that the parsing model benefits from incorporating the character-based encodings of words., MB was supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA). This research was supported by the U.S. Army Research Laboratory and the U.S. Army Research Office under contract/grant number W911NF-10-1-0533 and NSF IIS-1054319. This work was completed while NAS was at CMU.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A multimodal annotation schema for non-verbal affective analysis in the health-care domain

  • Sukno, Federico Mateo
  • Domínguez Bajo, Mónica
  • Ruiz, Adrià
  • Schiller, Dominik
  • Lingenfelser, Florian
  • Pragst, Louisa
  • Kamateri, Eleni
  • Vrochidis, Stefanos
The development of conversational agents with human interaction capabilities requires advanced affective state recognition integrating non-verbal cues from the different modalities constituting what in human communication we perceive as an overall affective state. Each of the modalities is often handled by a different subsystem that conveys only a partial interpretation of the whole and, as such, is evaluated only in/nterms of its partial view. To tackle this shortcoming, we investigate the generation of a unified multimodal annotation schema of non-verbal cues from the perspective of an interdisciplinary group of experts. We aim at obtaining a common ground-truth with a unique representation using the Valence and Arousal space and a discrete non-linear scale of values. The proposed annotation schema is demonstrated on/na corpus in the health-care domain but is scalable to other purposes. Preliminary results on inter-rater variability show a positive correlation of consensus level with high (absolute) values of Valence and Arousal as well as with the number of annotators labeling a given video sequence., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502) and is part of the Kristina project funded by the European UnionˇSs Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A Thematicity-based prosody enrichment tool for CTS

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: The 18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), celebrada a Estocolm, Suència, del 20 al 24 d'agost de 2017., This paper presents a demonstration of a stochastic prosody tool for enrichment of synthesized speech using SSML prosody tags applied over hierarchical thematicity spans in the context of a CTS application. The motivation for using hierarchical thematicity is exemplified, together with the capabilities of the module to generate a variety of SSML prosody tags within a controlled range of values depending on the input thematicity label., This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number H2020-RIA-645012. It has been also partly supported by the Spanish Ministry of Economy and Competitiveness under the María de Maeztu Units of Excellence Programme (MDM-2015- 0502). The second author is partially funded by the Ramón y Cajal program.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Transition-based dependency parsing with heuristic backtracking

  • Buckman, Jacob
  • Ballesteros, Miguel
  • Dyer, Chris
Comunicació presentada a Conference on Empirical Methods in Natural Language Processing, We introduce a novel approach to the decoding problem in transition-based parsing: heuristic backtracking. This algorithm uses a series of partial parses on the sentence to locate the best candidate parse, using confidence estimates/nof transition decisions as a heuristic to guide the starting points of the search. This allows us to achieve a parse accuracy comparable to beam search, despite using fewer transitions. When used to augment a Stack-LSTM transition-based parser, the parser shows an unlabeled attachment score of up to 93.30% for English and 87.61% for Chinese., Miguel Ballesteros was supported by the European Commission under the contract numbers FP7-/nICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Exploring the applicability of elaborateness and indirectness in dialogue management

  • Pragst, Louisa
  • Minker, Wolfgang
  • Ultes, Stefan
Comunicació presentada a: 8th International Workshop On Spoken Dialogue Systems (IWSDS), celebrat del 6 al 9 de juny a Farmington, Estats Units., In this paper, we investigate the applicability of soft changes to the system behaviour, namely changing the amount of elaborateness and indirectness displayed. To this end, we examine the impact of elaborateness and indirectness on the perception of human-computer communication in a user study. Here, we show that elaborateness and indirectness influence the user’s impression of a dialogue and discuss the implications of our results for adaptive dialogue management. We conclude that elaborateness and indirectness offer valuable possibilities for adaptation and should be incorporated in adaptive dialogue management., This paper is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645012.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A hybrid framework for news clustering based on the DBSCAN-Martingale and LDA

  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
  • Wanner, Leo
Nowadays there is an important need by journalists and media monitoring companies to cluster news in large amounts of web articles, in order to ensure fast access to their topics or events of interest. Our aim in this work is to identify groups of news articles that share a common topic or event, without a priori knowledge of the number of clusters. The estimation of the correct number of topics is a challenging issue, due to the existence of “noise”, i.e. news articles which are irrelevant to all other topics. In this context, we introduce a novel density-based news clustering framework, in which the assignment of news articles to topics is done by the well-established Latent Dirichlet Allocation, but the estimation of the number of clusters is performed by our novel method, called “DBSCAN-Martingale”, which allows for extracting noise from the dataset and progressively extracts clusters from an OPTICS reachability plot. We evaluate our framework and the DBSCAN-Martingale on the 20newsgroups-mini dataset and on 220 web news articles, which are references to specific Wikipedia pages. Among twenty methods for news clustering, without knowing the number of clusters k, the framework of DBSCAN-Martingale provides the correct number of clusters and the highest Normalized Mutual Information., This work was supported by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012), funded by the European Commission.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Thematicity-based prosody enrichment for text-to-speech applications

  • Domínguez Bajo, Mónica
  • Burga Díaz, Alicia
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: the 9th International Conference on Speech Prosody 2018, celebrat del 13 al 16 de juny a Poznań, Polònia., Theoretical studies on the information structure–prosody interface argue that the content packaged in terms of theme and rheme correlates with the intonation of the corresponding sentence as regards to rising and falling patterns (L*+H LH% and H* LL% respectively). When such a correspondence is used to derive prosody in text-to-speech applications, it is often the case that ToBI labels are statically mapped to acoustic parameters. Such an approach is insufficient to solve the problem of monotonous synthetic voices for two reasons: it is repetitive with respect to prosody enrichment, and a binary flat themerheme representation does not serve to describe properly long complex sentences. In this paper, we introduce a methodology for a more versatile thematicity-based prosody enrichment based on: (i) a hierarchical tripartite thematicity model as proposed in the Meaning–Text Theory, and (ii) a corpus-based approach for the automatic extraction of acoustic parameters (fundamental frequency, breaks and speech rate) that are mapped to a varied range of prosody control tags of the synthesized speech. Such a prosody enrichment has shown to provide higher results in a perception test when implemented in a TTS system., This work is part of the KRISTINA project, which has received funding from the European Unions Horizon 2020 Research and Innovation Programme under the Grant Agreement number H2020-RIA-645012. It has been also partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM- 2015-0502). The second author is partially funded by the Spanish Ministry of Economy and Competitivity through the Ram´on y Cajal program.
Dipòsit Digital de la UB
Publication Artículo científico (JournalArticle)

The Information Structure-prosody interface in text-to-speech technologies. An empirical perspective

  • Domínguez, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet, these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogs. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel'čuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-prosody interface.
Dipòsit Digital de la UB
Publication Artículo científico (JournalArticle)

Integrating lexical and prosodic features for automatic paragraph segmentation

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Spoken documents, such as podcasts or lectures, are a growing presence in everyday life. Being able to automatically identify their discourse structure is an important step to understanding what a spoken document is about. Moreover, finer-grained units, such as paragraphs, are highly desirable for presenting and analyzing spoken content. However, little work has been done on discourse based speech segmentation below the level of broad topics. In order to examine how discourse transitions are cued in speech, we investigate automatic paragraph segmentation of TED talks using lexical and prosodic features. Experiments using Support Vector Machines, AdaBoost, and Neural Networks show that models using supra-sentential prosodic features and induced cue words perform better than those based on the type of lexical cohesion measures often used in broad topic segmentation. Moreover, combining a wide range of individually weak lexical and prosodic predictors improves performance, and modelling contextual information using recurrent neural networks outperforms other approaches by a large margin. Our best results come from using late fusion methods that integrate representations generated by separate lexical and prosodic models while allowing interactions between these features streams rather than treating them as independent information sources. Application to ASR outputs shows that adding prosodic features, particularly using late fusion, can significantly ameliorate decreases in performance due to transcription errors.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Transition-based dependency parsing with heuristic backtracking

  • Buckman, Jacob
  • Ballesteros, Miguel
  • Dyer, Chris
Comunicació presentada a Conference on Empirical Methods in Natural Language Processing, We introduce a novel approach to the decoding problem in transition-based parsing: heuristic backtracking. This algorithm uses a series of partial parses on the sentence to locate the best candidate parse, using confidence estimates/nof transition decisions as a heuristic to guide the starting points of the search. This allows us to achieve a parse accuracy comparable to beam search, despite using fewer transitions. When used to augment a Stack-LSTM transition-based parser, the parser shows an unlabeled attachment score of up to 93.30% for English and 87.61% for Chinese., Miguel Ballesteros was supported by the European Commission under the contract numbers FP7-/nICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A pipeline for the creation of progressively rendered web 3D scenes

  • Evans, Alun Thomas
  • Agenjo, Javier
  • Blat, Josep
We present an end-to-end pipeline for the export of 3D scenes from content creaton tools to a real-tme rendering engine in an embeddable web-page, including a novel system for compression/decompression of textured polygonal meshes. We show that the compression/decompression outperforms the best state-of-the-art non-progressive alternatve, especially as bandwidth increases, showing that web-specifc techniques should consider the whole user pipeline. Our pipeline also includes progressivity, which is paramount for a good interactve user experience, and permits full user interacton with lower resoluton versions of the 3D scenes, while progressively higher resoluton data is downloaded. Finally, we discuss how our method may be used in the future to facilitate the transfer of animated meshes.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Towards expressive prosody generation in TTS for reading aloud applications

  • Domínguez Bajo, Mónica
  • Burga Díaz, Alicia
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: IberSpeech 2018, celebrat a Barcelona del 21 al 23 de novembre de 2018., Conversational technologies that assist elderly people need to adapt to common disabilities in old age. Visual, hearing and even more so cognitive impairments pose serious difficulties for our seniors to handle a standard conversation with a human. Understanding a virtual agent may be ever harder. In this case, communicative strategies are key to adapt the virtual agent to the needs of elderly users. This paper addresses the role of the communicative structure for expressive speech prosody, which is known to be crucial for better speech comprehension. It reports on efforts to improve prosody within a text-to-speech system based on one aspect of the communicative structure, namely thematicity. The work has been implemented as an application in a social virtual agent, KRISTINA, which reads aloud news articles upon request for elderly users in German.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Crater monitoring through social media observations

  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Pòster presentat a: European Planetary Science Congress 2017, celebrat a Riga, Letònia, del 17 al 22 de setembre de 2017., Lunar craters have attracted the attention of not only scientists but also citizens. Modern high-resolution cameras with zoom capabilities allow citizens to capture and share pictures of the Moon in Social Media platforms, such as Twitter. We have collected 69 pictures of the Moon, from 01-01-2017 to 17-04-2017, that have been uploaded on Twitter and have been associated with the keyword #crater. The lunar pictures are indexed using SIFT descriptors and are then clustered using density-based approaches to group them into the automatically detected levels of zoom., This work has supported by the EC-funded project KRISTINA (H2020-645012).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A Hybrid approach for biomedical relation extraction using finite state automata and random forest-weighted fusion

  • Mavropoulos, Thanassis
  • Liparas, Dimitris
  • Symeonidis, Spyridon
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: The 18th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2017), celebrada a Budapest, Hungria, del 17 al 23 d'abril de 2017., The automatic extraction of relations between medical entities found in related texts is considered to be a very important task, due to the multitude of applications that it can support, from question answering systems to the devel-opment of medical ontologies. Many different methodologies have been pre-sented and applied to this task over the years. Of particular interest are hybrid approaches, in which different techniques are combined in order to improve the individual performance of either one of them. In this study, we extend a previ-ously established hybrid framework for medical relation extraction, which we modify by enhancing the pattern-based part of the framework and by applying a more sophisticated weighting method. Most notably, we replace the use of regu-lar expressions with finite state automata for the pattern-building part, while the fusion part is replaced by a weighting strategy that is based on the operational capabilities of the Random Forests algorithm. The experimental results indicate the superiority of the proposed approach against the aforementioned well-established hybrid methodology and other state-of-the-art approaches., This work was supported by the project KRISTINA (H2020-645012), funded by the European Commission. Deidentified clinical records used in this research were provided by the i2b2 National Center for Biomedical Computing funded by U54LM008748 and were originally prepared for the Shared Tasks for Challenges in NLP for Clinical Data organized by Dr. Ozlem Uzuner, i2b2 and SUNY.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Transition-based spinal parsing

  • Ballesteros, Miguel
  • Carreras, Xavier
Comunicació presentada a la 19th Conference on Computational Language Learning, celebrada els dies 30 i 31 de juliol 2015 a Beijing (China) i organitzada per l'ACL Special Interest Group on Natural Language Learning., We present a transition-based arc-eager model to parse spinal trees, a dependencybased representation that includes phrasestructure information in the form of constituent spines assigned to tokens. As a main advantage, the arc-eager model can use a rich set of features combining dependency and constituent information, while parsing in linear time. We describe a set of conditions for the arc-eager system to produce valid spinal structures. In experiments using beam search we show that the model obtains a good trade-off between speed and accuracy, and yields state of the art performance for both dependency and constituent parsing measures., Miguel Ballesteros is supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).
Recercat. Dipósit de la Recerca de Catalunya
Publication Capítulo o parte de un libro (BookPart)

Web-based embodied conversational agents and older people

  • Llorach, Gerard
  • Agenjo, Javier
  • Blat, Josep
  • Sayago Barrantes, Sergio
Within Human-Computer Interaction, there has recently been an important turn to embodied and voice-based interaction. In this chapter, we discuss our ongoing research on building online Embodied Conversational Agents (ECAs), specifically, their interactive 3D web graphics aspects. We present ECAs based on our technological pipeline, which integrates a number of free online editors, such as Adobe Fuse CC or MakeHuman, and standards, mainly BML (Behaviour Markup Language). We claim that making embodiment available for online ECAs is attainable, and advantageous over current alternatives, mostly desktop-based. In this chapter we also report on initial results of activities aimed to explore the physical appearance of ECAs for older people. A group of them (N = 14) designed female ECAs. Designing them was easy and great fun. The perspective on older-adult HCI introduced in this chapter is mostly technological, allowing for rapid online experimentations to address key issues, such as anthropomorphic aspects, in the design of ECAs with, and for, older people., This work was funded by the EU’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 675324 (ENRICH) and under the contract number H2020-645012-RIA (KRISTINA).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Using prosody to classify discourse relations

  • Kleinhans, Janine
  • Farrús, Mireia
  • Gravano, Agustín
  • Pérez, Juan Manuel
  • Lai, Catherine
  • Wanner, Leo
Comunicació presentada a: The 18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), celebrada a Estocolm, Suència, del 20 al 24 d'agost de 2017., This work aims to explore the correlation between the discourse structure of a spoken monologue and its prosody by predicting discourse relations from different prosodic attributes. For this purpose, a corpus of semi-spontaneous monologues in English has been automatically annotated according to the Rhetorical Structure Theory, which models coherence in text via rhetorical relations. From corresponding audio files, prosodic features such as pitch, intensity, and speech rate have been extracted from different contexts of a relation. Supervised classification tasks using Support Vector Machines have been performed to find relationships between prosodic features and rhetorical relations. Preliminary results show that intensity combined with other features extracted from intra- and intersegmental environments is the feature with the highest predictability for a discourse relation. The prediction of rhetorical relations from prosodic features and their combinations is straightforwardly applicable to several tasks such as speech understanding or generation. Moreover, the knowledge of how rhetorical relations should be marked in terms of prosody will serve as a basis to improve speech synthesis applications and make voices sound more natural and expressive., This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number 645012. The second author is partially funded by the Spanish Ministry of Economy, Industry and Competitiveness through the Ramón y Cajal program. The third and fourth authors are partially funded by ANPCYT PICT 2014-1561, and the Air Force Office of Scientific Research, Air Force Material Command, USAF under Award No. FA9550-15-1-0055.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Unsupervised keyword extraction using the GoW model and centrality scores

  • Batziou, Elissavet
  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Comunicació presentada a: The 4th International Conference, INSCI 2017, celebrat a Thessaloniki, Grècia, del 22 al 24 de novembre de 2017., Nowadays, a large amount of text documents are produced on a daily basis, so we need e cient and e ective access to their con- tent. News articles, blogs and technical reports are often lengthy, so the reader needs a quick overview of the underlying content. To that end we present graph-based models for keyword extraction, in order to compare the Bag of Words model with the Graph of Words model in the key- word extraction problem. We compare their performance in two publicly available datasets using the evaluation measures Precision@10, mean Av- erage Precision and Jaccard coe cient. The methods we have selected for comparison are grouped into two main categories. On the one hand, centrality measures on the formulated Graph-of-Words (GoW) are able to rank all words in a document from the most central to the less central, according to their score in the GoW representation. On the other hand, community detection algorithms on the GoW provide the largest commu- nity that contains the key nodes (words) in the GoW. We selected these methods as the most prominent methods to identify central nodes in a GoW model. We conclude that term-frequency scores (BoW model) are useful only in the case of less structured text, while in more structured text documents, the order of words plays a key role and graph-based models are superior to the term-frequency scores per document., This work was supported by the projects H2020-645012 (KRISTINA) and H2020-700024 (TENSOR), funded by the European Commission.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Automatic extraction of parallel speech corpora from dubbed movies

  • Öktem, Alp
  • Farrús, Mireia
  • Wanner, Leo
Comunicació presentada a: 10th Workshop on Building and Using Comparable Corpora (BUCC), celebrat a Vancouver, Canadà, del 30 de juliol al 4 d'agost de 2017., This paper presents a methodology to extract parallel speech corpora based on any language pair from dubbed movies, together with an application framework in which some corresponding prosodic parameters are extracted. The obtained parallel corpora are especially suitable for speech-to-speech translation applications when a prosody transfer between source and target languages is desired., This work is part of the KRISTINA project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement number 645012. The second author is partially funded by the Spanish Ministry of Economy, Industry and Competitiveness through the Ram´on y Cajal program.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Semantics-driven recognition of collocations using word embeddings

  • Rodríguez Fernández, Sara
  • Espinosa-Anke, Luis
  • Carlini, Roberto
  • Wanner, Leo
L2 learners often produce “ungrammatical” word combinations such as, e.g., *give a suggestion or *make a walk. This is because of the “collocationality” of one of their items (the base) that limits the acceptance of collocates to express a specific meaning (‘perform’ above). We propose an algorithm that delivers, for a given base and the intended meaning of a collocate, the actual collocate lexeme(s) (make / take above). The algorithm exploits the/nlinear mapping between bases and collocates from examples and generates a collocation transformation matrix which is then applied to novel unseen cases. The evaluation shows a promising line of research in collocation discovery., The present work has been partially funded by the Spanish Ministry of Economy and Competitiveness/n(MINECO), through a predoctoral grant (BES-2012-057036) in the framework of the project HARenES (FFI2011-30219-C02-02), and by the European Commission under the grant number H2020–645012–RIA. We also acknowledge support from the Maria de Maeztu Excellence Program (MDM-2015-0502).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Visualizing deep-syntactic parser output

  • Soler Company, Juan
  • Ballesteros, Miguel
  • Bohnet, Bernd
  • Mille, Simon
  • Wanner, Leo
Comunicació presentada a la 2015 Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT 2015), celebrada del 31 de maig al 5 de juny 2015 a Denver (CO, EUA)., “Deep-syntactic” dependency structures bridge the gap between the surface-syntactic structures as produced by state-of-the-art dependency parsers and semantic logical forms in that they abstract away from surfacesyntactic idiosyncrasies, but still keep the linguistic structure of a sentence. They have thus a great potential for such downstream applications as machine translation and summarization. In this demo paper, we propose an online version of a deep-syntactic parser that outputs deep-syntactic structures from plain sentences and visualizes them using the Brat tool. Along with the deep syntactic structures, the user can also inspect the visual presentation of the surface-syntactic structures that serve as input to the deep-syntactic parser and that are produced by the joint tagger and syntactic transition-based parser ran in the pipeline before deep-syntactic parsing takes place., This work has been partially funded by the European Union’s Seventh Framework and Horizon 2020 Research and Innovation Programmes under the Grant Agreement numbers FP7-ICT-610411, FP7-SME- 606163, and H2020-RIA-645012.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Topic detection using the DBSCAN-Martingale and the time operator

  • Gialampoukidis, Ilias
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
  • Antoniou, Ioannis
Comunicació presentada a: The 17th Conference of the Applied Stochastic Models and Data Analysis (ASMDA), celebrada del 6 al 9 de juny de 2017 a Londres, Regne Unit., Topic detection is usually considered as a decision process implemented in some relevant context, for example clustering. In this case, clusters correspond to topics that should be identifed. Density-based clustering, for example, uses only a density level E and a lower bound for the number of points in a cluster. As the density level is hard to be estimated, a stochastic process, called the DBSCANMartingale, is constructed for the combination of several outputs of DBSCAN for various randomly selected values of E in a predefned closed interval [0; Emax] from the uniform distribution. We have observed that most of the clusters are extracted in the interval [0; Emax=2], and moreover in the interval [Emax=2; Emax] the DBSCANMartingale stochastic process is less innovative, i.e. extracts only a few or no clusters. Therefore, non-symmetric skewed distributions are needed to generate density levels for the extraction of all clusters in a fast way. In this work we show that skewed distributions may be used instead of the uniform, so as to extract all clusters as quickly as possible. Experiments on real datasets show that the average innovation time of the DBSCAN-Martingale stochastic process is reduced when skewed distributions are employed, so less time is needed to extract all clusters., The first author would like to thank the Research Committee of the Aristo- tle University of Thessaloniki for awarding him the \Aristeia" postdoctoral scholarship 2016. Moreover, this work has been partially supported by the EC-funded project KRISTINA (H2020-645012).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Integrating lexical and prosodic features for automatic paragraph segmentation

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Spoken documents, such as podcasts or lectures, are a growing presence in everyday life. Being able to automatically identify their discourse structure is an important step to understanding what a spoken document is about. Moreover, finer-grained units, such as paragraphs, are highly desirable for presenting and analyzing spoken content. However, little work has been done on discourse based speech segmentation below the level of broad topics. In order to examine how discourse transitions are cued in speech, we investigate automatic paragraph segmentation of TED talks using lexical and prosodic features. Experiments using Support Vector Machines, AdaBoost, and Neural Networks show that models using supra-sentential prosodic features and induced cue words perform better than those based on the type of lexical cohesion measures often used in broad topic segmentation. Moreover, combining a wide range of individually weak lexical and prosodic predictors improves performance, and modelling contextual information using recurrent neural networks outperforms other approaches by a large margin. Our best results come from using late fusion methods that integrate representations generated by separate lexical and prosodic models while allowing interactions between these features streams rather than treating them as independent information sources. Application to ASR outputs shows that adding prosodic features, particularly using late fusion, can significantly ameliorate decreases in performance due to transcription errors., The second author was funded from the EU’s Horizon 2020 Research and Innovation Programme under the GA H2020-RIA-645012 and the Spanish Ministry of Economy and Competitivity Juan de la Cierva program. The other authors were funded by the University of Edinburgh.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo preliminar (WorkingPaper)

FORGe at WebNLG 2017

  • Mille, Simon
  • Dasiopoulou, Stamatia
This paper describes the FORGe generator at E2E. The input triples are mapped onto sentences by applying a series of rule-based graph-transducers and aggregation grammars to template predicate-argument structures associated to each property. We submitted two primary systems to the task, one based on the grammars, and one based on templates, and one secondary system, which is a variation of the grammar-based one.