Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 24
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

  • Gialampoukidis, Ilias
  • Moumtzidou, Anastasia
  • Liparas, Dimitris
  • Tsikrika, Theodora
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Heterogeneous sources of information, such as images, videos, text and metadata are often used to describe di erent or complementary views of the same multimedia object, especially in the online news domain and in large annotated image collections. The retrieval of multimedia objects, given a mul- timodal query, requires the combination of several sources of information in an e cient and scalable way. Towards this direction, we provide a novel unsuper- vised framework for multimodal fusion of visual and textual similarities, which are based on visual features, visual concepts and textual metadata, integrating non-linear graph-based fusion and Partial Least Squares Regression. The fu- sion strategy is based on the construction of a multimodal contextual similarity matrix and the non-linear combination of relevance scores from query-based similarity vectors. Our framework can employ more than two modalities and high-level information, without increase in memory complexity, when com- pared to state-of-the-art baseline methods. The experimental comparison is done in three public multimedia collections in the multimedia retrieval task. The results have shown that the proposed method outperforms the baseline methods, in terms of Mean Average Precision and Precision@20., This work was partially supported by the European Commission by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Design of knowledge-based agent as a social companion

  • Wanner, Leo
  • André, Elisabeth
  • Blat, Josep
  • Dasiopoulou, Stamatia
  • Farrús, Mireia
  • Fraga, Thiago
  • Kamateri, Eleni
  • Lingenfelser, Florian
  • Llorach, Gerard
  • Martínez, Oriol
  • Meditskos, Georgios
  • Mille, Simon
  • Minker, Wolfgang
  • Pragst, Louisa
  • Schiller, Dominik
  • Stam, Andries
  • Stellingwerff, Ludo
  • Sukno, Federico Mateo
  • Vieru, Bianca
  • Vrochidis, Stefanos
Article de la comunicació presentada a International Conference on Health and Social Care Information Systems and Technologies (HCist 2017), celebrat a Barcelona el novembre de 2017., We present work in progress on an intelligent embodied conversation agent that is supposed to act as a social companion with linguistic and emotional competence in the context of basic and health care. The core of the agent is an ontology-based knowledge model that supports flexible reasoning-driven conversation planning strategies. A dedicated search engine ensures the provision of background information from the web, necessary for conducting a conversation on a specific topic. Multimodal communication analysis and generation modules analyze respectively generate facial expressions, gestures and multilingual speech. The assessment of the prototypical implementation of the agent shows that users accept it as a natural and trustworthy conversation counterpart. For the final release, all involved technologies will be further improved and matured., The presented work is funded by the European Commission as part of the H2020 Program under the contract number 645012. We would like to thank our colleagues Chiara Baudracco, Jutta Mohr, Eylem Ög, Valérie Sarholz, and Benjamin Schäfer for organizing and carrying out the evaluation trials and for their patience with the numerous inadequacies of KRISTINA.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

The Information structure–prosody interface in text-to-speech technologies: an empirical perspective

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogues. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel’ˇcuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-–prosody interface, This work has been partially funded by the European Commission in the context of its H2020 Programme under the contract numbers H2020-645012-RIA (KRISTINA) and H2020-870930-IA (WELCOME). The second author has been funded by the Agencia Estatal de Investigaci´on (AEI), Ministerio de Ciencia, Innovaci´on y Universidades and the Fondo Social Europeo (FSE), grant RYC2015-17239 (AEI/FSE, UE).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

KRISTINA, un asistente virtual para superar las barreras lingüísticas en la atención sanitaria

  • Dafa, Lehdia Mohamed
  • Baudracco, Chiara
  • Moragas, Ana
  • Llor Vila, Carles
  • Molero Trujillo, David
  • Vilanova Estrada, Pere
Las tecnologías de la información y comunicación (TIC) están posibilitando grandes beneficios en el campo de la salud. En este sentido, KRISTINA (Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities) es un proyecto que forma parte del programa marco de investigación e innovación de la Unión Europea HORIZON 2020, a través del cual el proyecto ha sido financiado por la Comisión Europea con tres millones de euros. Su objetivo es desarrollar mediante tecnología digital un agente multilingüe con funcionalidades de comunicación en entornos sanitarios y de bienestar. El proyecto está orientado a las personas migradas, mayores de edad, de origen turco, así como a cuidadores polacos que trabajan en Alemania, y a la población magrebí residente en España. Actualmente se están definiendo las nuevas funcionalidades del segundo prototipo de avatar y se está evaluando la interacción humana entre el mismo y los usuarios., Information and communication technologies are bringing about major breakthroughs and benefits in the health field. The KRISTINA (Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities) project is part of the EU Framework Programme for Research and Innovation HORIZON 2020 and has received funding of three million euros from the European Commission. The main objective is to develop, through digital technologies, a multilingual agent with communicative functions to help immigrants overcome language barriers when seeking medical information on the network and contacting the health system. The targeted users are elderly people coming from Turkey to Germany, Polish caregivers working in Germany and Maghrebi migrants in Spain. The main functionalities of the first avatar prototype and its human interaction are currently being evaluated and tested by the main partners of the project consortium.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

A pipeline for the creation of progressively rendered web 3D scenes

  • Evans, Alun Thomas
  • Agenjo, Javier
  • Blat, Josep
We present an end-to-end pipeline for the export of 3D scenes from content creaton tools to a real-tme rendering engine in an embeddable web-page, including a novel system for compression/decompression of textured polygonal meshes. We show that the compression/decompression outperforms the best state-of-the-art non-progressive alternatve, especially as bandwidth increases, showing that web-specifc techniques should consider the whole user pipeline. Our pipeline also includes progressivity, which is paramount for a good interactve user experience, and permits full user interacton with lower resoluton versions of the 3D scenes, while progressively higher resoluton data is downloaded. Finally, we discuss how our method may be used in the future to facilitate the transfer of animated meshes.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Integrating lexical and prosodic features for automatic paragraph segmentation

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Spoken documents, such as podcasts or lectures, are a growing presence in everyday life. Being able to automatically identify their discourse structure is an important step to understanding what a spoken document is about. Moreover, finer-grained units, such as paragraphs, are highly desirable for presenting and analyzing spoken content. However, little work has been done on discourse based speech segmentation below the level of broad topics. In order to examine how discourse transitions are cued in speech, we investigate automatic paragraph segmentation of TED talks using lexical and prosodic features. Experiments using Support Vector Machines, AdaBoost, and Neural Networks show that models using supra-sentential prosodic features and induced cue words perform better than those based on the type of lexical cohesion measures often used in broad topic segmentation. Moreover, combining a wide range of individually weak lexical and prosodic predictors improves performance, and modelling contextual information using recurrent neural networks outperforms other approaches by a large margin. Our best results come from using late fusion methods that integrate representations generated by separate lexical and prosodic models while allowing interactions between these features streams rather than treating them as independent information sources. Application to ASR outputs shows that adding prosodic features, particularly using late fusion, can significantly ameliorate decreases in performance due to transcription errors., The second author was funded from the EU’s Horizon 2020 Research and Innovation Programme under the GA H2020-RIA-645012 and the Spanish Ministry of Economy and Competitivity Juan de la Cierva program. The other authors were funded by the University of Edinburgh.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Survey on automatic lip-reading in the era of deep learning

  • Fernandez-Lopez, Adriana
  • Sukno, Federico Mateo
In the last few years, there has been an increasing interest in developing systems for Automatic Lip-Reading (ALR). Similarly to other computer vision applications, methods based on Deep Learning (DL) have become very popular and have permitted to substantially push forward the achievable performance. In this survey, we review ALR research during the last decade, highlighting the progression from approaches previous to DL (which we refer to as traditional) toward end-to-end DL architectures. We provide a comprehensive list of the audio-visual databases available for lip-reading, describing what tasks they can be used for, their popularity and their most important characteristics, such as the number of speakers, vocabulary size, recording settings and total duration. In correspondence with the shift toward DL, we show that there is a clear tendency toward large-scale datasets targeting realistic application settings and large numbers of samples per class. On the other hand, we summarize, discuss and compare the different ALR systems proposed in the last decade, separately considering traditional and DL approaches. We address a quantitative analysis of the different systems by organizing them in terms of the task that they target (e.g. recognition of letters or digits and words or sentences) and comparing their reported performance in the most commonly used datasets. As a result, we find that DL architectures perform similarly to traditional ones for simpler tasks but report significant improvements in more complex tasks, such as word or sentence recognition, with up to 40% improvement in word recognition rates. Hence, we provide a detailed description of the available ALR systems based on end-to-end DL architectures and identify a tendency to focus on the modeling of temporal context as the key to advance the field. Such modeling is dominated by recurrent neural networks due to their ability to retain context at multiple scales (e.g. short- and long-term information). In this sense, current efforts tend toward techniques that allow a more comprehensive modeling and interpretability of the retained context., This work is partly supported by the Spanish Ministry of Economy and Competitiveness, Spain under project grant TIN2017-90124-P, the Ramon y Cajal Programme, the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), and the Kristina project funded by the European Union Horizon 2020 - Research and Innovation Framework Programme under grant agreement No. 645012.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Data-driven deep-syntactic dependency parsing

  • Ballesteros, Miguel
  • Bohnet, Bernd
  • Mille, Simon
  • Wanner, Leo
‘Deep-syntactic’ dependency structures that capture the argumentative, attributive and co-/nordinative relations between full words of a sentence have a great potential for a number/nof NLP-applications. The abstraction degree of these structures is in between the output/nof a syntactic dependency parser (connected trees defined over all words of a sentence and/nlanguage-specific grammatical functions) and the output of a semantic parser (forests of trees/ndefined over individual lexemes or phrasal chunks and abstract semantic role labels which/ncapture the frame structures of predicative elements and drop all attributive and coordinative/ndependencies). We propose a parser that provides deep-syntactic structures. The parser has/nbeen tested on Spanish, English and Chinese, The work reported on in this paper has been partially funded by the European Commission under the contract numbers FP7-ICT-610411 (MULTISENSOR) and H2020-645012-RIA (KRISTINA).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Gradient-based steering for vision-based crowd simulation algorithms

  • Dutra, T. B.
  • Marques, Ricardo
  • Cavalcante‐Neto, J.B.
  • Vidal, C. A.
  • Pettré, J.
Most recent crowd simulation algorithms equip agents with a synthetic vision component for steering. They offer promising perspectives through a more realistic simulation of the way humans navigate according to their perception of the surrounding environment. In this paper, we propose a new perception/motion loop to steering agents along collision free trajectories that significantly improves the quality of vision-based crowd simulators. In contrast with solutions where agents avoid collisions in a purely reactive (binary) way, we suggest exploring the full range of possible adaptations and retaining the locally optimal one. To this end, we introduce a cost function, based on perceptual variables, which estimates an agent’s situation considering both the risks of future collision and a desired destination. We then compute the partial derivatives of that function with respect to all possible motion adaptations. The agent then adapts its motion by following the gradient. This paper has thus two main contributions: the definition of a general purpose control scheme for steering synthetic vision-based agents; and the proposition of cost functions for evaluating the perceived danger of the current situation. We demonstrate improvements in several cases., T. B. Dutra acknowl1edges CAPES for the fellowship (PDSE Proc. 0130/13-3) and J. Ondˇrej for helping in the implementation of his model. R. Marques acknowledges the projects Percolation (ANR- 13-JS02-0008), and Kristina (H2020-RIA-645012) for partially financing this research.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Multi-instance dynamic ordinal random fields for weakly supervised dacial behavior analysis

  • Ruiz, Adrià
  • Rudovic, Ognjen
  • Binefa i Valls, Xavier
  • Pantic, Maja
We propose a multi-instance-learning (MIL) approach for weakly supervised learning problems, where a training set is formed by bags (sets of feature vectors or instances) and only labels at bag-level are provided. Specifically, we consider the multi-instance dynamic-ordinal-regression (MI-DOR) setting, where the instance labels are naturally represented as ordinal variables and bags are structured as temporal sequences. To this end, we propose MI dynamic ordinal random fields (MI-DORF). In this paper, we treat instance-labels as temporally dependent latent variables in an undirected graphical model. Different MIL assumptions are modelled via newly introduced high-order potentials relating bag and instance-labels within the energy function of the model. We also extend our framework to address the partially observed MI-DOR problem, where a subset of instance labels is also available during training. We show on the tasks of weakly supervised facial action unit and pain intensity estimation, that the proposed framework outperforms alternative learning approaches. Furthermore, we show that MI-DORF can be employed to reduce the data annotation efforts in this context by large-scale., This work was supported by the European Union’s Horizon 2020 Research and Innovation Programme under Grant 645012. The work of O. Rudovic was supported by the H2020 Research Program through the Marie Skodowska-Curie Grant Agreement under Grant 701236 (EngageME). The work of X. Binefa was supported in part by Spanish Government under Grant MINECO TIN2017-90124-P and in part by the Generalitat de Catalunya under Grant MINECO TIN2017-90124-P and Grant AGAUR 2017-SGR-1311
Dipòsit Digital de la UB
Publicación Artículo científico (JournalArticle)

The Information Structure-prosody interface in text-to-speech technologies. An empirical perspective

  • Domínguez, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet, these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogs. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel'čuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-prosody interface.
Dipòsit Digital de la UB
Publicación Artículo científico (JournalArticle)

Integrating lexical and prosodic features for automatic paragraph segmentation

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Spoken documents, such as podcasts or lectures, are a growing presence in everyday life. Being able to automatically identify their discourse structure is an important step to understanding what a spoken document is about. Moreover, finer-grained units, such as paragraphs, are highly desirable for presenting and analyzing spoken content. However, little work has been done on discourse based speech segmentation below the level of broad topics. In order to examine how discourse transitions are cued in speech, we investigate automatic paragraph segmentation of TED talks using lexical and prosodic features. Experiments using Support Vector Machines, AdaBoost, and Neural Networks show that models using supra-sentential prosodic features and induced cue words perform better than those based on the type of lexical cohesion measures often used in broad topic segmentation. Moreover, combining a wide range of individually weak lexical and prosodic predictors improves performance, and modelling contextual information using recurrent neural networks outperforms other approaches by a large margin. Our best results come from using late fusion methods that integrate representations generated by separate lexical and prosodic models while allowing interactions between these features streams rather than treating them as independent information sources. Application to ASR outputs shows that adding prosodic features, particularly using late fusion, can significantly ameliorate decreases in performance due to transcription errors.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A pipeline for the creation of progressively rendered web 3D scenes

  • Evans, Alun Thomas
  • Agenjo, Javier
  • Blat, Josep
We present an end-to-end pipeline for the export of 3D scenes from content creaton tools to a real-tme rendering engine in an embeddable web-page, including a novel system for compression/decompression of textured polygonal meshes. We show that the compression/decompression outperforms the best state-of-the-art non-progressive alternatve, especially as bandwidth increases, showing that web-specifc techniques should consider the whole user pipeline. Our pipeline also includes progressivity, which is paramount for a good interactve user experience, and permits full user interacton with lower resoluton versions of the 3D scenes, while progressively higher resoluton data is downloaded. Finally, we discuss how our method may be used in the future to facilitate the transfer of animated meshes.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Integrating lexical and prosodic features for automatic paragraph segmentation

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Spoken documents, such as podcasts or lectures, are a growing presence in everyday life. Being able to automatically identify their discourse structure is an important step to understanding what a spoken document is about. Moreover, finer-grained units, such as paragraphs, are highly desirable for presenting and analyzing spoken content. However, little work has been done on discourse based speech segmentation below the level of broad topics. In order to examine how discourse transitions are cued in speech, we investigate automatic paragraph segmentation of TED talks using lexical and prosodic features. Experiments using Support Vector Machines, AdaBoost, and Neural Networks show that models using supra-sentential prosodic features and induced cue words perform better than those based on the type of lexical cohesion measures often used in broad topic segmentation. Moreover, combining a wide range of individually weak lexical and prosodic predictors improves performance, and modelling contextual information using recurrent neural networks outperforms other approaches by a large margin. Our best results come from using late fusion methods that integrate representations generated by separate lexical and prosodic models while allowing interactions between these features streams rather than treating them as independent information sources. Application to ASR outputs shows that adding prosodic features, particularly using late fusion, can significantly ameliorate decreases in performance due to transcription errors., The second author was funded from the EU’s Horizon 2020 Research and Innovation Programme under the GA H2020-RIA-645012 and the Spanish Ministry of Economy and Competitivity Juan de la Cierva program. The other authors were funded by the University of Edinburgh.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multi-instance dynamic ordinal random fields for weakly supervised dacial behavior analysis

  • Ruiz, Adrià
  • Rudovic, Ognjen
  • Binefa i Valls, Xavier
  • Pantic, Maja
We propose a multi-instance-learning (MIL) approach for weakly supervised learning problems, where a training set is formed by bags (sets of feature vectors or instances) and only labels at bag-level are provided. Specifically, we consider the multi-instance dynamic-ordinal-regression (MI-DOR) setting, where the instance labels are naturally represented as ordinal variables and bags are structured as temporal sequences. To this end, we propose MI dynamic ordinal random fields (MI-DORF). In this paper, we treat instance-labels as temporally dependent latent variables in an undirected graphical model. Different MIL assumptions are modelled via newly introduced high-order potentials relating bag and instance-labels within the energy function of the model. We also extend our framework to address the partially observed MI-DOR problem, where a subset of instance labels is also available during training. We show on the tasks of weakly supervised facial action unit and pain intensity estimation, that the proposed framework outperforms alternative learning approaches. Furthermore, we show that MI-DORF can be employed to reduce the data annotation efforts in this context by large-scale., This work was supported by the European Union’s Horizon 2020 Research and Innovation Programme under Grant 645012. The work of O. Rudovic was supported by the H2020 Research Program through the Marie Skodowska-Curie Grant Agreement under Grant 701236 (EngageME). The work of X. Binefa was supported in part by Spanish Government under Grant MINECO TIN2017-90124-P and in part by the Generalitat de Catalunya under Grant MINECO TIN2017-90124-P and Grant AGAUR 2017-SGR-1311
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Data-driven deep-syntactic dependency parsing

  • Ballesteros, Miguel
  • Bohnet, Bernd
  • Mille, Simon
  • Wanner, Leo
‘Deep-syntactic’ dependency structures that capture the argumentative, attributive and co-/nordinative relations between full words of a sentence have a great potential for a number/nof NLP-applications. The abstraction degree of these structures is in between the output/nof a syntactic dependency parser (connected trees defined over all words of a sentence and/nlanguage-specific grammatical functions) and the output of a semantic parser (forests of trees/ndefined over individual lexemes or phrasal chunks and abstract semantic role labels which/ncapture the frame structures of predicative elements and drop all attributive and coordinative/ndependencies). We propose a parser that provides deep-syntactic structures. The parser has/nbeen tested on Spanish, English and Chinese, The work reported on in this paper has been partially funded by the European Commission under the contract numbers FP7-ICT-610411 (MULTISENSOR) and H2020-645012-RIA (KRISTINA).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The Information Structure-prosody interface in text-to-speech technologies. An empirical perspective

  • Domínguez, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet, these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogs. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel'čuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-prosody interface.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

KRISTINA, un asistente virtual para superar las barreras lingüísticas en la atención sanitaria

  • Dafa, Lehdia Mohamed
  • Baudracco, Chiara
  • Moragas, Ana
  • Llor Vila, Carles
  • Molero Trujillo, David
  • Vilanova Estrada, Pere
Las tecnologías de la información y comunicación (TIC) están posibilitando grandes beneficios en el campo de la salud. En este sentido, KRISTINA (Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities) es un proyecto que forma parte del programa marco de investigación e innovación de la Unión Europea HORIZON 2020, a través del cual el proyecto ha sido financiado por la Comisión Europea con tres millones de euros. Su objetivo es desarrollar mediante tecnología digital un agente multilingüe con funcionalidades de comunicación en entornos sanitarios y de bienestar. El proyecto está orientado a las personas migradas, mayores de edad, de origen turco, así como a cuidadores polacos que trabajan en Alemania, y a la población magrebí residente en España. Actualmente se están definiendo las nuevas funcionalidades del segundo prototipo de avatar y se está evaluando la interacción humana entre el mismo y los usuarios., Information and communication technologies are bringing about major breakthroughs and benefits in the health field. The KRISTINA (Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities) project is part of the EU Framework Programme for Research and Innovation HORIZON 2020 and has received funding of three million euros from the European Commission. The main objective is to develop, through digital technologies, a multilingual agent with communicative functions to help immigrants overcome language barriers when seeking medical information on the network and contacting the health system. The targeted users are elderly people coming from Turkey to Germany, Polish caregivers working in Germany and Maghrebi migrants in Spain. The main functionalities of the first avatar prototype and its human interaction are currently being evaluated and tested by the main partners of the project consortium.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Gradient-based steering for vision-based crowd simulation algorithms

  • Dutra, T. B.
  • Marques, Ricardo
  • Cavalcante‐Neto, J.B.
  • Vidal, C. A.
  • Pettré, J.
Most recent crowd simulation algorithms equip agents with a synthetic vision component for steering. They offer promising perspectives through a more realistic simulation of the way humans navigate according to their perception of the surrounding environment. In this paper, we propose a new perception/motion loop to steering agents along collision free trajectories that significantly improves the quality of vision-based crowd simulators. In contrast with solutions where agents avoid collisions in a purely reactive (binary) way, we suggest exploring the full range of possible adaptations and retaining the locally optimal one. To this end, we introduce a cost function, based on perceptual variables, which estimates an agent’s situation considering both the risks of future collision and a desired destination. We then compute the partial derivatives of that function with respect to all possible motion adaptations. The agent then adapts its motion by following the gradient. This paper has thus two main contributions: the definition of a general purpose control scheme for steering synthetic vision-based agents; and the proposition of cost functions for evaluating the perceived danger of the current situation. We demonstrate improvements in several cases., T. B. Dutra acknowl1edges CAPES for the fellowship (PDSE Proc. 0130/13-3) and J. Ondˇrej for helping in the implementation of his model. R. Marques acknowledges the projects Percolation (ANR- 13-JS02-0008), and Kristina (H2020-RIA-645012) for partially financing this research.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

  • Gialampoukidis, Ilias
  • Moumtzidou, Anastasia
  • Liparas, Dimitris
  • Tsikrika, Theodora
  • Vrochidis, Stefanos
  • Kompatsiaris, Ioannis
Heterogeneous sources of information, such as images, videos, text and metadata are often used to describe di erent or complementary views of the same multimedia object, especially in the online news domain and in large annotated image collections. The retrieval of multimedia objects, given a mul- timodal query, requires the combination of several sources of information in an e cient and scalable way. Towards this direction, we provide a novel unsuper- vised framework for multimodal fusion of visual and textual similarities, which are based on visual features, visual concepts and textual metadata, integrating non-linear graph-based fusion and Partial Least Squares Regression. The fu- sion strategy is based on the construction of a multimodal contextual similarity matrix and the non-linear combination of relevance scores from query-based similarity vectors. Our framework can employ more than two modalities and high-level information, without increase in memory complexity, when com- pared to state-of-the-art baseline methods. The experimental comparison is done in three public multimedia collections in the multimedia retrieval task. The results have shown that the proposed method outperforms the baseline methods, in terms of Mean Average Precision and Precision@20., This work was partially supported by the European Commission by the projects MULTISENSOR (FP7-610411) and KRISTINA (H2020-645012).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Integrating lexical and prosodic features for automatic paragraph segmentation

  • Lai, Catherine
  • Farrús, Mireia
  • Moore, Johanna D.
Spoken documents, such as podcasts or lectures, are a growing presence in everyday life. Being able to automatically identify their discourse structure is an important step to understanding what a spoken document is about. Moreover, finer-grained units, such as paragraphs, are highly desirable for presenting and analyzing spoken content. However, little work has been done on discourse based speech segmentation below the level of broad topics. In order to examine how discourse transitions are cued in speech, we investigate automatic paragraph segmentation of TED talks using lexical and prosodic features. Experiments using Support Vector Machines, AdaBoost, and Neural Networks show that models using supra-sentential prosodic features and induced cue words perform better than those based on the type of lexical cohesion measures often used in broad topic segmentation. Moreover, combining a wide range of individually weak lexical and prosodic predictors improves performance, and modelling contextual information using recurrent neural networks outperforms other approaches by a large margin. Our best results come from using late fusion methods that integrate representations generated by separate lexical and prosodic models while allowing interactions between these features streams rather than treating them as independent information sources. Application to ASR outputs shows that adding prosodic features, particularly using late fusion, can significantly ameliorate decreases in performance due to transcription errors.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Design of knowledge-based agent as a social companion

  • Wanner, Leo
  • André, Elisabeth
  • Blat, Josep
  • Dasiopoulou, Stamatia
  • Farrús, Mireia
  • Fraga, Thiago
  • Kamateri, Eleni
  • Lingenfelser, Florian
  • Llorach, Gerard
  • Martínez, Oriol
  • Meditskos, Georgios
  • Mille, Simon
  • Minker, Wolfgang
  • Pragst, Louisa
  • Schiller, Dominik
  • Stam, Andries
  • Stellingwerff, Ludo
  • Sukno, Federico Mateo
  • Vieru, Bianca
  • Vrochidis, Stefanos
Article de la comunicació presentada a International Conference on Health and Social Care Information Systems and Technologies (HCist 2017), celebrat a Barcelona el novembre de 2017., We present work in progress on an intelligent embodied conversation agent that is supposed to act as a social companion with linguistic and emotional competence in the context of basic and health care. The core of the agent is an ontology-based knowledge model that supports flexible reasoning-driven conversation planning strategies. A dedicated search engine ensures the provision of background information from the web, necessary for conducting a conversation on a specific topic. Multimodal communication analysis and generation modules analyze respectively generate facial expressions, gestures and multilingual speech. The assessment of the prototypical implementation of the agent shows that users accept it as a natural and trustworthy conversation counterpart. For the final release, all involved technologies will be further improved and matured., The presented work is funded by the European Commission as part of the H2020 Program under the contract number 645012. We would like to thank our colleagues Chiara Baudracco, Jutta Mohr, Eylem Ög, Valérie Sarholz, and Benjamin Schäfer for organizing and carrying out the evaluation trials and for their patience with the numerous inadequacies of KRISTINA.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

The Information structure–prosody interface in text-to-speech technologies: an empirical perspective

  • Domínguez Bajo, Mónica
  • Farrús, Mireia
  • Wanner, Leo
The correspondence between the communicative intention of a speaker in terms of Information Structure and the way this speaker reflects communicative aspects by means of prosody have been a fruitful field of study in Linguistics. However, text-to-speech applications still lack the variability and richness found in human speech in terms of how humans display their communication skills. Some attempts were made in the past to model one aspect of Information Structure, namely thematicity for its application to intonation generation in text-to-speech technologies. Yet these applications suffer from two limitations: (i) they draw upon a small number of made-up simple question-answer pairs rather than on real (spoken or written) corpus material; and (ii) they do not explore whether any other interpretation would better suit a wider range of textual genres beyond dialogues. In this paper, two different interpretations of thematicity in the field of speech technologies are examined: the state-of-art binary (and flat) theme-rheme, and the hierarchical thematicity defined by Igor Mel’ˇcuk within the Meaning-Text Theory. The outcome of the experiments on a corpus of native speakers of US English suggests that the latter interpretation of thematicity has a versatile implementation potential for text-to-speech applications of the Information Structure-–prosody interface, This work has been partially funded by the European Commission in the context of its H2020 Programme under the contract numbers H2020-645012-RIA (KRISTINA) and H2020-870930-IA (WELCOME). The second author has been funded by the Agencia Estatal de Investigaci´on (AEI), Ministerio de Ciencia, Innovaci´on y Universidades and the Fondo Social Europeo (FSE), grant RYC2015-17239 (AEI/FSE, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Survey on automatic lip-reading in the era of deep learning

  • Fernandez-Lopez, Adriana
  • Sukno, Federico Mateo
In the last few years, there has been an increasing interest in developing systems for Automatic Lip-Reading (ALR). Similarly to other computer vision applications, methods based on Deep Learning (DL) have become very popular and have permitted to substantially push forward the achievable performance. In this survey, we review ALR research during the last decade, highlighting the progression from approaches previous to DL (which we refer to as traditional) toward end-to-end DL architectures. We provide a comprehensive list of the audio-visual databases available for lip-reading, describing what tasks they can be used for, their popularity and their most important characteristics, such as the number of speakers, vocabulary size, recording settings and total duration. In correspondence with the shift toward DL, we show that there is a clear tendency toward large-scale datasets targeting realistic application settings and large numbers of samples per class. On the other hand, we summarize, discuss and compare the different ALR systems proposed in the last decade, separately considering traditional and DL approaches. We address a quantitative analysis of the different systems by organizing them in terms of the task that they target (e.g. recognition of letters or digits and words or sentences) and comparing their reported performance in the most commonly used datasets. As a result, we find that DL architectures perform similarly to traditional ones for simpler tasks but report significant improvements in more complex tasks, such as word or sentence recognition, with up to 40% improvement in word recognition rates. Hence, we provide a detailed description of the available ALR systems based on end-to-end DL architectures and identify a tendency to focus on the modeling of temporal context as the key to advance the field. Such modeling is dominated by recurrent neural networks due to their ability to retain context at multiple scales (e.g. short- and long-term information). In this sense, current efforts tend toward techniques that allow a more comprehensive modeling and interpretability of the retained context., This work is partly supported by the Spanish Ministry of Economy and Competitiveness, Spain under project grant TIN2017-90124-P, the Ramon y Cajal Programme, the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), and the Kristina project funded by the European Union Horizon 2020 - Research and Innovation Framework Programme under grant agreement No. 645012.