Skip to main content

MAS ALLA DE LAS REDES NEURONALES BASADAS EN GRAFOS: APRENDIZAJE DE SU TOPOLOGIA Y SU INFERENCIA DE FORMA CONJUNTA EN VISION POR ORDENADOR

PID2019-110977GA-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2019
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS (CSIC)

Resultados relacionados

Resultados totales (Incluyendo duplicados): 12
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Learning grounded word meaning representations on similarity graphs

  • Dimiccoli, Mariella
  • Wendt, Herwig
  • Batlle Franch, Pau
Trabajo presentado en la Conference on Empirical Methods in Natural Language Processing (EMNLP), celebrada de forma híbrida (online y presencial) desde Punta Cana (República Dominicana), del 7 al 11 de noviembre de 2021, This paper introduces a novel approach to learn visually grounded meaning representations of words as low-dimensional node embeddings on an underlying graph hierarchy. The lower level of the hierarchy models modality-specific word representations through dedicated but communicating graphs, while the higher level puts these representations together on a single graph to learn a representation jointly from both modalities. The topology of each graph models similarity relations among words, and is estimated jointly with the graph embedding.The assumption underlying this model is that words sharing similar meaning correspond to communities in an underlying similarity graph in a low-dimensional space. We named this model Hierarchical Multi-Modal Similarity Graph Embedding (HM-SGE). Experimental results validate the ability of HM-SGE to simulate human similarity judgements and concept categorization, outperforming the state of the art., Work partially funded by projects MINECO/ERDF RyC, PID2019-110977GA-I00, MDM-2016-0656.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Understanding event boundaries for egocentric activity recognition from photo-streams

  • Cartas, Alejandro
  • Talavera, Estefanía
  • Radeva, Petia
  • Dimiccoli, Mariella
Trabajo presentado en el ICPR International Workshops and Challenges, celebrado de forma virtual del 10 al 15 de enero de 2021, The recognition of human activities captured by a wearable photo-camera is especially suited for understanding the behavior of a person. However, it has received comparatively little attention with respect to activity recognition from fixed cameras.In this work, we propose to use segmented events from photo-streams as temporal boundaries to improve the performance of activity recognition. Furthermore, we robustly measure its effectiveness when images of the evaluated person have been seen during training, and when the person is completely unknown during testing. Experimental results show that leveraging temporal boundary information on pictures of seen people improves all classification metrics, particularly it improves the classification accuracy up to 85.73%., Lecture Notes in Computer Science 12663, This work was partially funded by projects RTI2018-095232-B-C2, SGR 1742, CERCA, Nestore Horizon2020 SC1-PM-15-2017 (n 769643), Validithi EIT Health Program, and the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (MINECO/ERDF, EU) through the program Ramon y Cajal, the national Spanish project PID2019-110977GA-I00 and the Spanish national network RED2018-102511-T. A. Cartas supported by a doctoral fellowship from the Mexican Council of Science and Technology (CONACYT) (grant-no. 366596). The authors acknowledge the support of NVIDIA Corporation for hardware donation.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Interaction-GCN: a Graph Convolutional Network based framework for social interaction recognition in egocentric videos

  • Felicioni, Simone
  • Dimiccoli, Mariella
Trabajo presentado en el International Conference on Image Processing (ICIP), celebrado de forma virtual del 19 al 22 de septiembre de 2021, In this paper we propose a new framework to categorize social interactions in egocentric videos, we named InteractionGCN. Our method extracts patterns of relational and non-relational cues at the frame level and uses them to build a relational graph from which the interactional context at the frame level is estimated via a Graph Convolutional Network based approach. Then it propagates this context over time, together with first-person motion information, through a Gated Recurrent Unit architecture. Ablation studies and experimental evaluation on two publicly available datasets validate the proposed approach and establish state of the art results., Work partially funded by projects MINECO/ERDF RyC, PID2019- 110977GA-I00, RED2018-102511-T, and MdM-IP-2019-03.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Leveraging triplet loss for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Tura Vecino, Biel
  • Dimiccoli, Mariella
Trabajo presentado en la Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), celebrada en Vancouver (Canadá), del 17 al 24 de junio de 2023, In this paper, we propose a novel fully unsupervised framework that learns action representations suitable for the action segmentation task from the single input video itself, without requiring any training data. Our method is a deep metric learning approach rooted in a shallow network with a triplet loss operating on similarity distributions and a novel triplet selection strategy that effectively models temporal and semantic priors to discover actions in the new representational space. Under these circumstances, we successfully recover temporal boundaries in the learned action representations with higher quality compared with existing unsupervised approaches. The proposed method is evaluated on two widely used benchmark datasets for the action segmentation task and it achieves competitive performance by applying a generic clustering algorithm on the learned representations., This work was supported by the project PID2019- 110977GA-I00 funded by MCIN/ AEI/ 10.13039/ 501100011033 and by ”ESF Investing in your future”, Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

3M-Transformer: a multi-stage multi-stream multimodal transformer for embodied turn-taking prediction

  • Fatan, Mehdi
  • Mincato, Emanuele
  • Pintzou, Dimitra
  • Dimiccoli, Mariella
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Predicting turn-taking in multiparty conversations has many practical applications in human-computer/robot interaction. However, the complexity of human communication makes it a challenging task. Recent advances have shown that synchronous multi-perspective egocentric data can significantly improve turn-taking prediction compared to asynchronous, single-perspective transcriptions. Building on this research, we propose a new multimodal transformer-based architecture for predicting turn-taking in embodied, synchronized multi-perspective data. Our experimental results on the recently introduced EgoCom dataset show a substantial performance improvement of up to 14.01% on average compared to existing baselines and alternative transformer-based approaches., This work has been supported by project PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/ PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Graph constrained data representation learning for human motion segmentation

  • Dimiccoli, Mariella
  • Garrido Beltrán, Lluís
  • Rodríguez Corominas, Guillem
  • Wendt, Herwig
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Recently, transfer subspace learning based approaches have shown to be a valid alternative to unsupervised subspace clustering and temporal data clustering for human motion segmentation (HMS). These approaches leverage prior knowledge from a source domain to improve clustering performance on a target domain, and currently they represent the state of the art in HMS. Bucking this trend, in this paper, we propose a novel unsupervised model that learns a representation of the data and digs clustering information from the data itself. Our model is reminiscent of temporal subspace clustering, but presents two critical differences. First, we learn an auxiliary data matrix that can deviate from the initial data, hence confers more degrees of freedom to the coding matrix. Second, we introduce a regularization term for this auxiliary data matrix that preserves the local geometrical structure present in the high-dimensional space. The proposed model is efficiently optimized by using an original Alternating Direction Method of Multipliers (ADMM) formulation allowing to learn jointly the auxiliary data representation, a nonnegative dictionary and a coding matrix. Experimental results on four benchmark datasets for HMS demonstrate that our approach achieves significantly better clustering performance then state-of-the-art methods, including both unsupervised and more recent semi-supervised transfer learning approaches., Work partially funded by projects MINECO/ERDF RyC, PID2019-110977GA-I00, JAEINT19-EX-0014, RED2018- 102511-T., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

CLOT: closed loop optimal transport for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent seg- mentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leverag- ing its encoder-decoder architecture, CLOT learns pseudo- labels alongside frame and segment embeddings by solv- ing two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention be- tween the learned frame and segment embeddings, by in- tegrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cycli- cal learning for unsupervised action segmentation., This work was supported by grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00, funded by Ministerio de Ciencia e Innovación (MCIN)/ Agencia Estatal de Investigación (AEI) /10.13039/501100011033, by European Social Fund (ESF) Investing in your future and by ERDF, UE. It was also supported by an Alexander von Humboldt (AvH) fellowship for experienced researchers funded by the AvH Foundation., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Leveraging triplet loss for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Tura Vecino, Biel
  • Dimiccoli, Mariella
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we propose a novel fully unsupervised framework that learns action representations suitable for the action segmentation task from the single input video itself, without requiring any training data. Our method is a deep metric learning approach rooted in a shallow network with a triplet loss operating on similarity distributions and a novel triplet selection strategy that effectively models temporal and semantic priors to discover actions in the new representational space. Under these circumstances, we successfully recover temporal boundaries in the learned action representations with higher quality compared with existing unsupervised approaches. The proposed method is evaluated on two widely used benchmark datasets for the action segmentation task and it achieves competitive performance by applying a generic clustering algorithm on the learned representations., This work was supported by the project PID2019-110977GA-I00 funded by MCIN/ AEI/ 10.13039/501100011033 and by ”ESF Investing in your future”, Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

CLOT: closed loop optimal transport for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent seg- mentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leverag- ing its encoder-decoder architecture, CLOT learns pseudo- labels alongside frame and segment embeddings by solv- ing two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention be- tween the learned frame and segment embeddings, by in- tegrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cycli- cal learning for unsupervised action segmentation., This work was supported by grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00, funded by Ministerio de Ciencia e Innovación (MCIN)/ Agencia Estatal de Investigación (AEI) /10.13039/501100011033, by European Social Fund (ESF) Investing in your future and by ERDF, UE. It was also supported by an Alexander von Humboldt (AvH) fellowship for experienced researchers funded by the AvH Foundation., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Leveraging triplet loss for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Tura Vecino, Biel
  • Dimiccoli, Mariella
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we propose a novel fully unsupervised framework that learns action representations suitable for the action segmentation task from the single input video itself, without requiring any training data. Our method is a deep metric learning approach rooted in a shallow network with a triplet loss operating on similarity distributions and a novel triplet selection strategy that effectively models temporal and semantic priors to discover actions in the new representational space. Under these circumstances, we successfully recover temporal boundaries in the learned action representations with higher quality compared with existing unsupervised approaches. The proposed method is evaluated on two widely used benchmark datasets for the action segmentation task and it achieves competitive performance by applying a generic clustering algorithm on the learned representations., This work was supported by the project PID2019-110977GA-I00 funded by MCIN/ AEI/ 10.13039/501100011033 and by ”ESF Investing in your future”, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

3M-Transformer: a multi-stage multi-stream multimodal transformer for embodied turn-taking prediction

  • Fatan, Mehdi
  • Mincato, Emanuele
  • Pintzou, Dimitra
  • Dimiccoli, Mariella
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Predicting turn-taking in multiparty conversations has many practical applications in human-computer/robot interaction. However, the complexity of human communication makes it a challenging task. Recent advances have shown that synchronous multi-perspective egocentric data can significantly improve turn-taking prediction compared to asynchronous, single-perspective transcriptions. Building on this research, we propose a new multimodal transformer-based architecture for predicting turn-taking in embodied, synchronized multi-perspective data. Our experimental results on the recently introduced EgoCom dataset show a substantial performance improvement of up to 14.01% on average compared to existing baselines and alternative transformer-based approaches., This work has been supported by project PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/ PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Graph constrained data representation learning for human motion segmentation

  • Dimiccoli, Mariella
  • Garrido Beltrán, Lluís
  • Rodríguez Corominas, Guillem
  • Wendt, Herwig
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Recently, transfer subspace learning based approaches have shown to be a valid alternative to unsupervised subspace clustering and temporal data clustering for human motion segmentation (HMS). These approaches leverage prior knowledge from a source domain to improve clustering performance on a target domain, and currently they represent the state of the art in HMS. Bucking this trend, in this paper, we propose a novel unsupervised model that learns a representation of the data and digs clustering information from the data itself. Our model is reminiscent of temporal subspace clustering, but presents two critical differences. First, we learn an auxiliary data matrix that can deviate from the initial data, hence confers more degrees of freedom to the coding matrix. Second, we introduce a regularization term for this auxiliary data matrix that preserves the local geometrical structure present in the high-dimensional space. The proposed model is efficiently optimized by using an original Alternating Direction Method of Multipliers (ADMM) formulation allowing to learn jointly the auxiliary data representation, a nonnegative dictionary and a coding matrix. Experimental results on four benchmark datasets for HMS demonstrate that our approach achieves significantly better clustering performance then state-of-the-art methods, including both unsupervised and more recent semi-supervised transfer learning approaches., Work partially funded by projects MINECO/ERDF RyC, PID2019-110977GA-I00, JAEINT19-EX-0014, RED2018- 102511-T., Peer Reviewed, Postprint (author's final draft)