Skip to main content

MAS ALLA DE LAS REDES NEURONALES BASADAS EN GRAFOS: APRENDIZAJE DE SU TOPOLOGIA Y SU INFERENCIA DE FORMA CONJUNTA EN VISION POR ORDENADOR

PID2019-110977GA-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2019
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS (CSIC)

Resultados relacionados

Resultados totales (Incluyendo duplicados): 19
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Learning grounded word meaning representations on similarity graphs

  • Dimiccoli, Mariella
  • Wendt, Herwig
  • Batlle Franch, Pau
Trabajo presentado en la Conference on Empirical Methods in Natural Language Processing (EMNLP), celebrada de forma híbrida (online y presencial) desde Punta Cana (República Dominicana), del 7 al 11 de noviembre de 2021, This paper introduces a novel approach to learn visually grounded meaning representations of words as low-dimensional node embeddings on an underlying graph hierarchy. The lower level of the hierarchy models modality-specific word representations through dedicated but communicating graphs, while the higher level puts these representations together on a single graph to learn a representation jointly from both modalities. The topology of each graph models similarity relations among words, and is estimated jointly with the graph embedding.The assumption underlying this model is that words sharing similar meaning correspond to communities in an underlying similarity graph in a low-dimensional space. We named this model Hierarchical Multi-Modal Similarity Graph Embedding (HM-SGE). Experimental results validate the ability of HM-SGE to simulate human similarity judgements and concept categorization, outperforming the state of the art., Work partially funded by projects MINECO/ERDF RyC, PID2019-110977GA-I00, MDM-2016-0656.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Understanding event boundaries for egocentric activity recognition from photo-streams

  • Cartas, Alejandro
  • Talavera, Estefanía
  • Radeva, Petia
  • Dimiccoli, Mariella
Trabajo presentado en el ICPR International Workshops and Challenges, celebrado de forma virtual del 10 al 15 de enero de 2021, The recognition of human activities captured by a wearable photo-camera is especially suited for understanding the behavior of a person. However, it has received comparatively little attention with respect to activity recognition from fixed cameras.In this work, we propose to use segmented events from photo-streams as temporal boundaries to improve the performance of activity recognition. Furthermore, we robustly measure its effectiveness when images of the evaluated person have been seen during training, and when the person is completely unknown during testing. Experimental results show that leveraging temporal boundary information on pictures of seen people improves all classification metrics, particularly it improves the classification accuracy up to 85.73%., Lecture Notes in Computer Science 12663, This work was partially funded by projects RTI2018-095232-B-C2, SGR 1742, CERCA, Nestore Horizon2020 SC1-PM-15-2017 (n 769643), Validithi EIT Health Program, and the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (MINECO/ERDF, EU) through the program Ramon y Cajal, the national Spanish project PID2019-110977GA-I00 and the Spanish national network RED2018-102511-T. A. Cartas supported by a doctoral fellowship from the Mexican Council of Science and Technology (CONACYT) (grant-no. 366596). The authors acknowledge the support of NVIDIA Corporation for hardware donation.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Interaction-GCN: a Graph Convolutional Network based framework for social interaction recognition in egocentric videos

  • Felicioni, Simone
  • Dimiccoli, Mariella
Trabajo presentado en el International Conference on Image Processing (ICIP), celebrado de forma virtual del 19 al 22 de septiembre de 2021, In this paper we propose a new framework to categorize social interactions in egocentric videos, we named InteractionGCN. Our method extracts patterns of relational and non-relational cues at the frame level and uses them to build a relational graph from which the interactional context at the frame level is estimated via a Graph Convolutional Network based approach. Then it propagates this context over time, together with first-person motion information, through a Gated Recurrent Unit architecture. Ablation studies and experimental evaluation on two publicly available datasets validate the proposed approach and establish state of the art results., Work partially funded by projects MINECO/ERDF RyC, PID2019- 110977GA-I00, RED2018-102511-T, and MdM-IP-2019-03.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Enhancing egocentric 3D pose estimation with third person views

  • Dhamanaskar, Ameya
  • Dimiccoli, Mariella
  • Corona, Enric
  • Pumarola, Albert
  • Moreno-Noguer, Francesc
We propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. The main technical contribution consists of leveraging high-level features linking first- and third-views in a joint embedding space. To learn such embedding space we introduce First2Third-Pose, a new paired synchronized dataset of nearly 2000 videos depicting human activities captured from both first- and third-view perspectives. We explicitly consider spatial- and motion-domain features, combined using a semi-Siamese architecture trained in a self-supervised fashion. Experimental results demonstrate that the joint multi-view embedded space learned with our dataset is useful to extract discriminatory features from arbitrary single-view egocentric videos, with no need to perform any sort of domain adaptation or knowledge of camera parameters. An extensive evaluation demonstrates that we achieve significant improvement in egocentric 3D body pose estimation performance on two unconstrained datasets, over three supervised state-of-the-art approaches. The collected dataset and pre-trained model are available for research purposes., This work has been partially supported by projects PID2020-120049RB-I00 and PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033 and by “ESF Investing in your future”, and network RED2018-102511-T funded by MCIN/ AEI., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Leveraging triplet loss for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Tura Vecino, Biel
  • Dimiccoli, Mariella
Trabajo presentado en la Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), celebrada en Vancouver (Canadá), del 17 al 24 de junio de 2023, In this paper, we propose a novel fully unsupervised framework that learns action representations suitable for the action segmentation task from the single input video itself, without requiring any training data. Our method is a deep metric learning approach rooted in a shallow network with a triplet loss operating on similarity distributions and a novel triplet selection strategy that effectively models temporal and semantic priors to discover actions in the new representational space. Under these circumstances, we successfully recover temporal boundaries in the learned action representations with higher quality compared with existing unsupervised approaches. The proposed method is evaluated on two widely used benchmark datasets for the action segmentation task and it achieves competitive performance by applying a generic clustering algorithm on the learned representations., This work was supported by the project PID2019- 110977GA-I00 funded by MCIN/ AEI/ 10.13039/ 501100011033 and by ”ESF Investing in your future”, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

2by2: Weakly-Supervised Learning for Global Action Segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods., This work was supported by the grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033, by “ESF Investing in your future” and by ERDF, UE., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

3M-Transformer: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Prediction

  • Fatan, Mehdi
  • Mincato, Emanuele
  • Pintzou, Dimitra
  • Dimiccoli, Mariella
Predicting turn-taking in multiparty conversations has many practical applications in human-computer/robot interaction. However, the complexity of human communication makes it a challenging task. Recent advances have shown that synchronous multi-perspective egocentric data can significantly improve turn-taking prediction compared to asynchronous, single-perspective transcriptions. Building on this research, we propose a new multimodal transformer-based architecture for predicting turn-taking in embodied, synchronized multi-perspective data. Our experimental results on the recently introduced EgoCom dataset show a substantial performance improvement of up to 14.01% on average compared to existing baselines and alternative transformer-based approaches. The source code, and the pre-trained models of our 3T-Transformer will be available upon acceptance., This work has been supported by project PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/ PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033., Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Capítulo o parte de un libro (BookPart)

2by2: weakly-supervised learning for global action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods., This work was supported by the grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033, by "ESF Investing in your future" and by ERDF, UE
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Enhancing egocentric 3D pose estimation with third person views

  • Dhamanaskar, Ameya
  • Dimiccoli, Mariella
  • Corona Puyane, Enric
  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Moreno-Noguer, Francesc
© 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND, We propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. The main technical contribution consists of leveraging high-level features linking first- and third-views in a joint embedding space. To learn such embedding space we introduce First2Third-Pose, a new paired synchronized dataset of nearly 2000 videos depicting human activities captured from both first- and third-view perspectives. We explicitly consider spatial- and motion-domain features, combined using a semi-Siamese architecture trained in a self-supervised fashion. Experimental results demonstrate that the joint multi-view embedded space learned with our dataset is useful to extract discriminatory features from arbitrary single-view egocentric videos, with no need to perform any sort of domain adaptation or knowledge of camera parameters. An extensive evalu- ation demonstrates that we achieve significant improvement in egocentric 3D body pose estimation per- formance on two unconstrained datasets, over three supervised state-of-the-art approaches. The collected dataset and pre-trained model are available for research purposes., This work has been partially supported by projects PID2020-120 049RB-I00 and PID2019-110977GA-I00 funded by MCIN/ AEI/10.13039/501100 011033 and by the “European Union NextGener-ationEU/PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100 011033 and by “ESF Investing in your future”, and network RED2018-102511-T funded by MCIN/ AEI, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

3M-Transformer: a multi-stage multi-stream multimodal transformer for embodied turn-taking prediction

  • Fatan, Mehdi
  • Mincato, Emanuele
  • Pintzou, Dimitra
  • Dimiccoli, Mariella
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Predicting turn-taking in multiparty conversations has many practical applications in human-computer/robot interaction. However, the complexity of human communication makes it a challenging task. Recent advances have shown that synchronous multi-perspective egocentric data can significantly improve turn-taking prediction compared to asynchronous, single-perspective transcriptions. Building on this research, we propose a new multimodal transformer-based architecture for predicting turn-taking in embodied, synchronized multi-perspective data. Our experimental results on the recently introduced EgoCom dataset show a substantial performance improvement of up to 14.01% on average compared to existing baselines and alternative transformer-based approaches., This work has been supported by project PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/ PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Graph constrained data representation learning for human motion segmentation

  • Dimiccoli, Mariella
  • Garrido Beltrán, Lluís
  • Rodríguez Corominas, Guillem
  • Wendt, Herwig
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Recently, transfer subspace learning based approaches have shown to be a valid alternative to unsupervised subspace clustering and temporal data clustering for human motion segmentation (HMS). These approaches leverage prior knowledge from a source domain to improve clustering performance on a target domain, and currently they represent the state of the art in HMS. Bucking this trend, in this paper, we propose a novel unsupervised model that learns a representation of the data and digs clustering information from the data itself. Our model is reminiscent of temporal subspace clustering, but presents two critical differences. First, we learn an auxiliary data matrix that can deviate from the initial data, hence confers more degrees of freedom to the coding matrix. Second, we introduce a regularization term for this auxiliary data matrix that preserves the local geometrical structure present in the high-dimensional space. The proposed model is efficiently optimized by using an original Alternating Direction Method of Multipliers (ADMM) formulation allowing to learn jointly the auxiliary data representation, a nonnegative dictionary and a coding matrix. Experimental results on four benchmark datasets for HMS demonstrate that our approach achieves significantly better clustering performance then state-of-the-art methods, including both unsupervised and more recent semi-supervised transfer learning approaches., Work partially funded by projects MINECO/ERDF RyC, PID2019-110977GA-I00, JAEINT19-EX-0014, RED2018- 102511-T., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

CLOT: closed loop optimal transport for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent seg- mentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leverag- ing its encoder-decoder architecture, CLOT learns pseudo- labels alongside frame and segment embeddings by solv- ing two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention be- tween the learned frame and segment embeddings, by in- tegrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cycli- cal learning for unsupervised action segmentation., This work was supported by grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00, funded by Ministerio de Ciencia e Innovación (MCIN)/ Agencia Estatal de Investigación (AEI) /10.13039/501100011033, by European Social Fund (ESF) Investing in your future and by ERDF, UE. It was also supported by an Alexander von Humboldt (AvH) fellowship for experienced researchers funded by the AvH Foundation., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Leveraging triplet loss for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Tura Vecino, Biel
  • Dimiccoli, Mariella
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we propose a novel fully unsupervised framework that learns action representations suitable for the action segmentation task from the single input video itself, without requiring any training data. Our method is a deep metric learning approach rooted in a shallow network with a triplet loss operating on similarity distributions and a novel triplet selection strategy that effectively models temporal and semantic priors to discover actions in the new representational space. Under these circumstances, we successfully recover temporal boundaries in the learned action representations with higher quality compared with existing unsupervised approaches. The proposed method is evaluated on two widely used benchmark datasets for the action segmentation task and it achieves competitive performance by applying a generic clustering algorithm on the learned representations., This work was supported by the project PID2019-110977GA-I00 funded by MCIN/ AEI/ 10.13039/501100011033 and by ”ESF Investing in your future”, Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

CLOT: closed loop optimal transport for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent seg- mentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leverag- ing its encoder-decoder architecture, CLOT learns pseudo- labels alongside frame and segment embeddings by solv- ing two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention be- tween the learned frame and segment embeddings, by in- tegrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cycli- cal learning for unsupervised action segmentation., This work was supported by grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00, funded by Ministerio de Ciencia e Innovación (MCIN)/ Agencia Estatal de Investigación (AEI) /10.13039/501100011033, by European Social Fund (ESF) Investing in your future and by ERDF, UE. It was also supported by an Alexander von Humboldt (AvH) fellowship for experienced researchers funded by the AvH Foundation., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication

2by2: weakly-supervised learning for global action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods., This work was supported by the grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033, by "ESF Investing in your future" and by ERDF, UE, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Leveraging triplet loss for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Tura Vecino, Biel
  • Dimiccoli, Mariella
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we propose a novel fully unsupervised framework that learns action representations suitable for the action segmentation task from the single input video itself, without requiring any training data. Our method is a deep metric learning approach rooted in a shallow network with a triplet loss operating on similarity distributions and a novel triplet selection strategy that effectively models temporal and semantic priors to discover actions in the new representational space. Under these circumstances, we successfully recover temporal boundaries in the learned action representations with higher quality compared with existing unsupervised approaches. The proposed method is evaluated on two widely used benchmark datasets for the action segmentation task and it achieves competitive performance by applying a generic clustering algorithm on the learned representations., This work was supported by the project PID2019-110977GA-I00 funded by MCIN/ AEI/ 10.13039/501100011033 and by ”ESF Investing in your future”, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

3M-Transformer: a multi-stage multi-stream multimodal transformer for embodied turn-taking prediction

  • Fatan, Mehdi
  • Mincato, Emanuele
  • Pintzou, Dimitra
  • Dimiccoli, Mariella
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Predicting turn-taking in multiparty conversations has many practical applications in human-computer/robot interaction. However, the complexity of human communication makes it a challenging task. Recent advances have shown that synchronous multi-perspective egocentric data can significantly improve turn-taking prediction compared to asynchronous, single-perspective transcriptions. Building on this research, we propose a new multimodal transformer-based architecture for predicting turn-taking in embodied, synchronized multi-perspective data. Our experimental results on the recently introduced EgoCom dataset show a substantial performance improvement of up to 14.01% on average compared to existing baselines and alternative transformer-based approaches., This work has been supported by project PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/ PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Enhancing egocentric 3D pose estimation with third person views

  • Dhamanaskar, Ameya
  • Dimiccoli, Mariella
  • Corona Puyane, Enric
  • Pumarola Peris, Albert
  • Moreno-Noguer, Francesc
© 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND, We propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. The main technical contribution consists of leveraging high-level features linking first- and third-views in a joint embedding space. To learn such embedding space we introduce First2Third-Pose, a new paired synchronized dataset of nearly 2000 videos depicting human activities captured from both first- and third-view perspectives. We explicitly consider spatial- and motion-domain features, combined using a semi-Siamese architecture trained in a self-supervised fashion. Experimental results demonstrate that the joint multi-view embedded space learned with our dataset is useful to extract discriminatory features from arbitrary single-view egocentric videos, with no need to perform any sort of domain adaptation or knowledge of camera parameters. An extensive evalu- ation demonstrates that we achieve significant improvement in egocentric 3D body pose estimation per- formance on two unconstrained datasets, over three supervised state-of-the-art approaches. The collected dataset and pre-trained model are available for research purposes., This work has been partially supported by projects PID2020-120 049RB-I00 and PID2019-110977GA-I00 funded by MCIN/ AEI/10.13039/501100 011033 and by the “European Union NextGener-ationEU/PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100 011033 and by “ESF Investing in your future”, and network RED2018-102511-T funded by MCIN/ AEI, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Graph constrained data representation learning for human motion segmentation

  • Dimiccoli, Mariella
  • Garrido Beltrán, Lluís
  • Rodríguez Corominas, Guillem
  • Wendt, Herwig
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Recently, transfer subspace learning based approaches have shown to be a valid alternative to unsupervised subspace clustering and temporal data clustering for human motion segmentation (HMS). These approaches leverage prior knowledge from a source domain to improve clustering performance on a target domain, and currently they represent the state of the art in HMS. Bucking this trend, in this paper, we propose a novel unsupervised model that learns a representation of the data and digs clustering information from the data itself. Our model is reminiscent of temporal subspace clustering, but presents two critical differences. First, we learn an auxiliary data matrix that can deviate from the initial data, hence confers more degrees of freedom to the coding matrix. Second, we introduce a regularization term for this auxiliary data matrix that preserves the local geometrical structure present in the high-dimensional space. The proposed model is efficiently optimized by using an original Alternating Direction Method of Multipliers (ADMM) formulation allowing to learn jointly the auxiliary data representation, a nonnegative dictionary and a coding matrix. Experimental results on four benchmark datasets for HMS demonstrate that our approach achieves significantly better clustering performance then state-of-the-art methods, including both unsupervised and more recent semi-supervised transfer learning approaches., Work partially funded by projects MINECO/ERDF RyC, PID2019-110977GA-I00, JAEINT19-EX-0014, RED2018- 102511-T., Peer Reviewed, Postprint (author's final draft)