Pasar al contenido principal

CONECTANDO EL PASADO, EL PRESENTE Y EL FUTURO: ADQUIRIR Y APROVECHAR CONOCIMIENTO APRIORI PARA REVELAR LA ESTRUCTURA TEMPORAL DE VIDEOS SIN RECORTAR

PID2023-151351NB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2023
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS (CSIC)

Resultados relacionados

Resultados totales (Incluyendo duplicados): 5
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

2by2: Weakly-Supervised Learning for Global Action Segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods., This work was supported by the grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033, by “ESF Investing in your future” and by ERDF, UE., Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Capítulo o parte de un libro (BookPart)

2by2: weakly-supervised learning for global action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods., This work was supported by the grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033, by "ESF Investing in your future" and by ERDF, UE
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

CLOT: closed loop optimal transport for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent seg- mentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leverag- ing its encoder-decoder architecture, CLOT learns pseudo- labels alongside frame and segment embeddings by solv- ing two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention be- tween the learned frame and segment embeddings, by in- tegrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cycli- cal learning for unsupervised action segmentation., This work was supported by grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00, funded by Ministerio de Ciencia e Innovación (MCIN)/ Agencia Estatal de Investigación (AEI) /10.13039/501100011033, by European Social Fund (ESF) Investing in your future and by ERDF, UE. It was also supported by an Alexander von Humboldt (AvH) fellowship for experienced researchers funded by the AvH Foundation., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

CLOT: closed loop optimal transport for unsupervised action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent seg- mentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leverag- ing its encoder-decoder architecture, CLOT learns pseudo- labels alongside frame and segment embeddings by solv- ing two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention be- tween the learned frame and segment embeddings, by in- tegrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cycli- cal learning for unsupervised action segmentation., This work was supported by grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00, funded by Ministerio de Ciencia e Innovación (MCIN)/ Agencia Estatal de Investigación (AEI) /10.13039/501100011033, by European Social Fund (ESF) Investing in your future and by ERDF, UE. It was also supported by an Alexander von Humboldt (AvH) fellowship for experienced researchers funded by the AvH Foundation., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación

2by2: weakly-supervised learning for global action segmentation

  • Bueno Benito, Elena Belén
  • Dimiccoli, Mariella
This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods., This work was supported by the grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033, by "ESF Investing in your future" and by ERDF, UE, Postprint (author's final draft)