Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 12
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Self-supervised small soccer player detection and tracking

  • Hurault, Samuel
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al MMSports '20: The 3rd International Workshop on Multimedia Content Analysis in Sports, celebrat el 16 d'octubre de 2020 de manera virtual., In a soccer game, the information provided by detecting and tracking brings crucial clues to further analyze and understand some tactical aspects of the game, including individual and team actions. State-of-the-art tracking algorithms achieve impressive results in scenarios on which they have been trained for, but they fail in challenging ones such as soccer games. This is frequently due to the player small relative size and the similar appearance among players of the same team. Although a straightforward solution would be to retrain these models by using a more specific dataset, the lack of such publicly available annotated datasets entails searching for other effective solutions. In this work, we propose a self-supervised pipeline which is able to detect and track low-resolution soccer players under different recording conditions without any need of ground-truth data. Extensive quantitative and qualitative experimental results are presented evaluating its performance. We also present a comparison to several state-of-the-art methods showing that both the proposed detector and the proposed tracker achieve top-tier results, in particular in the presence of small players., SH acknowledges support by ENS Paris-Saclay. CB and GH acknowledge support by MICINN/FEDER UE project, ref. PGC2018-098625- B-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and RED2018-102511-T.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

VocaLiST: an audio-visual synchronisation model for lips and voices

  • Kadandale, Venkatesh S.
  • Montesinos García, Juan Felipe
  • Haro Ortega, Gloria
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audiovisual correspondence score. We propose an audio-visual crossmodal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the singing voice. The singing voice is a more challenging use case for synchronisation due to sustained vowel sounds. We also investigate the relevance of lip synchronisation models trained on speech datasets in the context of singing voice. Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end. The demos, source code, and the pre-trained models are available on https://ipcv.github.io/VocaLiST/, We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. V. S. K. has received support through “la Caixa” Foundation (ID 100010434), fellowship code: LCF/BQ/DI18/11660064 and the Marie SkłodowskaCurie grant agreement No. 713673. J. F. M. acknowledges support by FPI scholarship PRE2018-083920.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Semantic image inpainting through improved Wasserstein generative adversarial networks

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISAPP 2019, celebrada a Praga, República Txeca, del 25 al 27 de febrer de 2019, Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the missing content conditioned by the available data. It takes into account powerful contextual and perceptual content inherent in the image itself. The benefits include the ability to recover large regions by accumulating semantic information even it is not fully present in the damaged image. Experiments show that the presented method obtains qualitative and quantitative top-tier results in different experimental situations and also achieves accurate photo-realism comparable to state-of-the-art works., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015-70410-C2-1 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

A graph-based method for soccer action spotting using unsupervised player classification

  • Cartas, Alejandro
  • Ballester, Coloma
  • Haro Ortega, Gloria
Action spotting in soccer videos is the task of identifying the specific time when a certain key action of the game occurs. Lately, it has received a large amount of attention and powerful methods have been introduced. Action spotting involves understanding the dynamics of the game, the complexity of events, and the variation of video sequences. Most approaches have focused on the latter, given that their models exploit the global visual features of the sequences. In this work, we focus on the former by (a) identifying and representing the players, referees, and goalkeepers as nodes in a graph, and by (b) modeling their temporal interactions as sequences of graphs. For the player identification, or player classification task, we obtain an accuracy of 97.72% in our annotated benchmark. For the action spotting task, our method obtains an overall performance of 57.83% average-mAP by combining it with other audiovisual modalities. This performance surpasses similar graph-based methods and has competitive results with heavy computing methods. Code and data are available at https://github.com/IPCV/soccer_action_spotting., The authors acknowledge support by MICINN/FEDER UE project, ref. PID2021-127643NB-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and ReAViPeRo network, ref. RED2018-102511-T.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Semantic image completion through an adversarial strategy

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISIGRAPP 2019: Computer Vision, Imaging and Computer Graphics Theory and Applications. 14th International Joint Conference, celebrat del 25 al 27 de febrer de 2019 a Praga, República Txeca., Image completion or image inpainting is the task of filling in missing regions of an image. When those areas are large and the missing information is unique such that the information and redundancy available in the image is not useful to guide the completion, the task becomes even more challenging. This paper proposes an automatic semantic inpainting method able to reconstruct corrupted information of an image by semantically interpreting the image itself. It is based on an adversarial strategy followed by an energy-based completion algorithm. First, the data latent space is learned by training a modified Wasserstein generative adversarial network. Second, the learned semantic information is combined with a novel optimization loss able to recover missing regions conditioned by the available information. Moreover, we present an application in the context of face inpainting, where our method is used to generate a new face by integrating desired facial attributes or expressions from a reference face. This is achieved by slightly modifying the objective energy. Quantitative and qualitative top-tier results show the power and realism of the presented method., The authors acknowledge partial support by MICINN/FEDER UE project, reference PGC2018-098625-B-I00 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS. We also thank NVIDIA for the Quadro P6000 GPU donation.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

VoViT: low latency graph-based audio-visual voice separation transformer

  • Montesinos García, Juan Felipe
  • Kadandale, Venkatesh S.
  • Haro Ortega, Gloria
This paper presents an audio-visual approach for voice separation which produces state-of-the-art results at a low latency in two scenarios: speech and singing voice. The model is based on a two-stage network. Motion cues are obtained with a lightweight graph convolutional network that processes face landmarks. Then, both audio and motion features are fed to an audio-visual transformer which produces a fairly good estimation of the isolated target source. In a second stage, the predominant voice is enhanced with an audio-only network. We present different ablation studies and comparison to state-of-the-art methods. Finally, we explore the transferability of models trained for speech separation in the task of singing voice separation. The demos, code, and weights are available in https://ipcv.github.io/VoViT/., We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. J.F.M. acknowledges support by FPI scholarship PRE2018-083920. We acknowledge NVIDIA Corporation for the donation of GPUs used for the experiments.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Self-supervised small soccer player detection and tracking

  • Hurault, Samuel
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al MMSports '20: The 3rd International Workshop on Multimedia Content Analysis in Sports, celebrat el 16 d'octubre de 2020 de manera virtual., In a soccer game, the information provided by detecting and tracking brings crucial clues to further analyze and understand some tactical aspects of the game, including individual and team actions. State-of-the-art tracking algorithms achieve impressive results in scenarios on which they have been trained for, but they fail in challenging ones such as soccer games. This is frequently due to the player small relative size and the similar appearance among players of the same team. Although a straightforward solution would be to retrain these models by using a more specific dataset, the lack of such publicly available annotated datasets entails searching for other effective solutions. In this work, we propose a self-supervised pipeline which is able to detect and track low-resolution soccer players under different recording conditions without any need of ground-truth data. Extensive quantitative and qualitative experimental results are presented evaluating its performance. We also present a comparison to several state-of-the-art methods showing that both the proposed detector and the proposed tracker achieve top-tier results, in particular in the presence of small players., SH acknowledges support by ENS Paris-Saclay. CB and GH acknowledge support by MICINN/FEDER UE project, ref. PGC2018-098625- B-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and RED2018-102511-T.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

A graph-based method for soccer action spotting using unsupervised player classification

  • Cartas, Alejandro
  • Ballester, Coloma
  • Haro Ortega, Gloria
Action spotting in soccer videos is the task of identifying the specific time when a certain key action of the game occurs. Lately, it has received a large amount of attention and powerful methods have been introduced. Action spotting involves understanding the dynamics of the game, the complexity of events, and the variation of video sequences. Most approaches have focused on the latter, given that their models exploit the global visual features of the sequences. In this work, we focus on the former by (a) identifying and representing the players, referees, and goalkeepers as nodes in a graph, and by (b) modeling their temporal interactions as sequences of graphs. For the player identification, or player classification task, we obtain an accuracy of 97.72% in our annotated benchmark. For the action spotting task, our method obtains an overall performance of 57.83% average-mAP by combining it with other audiovisual modalities. This performance surpasses similar graph-based methods and has competitive results with heavy computing methods. Code and data are available at https://github.com/IPCV/soccer_action_spotting., The authors acknowledge support by MICINN/FEDER UE project, ref. PID2021-127643NB-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and ReAViPeRo network, ref. RED2018-102511-T.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

VocaLiST: an audio-visual synchronisation model for lips and voices

  • Kadandale, Venkatesh S.
  • Montesinos García, Juan Felipe
  • Haro Ortega, Gloria
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audiovisual correspondence score. We propose an audio-visual crossmodal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the singing voice. The singing voice is a more challenging use case for synchronisation due to sustained vowel sounds. We also investigate the relevance of lip synchronisation models trained on speech datasets in the context of singing voice. Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end. The demos, source code, and the pre-trained models are available on https://ipcv.github.io/VocaLiST/, We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. V. S. K. has received support through “la Caixa” Foundation (ID 100010434), fellowship code: LCF/BQ/DI18/11660064 and the Marie SkłodowskaCurie grant agreement No. 713673. J. F. M. acknowledges support by FPI scholarship PRE2018-083920.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Semantic image completion through an adversarial strategy

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISIGRAPP 2019: Computer Vision, Imaging and Computer Graphics Theory and Applications. 14th International Joint Conference, celebrat del 25 al 27 de febrer de 2019 a Praga, República Txeca., Image completion or image inpainting is the task of filling in missing regions of an image. When those areas are large and the missing information is unique such that the information and redundancy available in the image is not useful to guide the completion, the task becomes even more challenging. This paper proposes an automatic semantic inpainting method able to reconstruct corrupted information of an image by semantically interpreting the image itself. It is based on an adversarial strategy followed by an energy-based completion algorithm. First, the data latent space is learned by training a modified Wasserstein generative adversarial network. Second, the learned semantic information is combined with a novel optimization loss able to recover missing regions conditioned by the available information. Moreover, we present an application in the context of face inpainting, where our method is used to generate a new face by integrating desired facial attributes or expressions from a reference face. This is achieved by slightly modifying the objective energy. Quantitative and qualitative top-tier results show the power and realism of the presented method., The authors acknowledge partial support by MICINN/FEDER UE project, reference PGC2018-098625-B-I00 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS. We also thank NVIDIA for the Quadro P6000 GPU donation.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Semantic image inpainting through improved Wasserstein generative adversarial networks

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISAPP 2019, celebrada a Praga, República Txeca, del 25 al 27 de febrer de 2019, Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the missing content conditioned by the available data. It takes into account powerful contextual and perceptual content inherent in the image itself. The benefits include the ability to recover large regions by accumulating semantic information even it is not fully present in the damaged image. Experiments show that the presented method obtains qualitative and quantitative top-tier results in different experimental situations and also achieves accurate photo-realism comparable to state-of-the-art works., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015-70410-C2-1 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

VoViT: low latency graph-based audio-visual voice separation transformer

  • Montesinos García, Juan Felipe
  • Kadandale, Venkatesh S.
  • Haro Ortega, Gloria
This paper presents an audio-visual approach for voice separation which produces state-of-the-art results at a low latency in two scenarios: speech and singing voice. The model is based on a two-stage network. Motion cues are obtained with a lightweight graph convolutional network that processes face landmarks. Then, both audio and motion features are fed to an audio-visual transformer which produces a fairly good estimation of the isolated target source. In a second stage, the predominant voice is enhanced with an audio-only network. We present different ablation studies and comparison to state-of-the-art methods. Finally, we explore the transferability of models trained for speech separation in the task of singing voice separation. The demos, code, and weights are available in https://ipcv.github.io/VoViT/., We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. J.F.M. acknowledges support by FPI scholarship PRE2018-083920. We acknowledge NVIDIA Corporation for the donation of GPUs used for the experiments.