Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 28
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

An affine invariant patch similarity

  • Fedorov, Vadim
  • Ballester, Coloma
Image and video comparison is often approached by comparing patches of visual information. In this work we present a detailed description and implementation of an affine invariant patch simi- larity measure that performs an appropriate patch comparison by automatically and intrinsically adapting the size and shape of the patches. We also describe the complete implementation of the proposed iterative algorithm for computation of those shape-adaptive patches around each point in the image domain., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015- 70410-C2-1-R, by GRC reference 2014 SGR 1301 Generalitat de Catalunya, and by H2020-MSCARISE-2017 project, reference 777826 NoMADS.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Self-supervised small soccer player detection and tracking

  • Hurault, Samuel
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al MMSports '20: The 3rd International Workshop on Multimedia Content Analysis in Sports, celebrat el 16 d'octubre de 2020 de manera virtual., In a soccer game, the information provided by detecting and tracking brings crucial clues to further analyze and understand some tactical aspects of the game, including individual and team actions. State-of-the-art tracking algorithms achieve impressive results in scenarios on which they have been trained for, but they fail in challenging ones such as soccer games. This is frequently due to the player small relative size and the similar appearance among players of the same team. Although a straightforward solution would be to retrain these models by using a more specific dataset, the lack of such publicly available annotated datasets entails searching for other effective solutions. In this work, we propose a self-supervised pipeline which is able to detect and track low-resolution soccer players under different recording conditions without any need of ground-truth data. Extensive quantitative and qualitative experimental results are presented evaluating its performance. We also present a comparison to several state-of-the-art methods showing that both the proposed detector and the proposed tracker achieve top-tier results, in particular in the presence of small players., SH acknowledges support by ENS Paris-Saclay. CB and GH acknowledge support by MICINN/FEDER UE project, ref. PGC2018-098625- B-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and RED2018-102511-T.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

VocaLiST: an audio-visual synchronisation model for lips and voices

  • Kadandale, Venkatesh S.
  • Montesinos García, Juan Felipe
  • Haro Ortega, Gloria
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audiovisual correspondence score. We propose an audio-visual crossmodal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the singing voice. The singing voice is a more challenging use case for synchronisation due to sustained vowel sounds. We also investigate the relevance of lip synchronisation models trained on speech datasets in the context of singing voice. Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end. The demos, source code, and the pre-trained models are available on https://ipcv.github.io/VocaLiST/, We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. V. S. K. has received support through “la Caixa” Foundation (ID 100010434), fellowship code: LCF/BQ/DI18/11660064 and the Marie SkłodowskaCurie grant agreement No. 713673. J. F. M. acknowledges support by FPI scholarship PRE2018-083920.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Semantic image inpainting through improved Wasserstein generative adversarial networks

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISAPP 2019, celebrada a Praga, República Txeca, del 25 al 27 de febrer de 2019, Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the missing content conditioned by the available data. It takes into account powerful contextual and perceptual content inherent in the image itself. The benefits include the ability to recover large regions by accumulating semantic information even it is not fully present in the damaged image. Experiments show that the presented method obtains qualitative and quantitative top-tier results in different experimental situations and also achieves accurate photo-realism comparable to state-of-the-art works., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015-70410-C2-1 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Towards soccer pass feasibility maps: the role of players’ orientation

  • Arbués Sangüesa, Adrià
  • Martin, Adrian
  • Fernández, Javier
  • Haro Ortega, Gloria
  • Ballester, Coloma
Once player tracking has been established as one of the main data sources in soccer, many challenges have emerged for data scientists, who attempt to recognize patterns from 2D trajectories in order to build tools that might help coaches to improve the performance of their teams. For instance, pass models predict where the ball should go next during pass events. However, existing models are mainly fed with players’ location and prior data, hence omitting critical pieces of information such as players’ body orientation. This paper presents a computational model to obtain pass feasibility maps, where player orientation is exploited and analysed. As a matter of fact, orientation proves to be crucial when modelling field-of-view and correct positioning of players, since it limits the potential receiving area of all candidates. Different proposals are given to evaluate the proposed pass feasibility map, reaching 0.46 and 0.79 in Top1 and Top3 accuracy, respectively, with a + 0.2 boost obtained after merging positional data with orientation., This work was supported by MICINN / FEDER UE (Funded by Ministerio de Ciencia, Innovación y Universidades) under Grants PGC2018-098625-B-I00 and FJCI-2017-31758, European Comission, HORIZON 2020 under grant 777826 NoMADS, and Ministerio de Ciencia, Innovación y Universidades (Redes Temáticas de Investigación) under grant RED2018-102511-T.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

A graph-based method for soccer action spotting using unsupervised player classification

  • Cartas, Alejandro
  • Ballester, Coloma
  • Haro Ortega, Gloria
Action spotting in soccer videos is the task of identifying the specific time when a certain key action of the game occurs. Lately, it has received a large amount of attention and powerful methods have been introduced. Action spotting involves understanding the dynamics of the game, the complexity of events, and the variation of video sequences. Most approaches have focused on the latter, given that their models exploit the global visual features of the sequences. In this work, we focus on the former by (a) identifying and representing the players, referees, and goalkeepers as nodes in a graph, and by (b) modeling their temporal interactions as sequences of graphs. For the player identification, or player classification task, we obtain an accuracy of 97.72% in our annotated benchmark. For the action spotting task, our method obtains an overall performance of 57.83% average-mAP by combining it with other audiovisual modalities. This performance surpasses similar graph-based methods and has competitive results with heavy computing methods. Code and data are available at https://github.com/IPCV/soccer_action_spotting., The authors acknowledge support by MICINN/FEDER UE project, ref. PID2021-127643NB-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and ReAViPeRo network, ref. RED2018-102511-T.
Repositorio Digital de la UPF
Publicación Capítulo o parte de un libro (BookPart)

An analysis of generative methods for multiple image inpainting

  • Ballester, Coloma
  • Bugeau, Aurélie
  • Hurault, Samuel
  • Parisotto, Simone
  • Vitoria, Patricia
Image inpainting refers to the restoration of an image with missing regions in a way that is not detectable by the observer. The inpainting regions can be of any size and shape. This is an ill-posed inverse problem that does not have a unique solution. In this work, we focus on learning-based image completion methods for multiple and diverse inpainting which goal is to provide a set of distinct solutions for a given damaged image. These methods capitalize on the probabilistic nature of certain deep generative models to sample various solutions that coherently restore the missing content. Throughout the chapter, we will analyze the underlying theory and analyze the recent proposals for multiple inpainting. To investigate the pros and cons of each method, we present quantitative and qualitative comparisons, on common datasets, regarding both the quality and the diversity of the set of inpainted solutions. Our analysis allows us to identify the most successful generative strategies in both inpainting quality and inpainting diversity. This task is closely related to the learning of an accurate probability distribution of images. Depending on the dataset in use, the challenges that entail the training of such a model will be discussed through the analysis., PV, CB and AB acknowledge the EU Horizon 2020 research and innovation programme NoMADS (Marie Skłodowska-Curie grant agreement No 777826). SP acknowledges the Leverhulme Trust Research Project Grant “Unveiling the invisible: Mathematics for Conservation in Arts and Humanities”. CB and PV also acknowledge partial support by MICINN/FEDER UE project, ref. PGC2018-098625-B-I00, and RED2018-102511-T. AB also acknowledges the French Research Agency through the PostProdLEAP project (ANR-19-CE23-0027-01). SH acknowledges the French ministry of research through a CDSN grant of ENS Paris-Saclay
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Semantic image completion through an adversarial strategy

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISIGRAPP 2019: Computer Vision, Imaging and Computer Graphics Theory and Applications. 14th International Joint Conference, celebrat del 25 al 27 de febrer de 2019 a Praga, República Txeca., Image completion or image inpainting is the task of filling in missing regions of an image. When those areas are large and the missing information is unique such that the information and redundancy available in the image is not useful to guide the completion, the task becomes even more challenging. This paper proposes an automatic semantic inpainting method able to reconstruct corrupted information of an image by semantically interpreting the image itself. It is based on an adversarial strategy followed by an energy-based completion algorithm. First, the data latent space is learned by training a modified Wasserstein generative adversarial network. Second, the learned semantic information is combined with a novel optimization loss able to recover missing regions conditioned by the available information. Moreover, we present an application in the context of face inpainting, where our method is used to generate a new face by integrating desired facial attributes or expressions from a reference face. This is achieved by slightly modifying the objective energy. Quantitative and qualitative top-tier results show the power and realism of the presented method., The authors acknowledge partial support by MICINN/FEDER UE project, reference PGC2018-098625-B-I00 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS. We also thank NVIDIA for the Quadro P6000 GPU donation.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Conditioned source separation for musical instrument performances

  • Slizovskaia, Olga
  • Haro Ortega, Gloria
  • Gómez Gutiérrez, Emilia, 1975-
In music source separation, the number of sources may vary for each piece and some of the sources may belong to the same family of instruments, thus sharing timbral characteristics and making the sources more correlated. This leads to additional challenges in the source separation problem. This paper proposes a source separation method for multiple musical instruments sounding simultaneously and explores how much additional information apart from the audio stream can lift the quality of source separation. We explore conditioning techniques at different levels of a primary source separation network and utilize two extra modalities of data, namely presence or absence of instruments in the mixture, and the corresponding video stream data., This work was funded in part by ERC Innovation Programme (grant 770376, TROMPA); Spanish Ministry of Economy and Competitiveness under the Mar´ıa de Maeztu Units of Excellence Program (MDM-2015-0502) and the Social European Funds; the MICINN/FEDER UE project with reference PGC2018-098625-B-I00; and the H2020-MSCARISE-2017 project with reference 777826 NoMADS. We gratefully acknowledge NVIDIA for the donation of GPUs used for the experiments.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

An analysis and speedup of the FALDOI method for Optical Flow estimation

  • Gamonal, Ferran P.
  • Ballester, Coloma
  • Haro Ortega, Gloria
  • Meinhardt Llopis, Enric
  • Palomares, Roberto P.
We present a detailed analysis of FALDOI, a large displacement optical flow method proposed by P. Palomares et al. This method requires a set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from the matches. It follows a two-step minimization method at the finest scale which is very robust to the outliers of the sparse matcher and can capture large displacements of small objects. The results shown in the original paper consistently outperformed the coarse-to-fine approaches and achieved good qualitative and quantitative performance on the standard optical flow benchmarks. In this paper we revise the proposed method and the changes done to significantly reduce its execution time while reporting nearly the same accuracy. Finally, we also compare it against the current state-of-the-art to assess its performance., The authors acknowledge support by TIN2015-70410-C2-1-R(MINECO/FEDER, UE) project and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

On anisotropic optical flow inpainting algorithms

  • Raad, Lara
  • Oliver Parera, Maria
  • Ballester, Coloma
  • Haro Ortega, Gloria
  • Meinhardt Llopis, Enric
This work describes two anisotropic optical flow inpainting algorithms. The first one recovers the missing flow values using the Absolutely Minimizing Lipschitz Extension partial differential equation (also called infinity Laplacian equation) and the second one uses the Laplace partial differential equation, both defined on a Riemmanian manifold. The Riemannian manifold is defined by endowing the plane domain with an appropriate metric depending on the reference video frame. A detailed analysis of both approaches is provided and their results are compared on three different applications: flow densification, occlusion inpainting and large hole inpainting.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

VoViT: low latency graph-based audio-visual voice separation transformer

  • Montesinos García, Juan Felipe
  • Kadandale, Venkatesh S.
  • Haro Ortega, Gloria
This paper presents an audio-visual approach for voice separation which produces state-of-the-art results at a low latency in two scenarios: speech and singing voice. The model is based on a two-stage network. Motion cues are obtained with a lightweight graph convolutional network that processes face landmarks. Then, both audio and motion features are fed to an audio-visual transformer which produces a fairly good estimation of the isolated target source. In a second stage, the predominant voice is enhanced with an audio-only network. We present different ablation studies and comparison to state-of-the-art methods. Finally, we explore the transferability of models trained for speech separation in the task of singing voice separation. The demos, code, and weights are available in https://ipcv.github.io/VoViT/., We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. J.F.M. acknowledges support by FPI scholarship PRE2018-083920. We acknowledge NVIDIA Corporation for the donation of GPUs used for the experiments.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Automatic flare spot artifacts detection and removal in photographs

  • Vitoria, Patricia
  • Ballester, Coloma
Flare spot is one type of are artifact caused by a number of conditions, frequently provoked by one or more high-luminance sources within or close to the camera field of view. When light rays coming from a high-luminance source reach the front element of a camera, it can produce intra-reections within camera elements that emerge at the film plane forming non-image information or are on the captured image. Even though preventive mechanisms are used, artifacts can appear. In this paper, we propose a robust computational method to automatically detect and remove are spot artifacts. Our contribution is threefold: firstly, we propose a characterization which is based on intrinsic properties that a are spot is likely to satisfy; secondly, we define a new confidence measure able to select are spots among the candidates; and, finally, a method to accurately determine the are region is given. Then, the detected artifacts are removed by using exemplar-based inpainting. We show that our algorithm achieve top-tier quantitative and qualitative performance., The authors acknowledge partial support by MINECO/FEDER UE project, with reference TIN2015-70410-C2-1-R, and by H2020-MSCA-RISE-2017 project with reference 777826 NoMADS.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Riemannian geometry for efficient analysis of protein dynamics data

  • Diepeveen, Willem
  • Esteve-Yagüe, Carlos
  • Lellmann, Jan
  • Öktem, Ozan
  • Schönlieb, Carola-Bibiane
  • Esteve-Yagüe, Carlos|||/items/48d9dfa4-ae70-4371-ac70-d37cada79962
An increasingly common viewpoint is that protein dynamics datasets reside in a nonlinear subspace of low conformational energy. Ideal data analysis tools should therefore account for such nonlinear geometry. The Riemannian geometry setting can be suitable for a variety of reasons. First, it comes with a rich mathematical structure to account for a wide range of geometries that can be modeled after an energy landscape. Second, many standard data analysis tools developed for data in Euclidean space can be generalized to Riemannian manifolds. In the context of protein dynamics, a conceptual challenge comes from the lack of guidelines for constructing a smooth Riemannian structure based on an energy landscape. In addition, computational feasibility in computing geodesics and related mappings poses a major challenge. This work considers these challenges. The first part of the paper develops a local approximation technique for computing geodesics and related mappings on Riemannian manifolds in a computationally feasible manner. The second part constructs a smooth manifold and a Riemannian structure that is based on an energy landscape for protein conformations. The resulting Riemannian geometry is tested on several data analysis tasks relevant for protein dynamics data. In particular, the geodesics with given start- and end-points approximately recover corresponding molecular dynamics trajectories for proteins that undergo relatively ordered transitions with medium-sized deformations. The Riemannian protein geometry also gives physically realistic summary statistics and retrieves the underlying dimension even for large-sized deformations within seconds on a laptop.
RUA. Repositorio Institucional de la Universidad de Alicante
Publicación Otros (Other)

Spectral decomposition of atomic structures in heterogeneous cryo-EM

  • Esteve-Yagüe, Carlos
  • Diepeveen, Willem
  • Öktem, Ozan
  • Schönlieb, Carola-Bibiane
  • Esteve-Yagüe, Carlos|||/items/48d9dfa4-ae70-4371-ac70-d37cada79962
We consider the problem of recovering the three-dimensional atomic structure of a flexible macromolecule from a heterogeneous cryogenic electron microscopy (cryo-EM) dataset. The dataset contains noisy tomographic projections of the electrostatic potential of the macromolecule, taken from different viewing directions, and in the heterogeneous case, each cryo-EM image corresponds to a different conformation of the macromolecule. Under the assumption that the macromolecule can be modelled as a chain, or discrete curve (as it is for instance the case for a protein backbone with a single chain of amino-acids), we introduce a method to estimate the deformation of the atomic model with respect to a given conformation, which is assumed to be known a priori. Our method consists on estimating the torsion and bond angles of the atomic model in each conformation as a linear combination of the eigenfunctions of the Laplace operator in the manifold of conformations. These eigenfunctions can be approximated by means of a well-known technique in manifold learning, based on the construction of a graph Laplacian using the cryo-EM dataset. Finally, we test our approach with synthetic datasets, for which we recover the atomic model of two-dimensional and three-dimensional flexible structures from simulated cryo-EM images.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

On anisotropic optical flow inpainting algorithms

  • Raad, Lara
  • Oliver Parera, Maria
  • Ballester, Coloma
  • Haro Ortega, Gloria
  • Meinhardt Llopis, Enric
This work describes two anisotropic optical flow inpainting algorithms. The first one recovers the missing flow values using the Absolutely Minimizing Lipschitz Extension partial differential equation (also called infinity Laplacian equation) and the second one uses the Laplace partial differential equation, both defined on a Riemmanian manifold. The Riemannian manifold is defined by endowing the plane domain with an appropriate metric depending on the reference video frame. A detailed analysis of both approaches is provided and their results are compared on three different applications: flow densification, occlusion inpainting and large hole inpainting.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Self-supervised small soccer player detection and tracking

  • Hurault, Samuel
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al MMSports '20: The 3rd International Workshop on Multimedia Content Analysis in Sports, celebrat el 16 d'octubre de 2020 de manera virtual., In a soccer game, the information provided by detecting and tracking brings crucial clues to further analyze and understand some tactical aspects of the game, including individual and team actions. State-of-the-art tracking algorithms achieve impressive results in scenarios on which they have been trained for, but they fail in challenging ones such as soccer games. This is frequently due to the player small relative size and the similar appearance among players of the same team. Although a straightforward solution would be to retrain these models by using a more specific dataset, the lack of such publicly available annotated datasets entails searching for other effective solutions. In this work, we propose a self-supervised pipeline which is able to detect and track low-resolution soccer players under different recording conditions without any need of ground-truth data. Extensive quantitative and qualitative experimental results are presented evaluating its performance. We also present a comparison to several state-of-the-art methods showing that both the proposed detector and the proposed tracker achieve top-tier results, in particular in the presence of small players., SH acknowledges support by ENS Paris-Saclay. CB and GH acknowledge support by MICINN/FEDER UE project, ref. PGC2018-098625- B-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and RED2018-102511-T.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

A graph-based method for soccer action spotting using unsupervised player classification

  • Cartas, Alejandro
  • Ballester, Coloma
  • Haro Ortega, Gloria
Action spotting in soccer videos is the task of identifying the specific time when a certain key action of the game occurs. Lately, it has received a large amount of attention and powerful methods have been introduced. Action spotting involves understanding the dynamics of the game, the complexity of events, and the variation of video sequences. Most approaches have focused on the latter, given that their models exploit the global visual features of the sequences. In this work, we focus on the former by (a) identifying and representing the players, referees, and goalkeepers as nodes in a graph, and by (b) modeling their temporal interactions as sequences of graphs. For the player identification, or player classification task, we obtain an accuracy of 97.72% in our annotated benchmark. For the action spotting task, our method obtains an overall performance of 57.83% average-mAP by combining it with other audiovisual modalities. This performance surpasses similar graph-based methods and has competitive results with heavy computing methods. Code and data are available at https://github.com/IPCV/soccer_action_spotting., The authors acknowledge support by MICINN/FEDER UE project, ref. PID2021-127643NB-I00, H2020-MSCA-RISE-2017 project, ref. 777826 NoMADS, and ReAViPeRo network, ref. RED2018-102511-T.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

VocaLiST: an audio-visual synchronisation model for lips and voices

  • Kadandale, Venkatesh S.
  • Montesinos García, Juan Felipe
  • Haro Ortega, Gloria
Comunicació presentada a Interspeech 2022, celebrat del 18 al 22 de setembre de 2022 a Inchon, Corea del Sud., In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audiovisual correspondence score. We propose an audio-visual crossmodal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the singing voice. The singing voice is a more challenging use case for synchronisation due to sustained vowel sounds. We also investigate the relevance of lip synchronisation models trained on speech datasets in the context of singing voice. Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end. The demos, source code, and the pre-trained models are available on https://ipcv.github.io/VocaLiST/, We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. V. S. K. has received support through “la Caixa” Foundation (ID 100010434), fellowship code: LCF/BQ/DI18/11660064 and the Marie SkłodowskaCurie grant agreement No. 713673. J. F. M. acknowledges support by FPI scholarship PRE2018-083920.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Semantic image completion through an adversarial strategy

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISIGRAPP 2019: Computer Vision, Imaging and Computer Graphics Theory and Applications. 14th International Joint Conference, celebrat del 25 al 27 de febrer de 2019 a Praga, República Txeca., Image completion or image inpainting is the task of filling in missing regions of an image. When those areas are large and the missing information is unique such that the information and redundancy available in the image is not useful to guide the completion, the task becomes even more challenging. This paper proposes an automatic semantic inpainting method able to reconstruct corrupted information of an image by semantically interpreting the image itself. It is based on an adversarial strategy followed by an energy-based completion algorithm. First, the data latent space is learned by training a modified Wasserstein generative adversarial network. Second, the learned semantic information is combined with a novel optimization loss able to recover missing regions conditioned by the available information. Moreover, we present an application in the context of face inpainting, where our method is used to generate a new face by integrating desired facial attributes or expressions from a reference face. This is achieved by slightly modifying the objective energy. Quantitative and qualitative top-tier results show the power and realism of the presented method., The authors acknowledge partial support by MICINN/FEDER UE project, reference PGC2018-098625-B-I00 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS. We also thank NVIDIA for the Quadro P6000 GPU donation.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Semantic image inpainting through improved Wasserstein generative adversarial networks

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISAPP 2019, celebrada a Praga, República Txeca, del 25 al 27 de febrer de 2019, Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the missing content conditioned by the available data. It takes into account powerful contextual and perceptual content inherent in the image itself. The benefits include the ability to recover large regions by accumulating semantic information even it is not fully present in the damaged image. Experiments show that the presented method obtains qualitative and quantitative top-tier results in different experimental situations and also achieves accurate photo-realism comparable to state-of-the-art works., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015-70410-C2-1 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

An analysis and speedup of the FALDOI method for Optical Flow estimation

  • Gamonal, Ferran P.
  • Ballester, Coloma
  • Haro Ortega, Gloria
  • Meinhardt Llopis, Enric
  • Palomares, Roberto P.
We present a detailed analysis of FALDOI, a large displacement optical flow method proposed by P. Palomares et al. This method requires a set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from the matches. It follows a two-step minimization method at the finest scale which is very robust to the outliers of the sparse matcher and can capture large displacements of small objects. The results shown in the original paper consistently outperformed the coarse-to-fine approaches and achieved good qualitative and quantitative performance on the standard optical flow benchmarks. In this paper we revise the proposed method and the changes done to significantly reduce its execution time while reporting nearly the same accuracy. Finally, we also compare it against the current state-of-the-art to assess its performance., The authors acknowledge support by TIN2015-70410-C2-1-R(MINECO/FEDER, UE) project and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

An analysis of generative methods for multiple image inpainting

  • Ballester, Coloma
  • Bugeau, Aurélie
  • Hurault, Samuel
  • Parisotto, Simone
  • Vitoria, Patricia
Image inpainting refers to the restoration of an image with missing regions in a way that is not detectable by the observer. The inpainting regions can be of any size and shape. This is an ill-posed inverse problem that does not have a unique solution. In this work, we focus on learning-based image completion methods for multiple and diverse inpainting which goal is to provide a set of distinct solutions for a given damaged image. These methods capitalize on the probabilistic nature of certain deep generative models to sample various solutions that coherently restore the missing content. Throughout the chapter, we will analyze the underlying theory and analyze the recent proposals for multiple inpainting. To investigate the pros and cons of each method, we present quantitative and qualitative comparisons, on common datasets, regarding both the quality and the diversity of the set of inpainted solutions. Our analysis allows us to identify the most successful generative strategies in both inpainting quality and inpainting diversity. This task is closely related to the learning of an accurate probability distribution of images. Depending on the dataset in use, the challenges that entail the training of such a model will be discussed through the analysis., PV, CB and AB acknowledge the EU Horizon 2020 research and innovation programme NoMADS (Marie Skłodowska-Curie grant agreement No 777826). SP acknowledges the Leverhulme Trust Research Project Grant “Unveiling the invisible: Mathematics for Conservation in Arts and Humanities”. CB and PV also acknowledge partial support by MICINN/FEDER UE project, ref. PGC2018-098625-B-I00, and RED2018-102511-T. AB also acknowledges the French Research Agency through the PostProdLEAP project (ANR-19-CE23-0027-01). SH acknowledges the French ministry of research through a CDSN grant of ENS Paris-Saclay
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

An affine invariant patch similarity

  • Fedorov, Vadim
  • Ballester, Coloma
Image and video comparison is often approached by comparing patches of visual information. In this work we present a detailed description and implementation of an affine invariant patch simi- larity measure that performs an appropriate patch comparison by automatically and intrinsically adapting the size and shape of the patches. We also describe the complete implementation of the proposed iterative algorithm for computation of those shape-adaptive patches around each point in the image domain., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015- 70410-C2-1-R, by GRC reference 2014 SGR 1301 Generalitat de Catalunya, and by H2020-MSCARISE-2017 project, reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

VoViT: low latency graph-based audio-visual voice separation transformer

  • Montesinos García, Juan Felipe
  • Kadandale, Venkatesh S.
  • Haro Ortega, Gloria
This paper presents an audio-visual approach for voice separation which produces state-of-the-art results at a low latency in two scenarios: speech and singing voice. The model is based on a two-stage network. Motion cues are obtained with a lightweight graph convolutional network that processes face landmarks. Then, both audio and motion features are fed to an audio-visual transformer which produces a fairly good estimation of the isolated target source. In a second stage, the predominant voice is enhanced with an audio-only network. We present different ablation studies and comparison to state-of-the-art methods. Finally, we explore the transferability of models trained for speech separation in the task of singing voice separation. The demos, code, and weights are available in https://ipcv.github.io/VoViT/., We acknowledge support by MICINN/FEDER UE project PID2021-127643NB-I00; H2020-MSCA-RISE-2017 project 777826 NoMADS. J.F.M. acknowledges support by FPI scholarship PRE2018-083920. We acknowledge NVIDIA Corporation for the donation of GPUs used for the experiments.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Conditioned source separation for musical instrument performances

  • Slizovskaia, Olga
  • Haro Ortega, Gloria
  • Gómez Gutiérrez, Emilia, 1975-
In music source separation, the number of sources may vary for each piece and some of the sources may belong to the same family of instruments, thus sharing timbral characteristics and making the sources more correlated. This leads to additional challenges in the source separation problem. This paper proposes a source separation method for multiple musical instruments sounding simultaneously and explores how much additional information apart from the audio stream can lift the quality of source separation. We explore conditioning techniques at different levels of a primary source separation network and utilize two extra modalities of data, namely presence or absence of instruments in the mixture, and the corresponding video stream data., This work was funded in part by ERC Innovation Programme (grant 770376, TROMPA); Spanish Ministry of Economy and Competitiveness under the Mar´ıa de Maeztu Units of Excellence Program (MDM-2015-0502) and the Social European Funds; the MICINN/FEDER UE project with reference PGC2018-098625-B-I00; and the H2020-MSCARISE-2017 project with reference 777826 NoMADS. We gratefully acknowledge NVIDIA for the donation of GPUs used for the experiments.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Towards soccer pass feasibility maps: the role of players’ orientation

  • Arbués Sangüesa, Adrià
  • Martin, Adrian
  • Fernández, Javier
  • Haro Ortega, Gloria
  • Ballester, Coloma
Once player tracking has been established as one of the main data sources in soccer, many challenges have emerged for data scientists, who attempt to recognize patterns from 2D trajectories in order to build tools that might help coaches to improve the performance of their teams. For instance, pass models predict where the ball should go next during pass events. However, existing models are mainly fed with players’ location and prior data, hence omitting critical pieces of information such as players’ body orientation. This paper presents a computational model to obtain pass feasibility maps, where player orientation is exploited and analysed. As a matter of fact, orientation proves to be crucial when modelling field-of-view and correct positioning of players, since it limits the potential receiving area of all candidates. Different proposals are given to evaluate the proposed pass feasibility map, reaching 0.46 and 0.79 in Top1 and Top3 accuracy, respectively, with a + 0.2 boost obtained after merging positional data with orientation., This work was supported by MICINN / FEDER UE (Funded by Ministerio de Ciencia, Innovación y Universidades) under Grants PGC2018-098625-B-I00 and FJCI-2017-31758, European Comission, HORIZON 2020 under grant 777826 NoMADS, and Ministerio de Ciencia, Innovación y Universidades (Redes Temáticas de Investigación) under grant RED2018-102511-T.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Automatic flare spot artifacts detection and removal in photographs

  • Vitoria, Patricia
  • Ballester, Coloma
Flare spot is one type of are artifact caused by a number of conditions, frequently provoked by one or more high-luminance sources within or close to the camera field of view. When light rays coming from a high-luminance source reach the front element of a camera, it can produce intra-reections within camera elements that emerge at the film plane forming non-image information or are on the captured image. Even though preventive mechanisms are used, artifacts can appear. In this paper, we propose a robust computational method to automatically detect and remove are spot artifacts. Our contribution is threefold: firstly, we propose a characterization which is based on intrinsic properties that a are spot is likely to satisfy; secondly, we define a new confidence measure able to select are spots among the candidates; and, finally, a method to accurately determine the are region is given. Then, the detected artifacts are removed by using exemplar-based inpainting. We show that our algorithm achieve top-tier quantitative and qualitative performance., The authors acknowledge partial support by MINECO/FEDER UE project, with reference TIN2015-70410-C2-1-R, and by H2020-MSCA-RISE-2017 project with reference 777826 NoMADS.