Skip to main content

MODELOS Y TECNICAS COMPUTACIONALES PARA EL ANALISIS Y PROCESAMIENTO DE VIDEO

TIN2015-70410-C2-1-R

•
Funding agency name Ministerio de Economía y Competitividad
Funding agency acronym MINECO
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos I+D+i Retos de Investigación
Call year 2015
State Plan 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Management unit Dirección General de Investigación Científica y Técnica
Beneficiary centre UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 30
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

An affine invariant patch similarity

  • Fedorov, Vadim
  • Ballester, Coloma
Image and video comparison is often approached by comparing patches of visual information. In this work we present a detailed description and implementation of an affine invariant patch simi- larity measure that performs an appropriate patch comparison by automatically and intrinsically adapting the size and shape of the patches. We also describe the complete implementation of the proposed iterative algorithm for computation of those shape-adaptive patches around each point in the image domain., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015- 70410-C2-1-R, by GRC reference 2014 SGR 1301 Generalitat de Catalunya, and by H2020-MSCARISE-2017 project, reference 777826 NoMADS.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

End-to-end sound source separation conditioned on instrument labels

  • Slizovskaia, Olga
  • Kim, Leo
  • Haro Ortega, Gloria
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing celebrat de 12 al 17 de maig de 2019 a Brighton, Regne Unit., Can we perform an end-to-end music source separation with a variable number of sources using a deep learning model? This paper presents an extension of the Wave-UNet [1] model which allows end-to-end monaural source separation with a non-fixed number of sources. Furthermore, we propose multiplicative conditioning with instrument labels at the bottleneck of the Wave-U-Net and show its effect on the separation results. This approach can be further extended to other types of conditioning such as audio-visual source separation and score-informed source separation., This work has received funding from the Maria de Maeztu Programme (MDM-2015-0502), ERC Innovation Programme (grant 770376, TROMPA), and MINECO/FEDER UE project (TIN2015-70410-C2-1-R).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Musical instrument recognition in user-generated videos using a multimodal convolutional neural network architecture

  • Slizovskaia, Olga
  • Gómez Gutiérrez, Emilia, 1975-
  • Haro Ortega, Gloria
Comunicació presentada a la International Conference on Multimedia Retrieval celebrada del 6 al 9 de juny de 2017 a Bucarest, Romania., This paper presents a method for recognizing musical instruments in user-generated videos. Musical instrument recognition from music signals is a well-known task in the music information retrieval (MIR) field, where current approaches rely on the analysis of the good-quality audio material. This work addresses a real-world scenario with several research challenges, i.e. the analysis of user-generated videos that are varied in terms of recording conditions and quality and may contain multiple instruments sounding simultaneously and background noise. Our approach does not only focus on the analysis of audio information, but we exploit the multimodal information embedded in the audio and visual domains. In order to do so, we develop a Convolutional Neural Network (CNN) architecture which combines learned representations from both modalities at a late fusion stage. Our approach is trained and evaluated on two large-scale video datasets: YouTube-8M and FCVID. The proposed architectures demonstrate state-of-the-art results in audio and video object recognition, provide additional robustness to missing modalities, and remains computationally cheap to train., is work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), the CASAS Spanish research project (TIN2015-70816-R), and project TIN2015-70410-C2- 1-R (MINECO/FEDER, UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X GPU used for this research.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Motion inpainting by an image-based geodesic AMLE method

  • Oliver, Maria
  • Raad, Lara
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al congrés 25th IEEE International Conference on Image Processing (ICIP) celebrat del 7 al 10 d'octubre de 2018 a Atenes, Grècia., This work presents an automatic method for optical flow inpainting. Given a video, each frame domain is endowed with a Riemannian metric based on the video pixel values. The missing optical flow is recovered by solving the Absolutely Minimizing Lipschitz Extension (AMLE) partial differential equation on the Riemannian manifold. An efficient numerical algorithm is proposed using eikonal operators for nonlinear elliptic partial differential equations on a finite graph. The choice of the metric is discussed and the method is applied to optical flow inpainting and sparse-to-dense optical flow estimation, achieving top-tier performance in terms of End-Point-Error (EPE)., This work is partially supported by MINECO/FEDER UE project, with reference TIN2015-70410-C2-1-R and by RISE project NoMADS.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Semantic image inpainting through improved Wasserstein generative adversarial networks

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISAPP 2019, celebrada a Praga, República Txeca, del 25 al 27 de febrer de 2019, Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the missing content conditioned by the available data. It takes into account powerful contextual and perceptual content inherent in the image itself. The benefits include the ability to recover large regions by accumulating semantic information even it is not fully present in the damaged image. Experiments show that the presented method obtains qualitative and quantitative top-tier results in different experimental situations and also achieves accurate photo-realism comparable to state-of-the-art works., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015-70410-C2-1 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

A multiclass anisotropic Mumford-Shah functional for segmentation of d-dimensional vectorial images

  • Garamendi, Juan Francisco
  • Schiavi, Emanuele
Comunicació presentada a la 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), celebrada del 27 de febrer l'1 de març de 2017 a Porto, Portugal., We present a general model for multi-class segmentation of multi-channel digital images. It is based on the minimization of an anisotropic version of the Mumford-Shah energy functional in the class of piecewise constant functions. In the framework of geometric measure theory we use the concept of common interphases between regions (classes) and the value of the jump discontinuities of the (weak) solution between adjacent regions in order to define a minimal partition energy functional. The resulting problem is non-smooth and non-convex. Non-smoothness is dealt with highlighting the relationship of the proposed model with the well known Rudin, Osher and Fatemi model for image denoising when piecewise constant solutions (i.e partitions) are considered. Non-convexity is tackled with an optimal threshold of the ROF solution which we which generalize to multi-channel images through a probabilistic clustering. The optimal solution is then computed with a fixed point iteration. The result ing algorithm is described and results are presented showing the successful application of the method to Light Field (LF) images., The authors acknowledge partial support by TIN2015-70410-C2-1-R (MINECO/FEDER, UE) and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Exemplar-based image inpainting using an affine invariant similarity measure

  • Fedorov, Vadim
  • Arias Martínez, Pablo
  • Facciolo Furlan, Gabriele
  • Ballester, Coloma
Comunicació presentada a 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2016), celebrat del 27 al 29 de febrer de 2016 a Roma, Itàlia., Patch-based approaches are used in state-of-the-art methods for image inpainting. This paper presents a new method for exemplar-based image inpainting using transformed patches. The transformation is determined for each patch in a fully automatic way from a surrounding texture content. We build upon a recent affine invariant patch similarity measure that performs an appropriate patch comparison by automatically adapting the size and shape of the patches. As a consequence, it intrinsically extends the set of available source patches to copy information from. We incorporate this measure into a variational formulation for inpainting and present a numerical algorithm for optimizing it. We show that our method can be applied to complete a perspectively distorted texture as well as to automatically inpaint one view of a scene using other view of the same scene as a source. We present experimental results both for gray and color images, and a comparison with some exemplar-based image inpainting methods., The first, second and fourth authors acknowledge partial support by the MINECO/FEDER project with reference TIN2015-70410-C2-1-R , the MICINN project with reference MTM2012- 30772, and by GRC reference 2014 SGR 1301, Generalitat de Catalunya. The second and third authors were partly founded by the Centre National dEtudes Spatiales (CNES, MISS Project), BPIFrance and R´egion Ile de France, in the framework of the FUI 18 Plein Phare project, the European Research Council (advanced grant Twelve Labours n246961), the Office of Naval research (ONR grant N00014-14-1-0023), and ANRDGA project ANR-12-ASTR-0035.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

An analysis and speedup of the FALDOI method for Optical Flow estimation

  • Gamonal, Ferran P.
  • Ballester, Coloma
  • Haro Ortega, Gloria
  • Meinhardt Llopis, Enric
  • Palomares, Roberto P.
We present a detailed analysis of FALDOI, a large displacement optical flow method proposed by P. Palomares et al. This method requires a set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from the matches. It follows a two-step minimization method at the finest scale which is very robust to the outliers of the sparse matcher and can capture large displacements of small objects. The results shown in the original paper consistently outperformed the coarse-to-fine approaches and achieved good qualitative and quantitative performance on the standard optical flow benchmarks. In this paper we revise the proposed method and the changes done to significantly reduce its execution time while reporting nearly the same accuracy. Finally, we also compare it against the current state-of-the-art to assess its performance., The authors acknowledge support by TIN2015-70410-C2-1-R(MINECO/FEDER, UE) project and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

From occlusion to global depth order, a monocular approach

  • Rezaeirowshan, Babak
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al 11th International Joint Conference (VISIGRAPP), celebrat del 27 al 29 de febrer de 2016 a Roma, Itàlia., Estimating 3D structure of the scene from a single image remains a challenging problem in computer vision. This paper proposes a novel approach to obtain a global depth order of objects by incorporating monocular perceptual cues such as T-junctions and object boundary convexity, which are local indicators of occlusions, together with physical cues, namely ground contact points. The proposed combination of these local cues complement each other and creates a more thorough partial depth order relationship. The different partial orders are then robustly aggregated using a Markov random chain approximation to obtain the most plausible global depth order. Experiments show that the proposed method excels in comparison to state of the art methods., The authors acknowledge partial support by the MINECO/FEDER project with reference TIN2015-70410-C2-1-R, the MICINN project with reference MTM2012-30772, and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Joint TV-L1 optical flow and occlusion estimation

  • Garamendi, Juan Francisco
  • Lazcano, Vanel
  • Ballester, Coloma
This document describes an implementation of the energy functional minimization proposed by Ballester, Garrido, Lazcano and Caselles for joint optical flow and occlusion estimation. The method build up from the ideas behind the TV-L1 approach introduced by Zach, Pock and Bischof in 2007 but incorporating information that allows to detect occlusions. This information is based on the divergence of the flow and the proposed energy favors the location of occlusions on regions where this divergence is negative. The implemented variational method uses three consecutive frames. The energy functional is composed of regularization terms using the total variation, a data term using the L1 norm, and a term dealing with the occlusions. In the present implementation, we solve the stationary system of partial differential equations arising from the dual minimization problem associated with the TV operator by a variation of the box relaxation numerical scheme proposed by Garamendi, Gaspar, Malpica and Schiavi. This makes the overall algorithm faster than previous implementations based on a gradient descent method., The first author was partially supported by the Ag`encia de Gestió d’Ajuts Universitaris i de Recerca de la Generalitat de Catalunya within the project PDJ-2014-00038. The first and third authors acknowledge partial support by the MINECO/FEDER project with reference TIN2015- 70410-C2-1-R and by the MICINN project with reference MTM2012-30772.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Affine non-local means image denoising

  • Fedorov, Vadim
  • Ballester, Coloma
This work presents an extension of the Non-Local Means denoising method, that effectively exploits the affine invariant self-similarities present in images of real scenes. Our method provides a better image denoising result by grounding on the fact that in many occasions similar patches exist in the image but have undergone a transformation. The proposal uses an affine invariant patch similarity measure that performs an appropriate patch comparison by automatically and intrinsically adapting the size and shape of the patches. As a result, more similar patches are found and appropriately used. We show that this image denoising method achieves top-tier performance in terms of PSNR, outperforming consistently the results of the regular Non-Local Means, and that it provides state-of-the-art qualitative results., The authors acknowledge partial support by MICINN project, reference MTM2012-30772, by TIN2015-70410-C2- 1-R (MINECO/FEDER, UE), and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

FALDOI: a new minimization strategy for large displacement variational optical flow

  • Palomares, Roberto P.
  • Meinhardt Llopis, Enric
  • Ballester, Coloma
  • Haro Ortega, Gloria
We propose a large displacement optical flow method that introduces a new strategy to compute a good local minimum of any optical flow energy functional. The method requires a given set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from those matches. In particular, the matches are used to guide a structured coordinate descent of the energy functional around these keypoints. It results in a two-step minimization method at the finest scale which is very robust to the inevitable outliers of the sparse matcher and able to capture large displacements of small objects. Its benefits over other variational methods that also rely on a set of sparse matches are its robustness against very few matches, high levels of noise, and outliers. We validate our proposal using several optical flow variational models. The results consistently outperform the coarse-to-fine approaches and achieve good qualitative and quantitative performance on the standard optical flow benchmarks., The first, second, and third authors acknowledge partial support by MICINN project, reference MTM2012-30772, by TIN2015-70410-C2-1-R (MINECO/FEDER, UE) and by GRC reference 2014 SGR 1301, Generalitat de Catalunya, and the fourth author by the European Research Council (advanced Grant Twelve Labours), Office of Naval research (ONR Grant N00014-14-1-0023).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Joint large displacement scene flow and occlusion variational estimation

  • Palomares, Roberto P.
  • Haro Ortega, Gloria
  • Ballester, Coloma
Comunicació presentada al congrés International Conference on Computer Vision Theory and Applications (VISSAP) celebrat a Porto (Portugal) del 27 de febrer a l'1 de març de 2017., This paper presents a novel variational approach for the joint estimation of scene flow and occlusions. Our method does not assume that a depth sensor is available. Instead, we use a stereo sequence and exploit the fact that points that are occluded in time, might be visible from the other view and thus the 3D geometry can be densely reinforced in an appropriate manner through a simultaneous motion occlusion characterization. Moreover, large displacements are correctly captured thanks to an optimization strategy that uses a set of sparse image correspondences to guide the minimization process. We include qualitative and quantitative experimental results on several datasets illustrating that both proposals help to improve the baseline results., The authors acknowledge partial support by TIN2015-70410-C2-1-R (MINECO/FEDER, UE) and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Automatic flare spot artifacts detection and removal in photographs

  • Vitoria, Patricia
  • Ballester, Coloma
Flare spot is one type of are artifact caused by a number of conditions, frequently provoked by one or more high-luminance sources within or close to the camera field of view. When light rays coming from a high-luminance source reach the front element of a camera, it can produce intra-reections within camera elements that emerge at the film plane forming non-image information or are on the captured image. Even though preventive mechanisms are used, artifacts can appear. In this paper, we propose a robust computational method to automatically detect and remove are spot artifacts. Our contribution is threefold: firstly, we propose a characterization which is based on intrinsic properties that a are spot is likely to satisfy; secondly, we define a new confidence measure able to select are spots among the candidates; and, finally, a method to accurately determine the are region is given. Then, the detected artifacts are removed by using exemplar-based inpainting. We show that our algorithm achieve top-tier quantitative and qualitative performance., The authors acknowledge partial support by MINECO/FEDER UE project, with reference TIN2015-70410-C2-1-R, and by H2020-MSCA-RISE-2017 project with reference 777826 NoMADS.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Visual music transcription of clarinet video recordings trained with audio-based labelled data

  • Gómez Gutiérrez, Emilia, 1975-
  • Arias Martínez, Pablo
  • Zinemanas, Pablo
  • Haro Ortega, Gloria
Comunicació presentada a la 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), celebrada els dies 22 a 29 d'octubre de 2017 a Venècia, Itàlia., Automatic transcription is a well-known task in the music information retrieval (MIR) domain, and consists on the computation of a symbolic music representation (e.g. MIDI) from an audio recording. In this work, we address the automatic transcription of video recordings when the audio modality is missing or it does not have enough quality, and thus analyze the visual information. We focus on the clarinet which is played by opening/closing a set of holes and keys. We propose a method for automatic visual note estimation by detecting the fingertips of the player and measuring their displacement with respect to the holes and keys of the clarinet. To this aim, we track the clarinet and determine its position on every frame. The relative positions of the fingertips are used as features of a machine learning algorithm trained for note pitch classification. For that purpose, a dataset is built in a semiautomatic way by estimating pitch information from audio signals in an existing collection of 4.5 hours of video recordings from six different songs performed by nine different players. Our results confirm the difficulty of performing visual vs audio automatic transcription mainly due to motion blur and occlusions that cannot be solved with a single view., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502); the Spanish research projects CASAS (TIN2015-70816-R, MINECO/FEDER, UE) and MOTECAVI (TIN2015-70410-C2-1-R, MINECO/FEDER, UE); BPIFrance and Région Île de France in the framework of the FUI 18 Plein Phare project; the Office of Naval research by grant N00014-17-1-2552, ANR-DGA project ANR-12-ASTR-0035; and ANR-14-CE27-001 (MIRIAM).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Joint TV-L1 optical flow and occlusion estimation

  • Garamendi, Juan Francisco
  • Lazcano, Vanel
  • Ballester, Coloma
This document describes an implementation of the energy functional minimization proposed by Ballester, Garrido, Lazcano and Caselles for joint optical flow and occlusion estimation. The method build up from the ideas behind the TV-L1 approach introduced by Zach, Pock and Bischof in 2007 but incorporating information that allows to detect occlusions. This information is based on the divergence of the flow and the proposed energy favors the location of occlusions on regions where this divergence is negative. The implemented variational method uses three consecutive frames. The energy functional is composed of regularization terms using the total variation, a data term using the L1 norm, and a term dealing with the occlusions. In the present implementation, we solve the stationary system of partial differential equations arising from the dual minimization problem associated with the TV operator by a variation of the box relaxation numerical scheme proposed by Garamendi, Gaspar, Malpica and Schiavi. This makes the overall algorithm faster than previous implementations based on a gradient descent method., The first author was partially supported by the Ag`encia de Gestió d’Ajuts Universitaris i de Recerca de la Generalitat de Catalunya within the project PDJ-2014-00038. The first and third authors acknowledge partial support by the MINECO/FEDER project with reference TIN2015- 70410-C2-1-R and by the MICINN project with reference MTM2012-30772.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

FALDOI: a new minimization strategy for large displacement variational optical flow

  • Palomares, Roberto P.
  • Meinhardt Llopis, Enric
  • Ballester, Coloma
  • Haro Ortega, Gloria
We propose a large displacement optical flow method that introduces a new strategy to compute a good local minimum of any optical flow energy functional. The method requires a given set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from those matches. In particular, the matches are used to guide a structured coordinate descent of the energy functional around these keypoints. It results in a two-step minimization method at the finest scale which is very robust to the inevitable outliers of the sparse matcher and able to capture large displacements of small objects. Its benefits over other variational methods that also rely on a set of sparse matches are its robustness against very few matches, high levels of noise, and outliers. We validate our proposal using several optical flow variational models. The results consistently outperform the coarse-to-fine approaches and achieve good qualitative and quantitative performance on the standard optical flow benchmarks., The first, second, and third authors acknowledge partial support by MICINN project, reference MTM2012-30772, by TIN2015-70410-C2-1-R (MINECO/FEDER, UE) and by GRC reference 2014 SGR 1301, Generalitat de Catalunya, and the fourth author by the European Research Council (advanced Grant Twelve Labours), Office of Naval research (ONR Grant N00014-14-1-0023).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Joint large displacement scene flow and occlusion variational estimation

  • Palomares, Roberto P.
  • Haro Ortega, Gloria
  • Ballester, Coloma
Comunicació presentada al congrés International Conference on Computer Vision Theory and Applications (VISSAP) celebrat a Porto (Portugal) del 27 de febrer a l'1 de març de 2017., This paper presents a novel variational approach for the joint estimation of scene flow and occlusions. Our method does not assume that a depth sensor is available. Instead, we use a stereo sequence and exploit the fact that points that are occluded in time, might be visible from the other view and thus the 3D geometry can be densely reinforced in an appropriate manner through a simultaneous motion occlusion characterization. Moreover, large displacements are correctly captured thanks to an optimization strategy that uses a set of sparse image correspondences to guide the minimization process. We include qualitative and quantitative experimental results on several datasets illustrating that both proposals help to improve the baseline results., The authors acknowledge partial support by TIN2015-70410-C2-1-R (MINECO/FEDER, UE) and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Exemplar-based image inpainting using an affine invariant similarity measure

  • Fedorov, Vadim
  • Arias Martínez, Pablo
  • Facciolo Furlan, Gabriele
  • Ballester, Coloma
Comunicació presentada a 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2016), celebrat del 27 al 29 de febrer de 2016 a Roma, Itàlia., Patch-based approaches are used in state-of-the-art methods for image inpainting. This paper presents a new method for exemplar-based image inpainting using transformed patches. The transformation is determined for each patch in a fully automatic way from a surrounding texture content. We build upon a recent affine invariant patch similarity measure that performs an appropriate patch comparison by automatically adapting the size and shape of the patches. As a consequence, it intrinsically extends the set of available source patches to copy information from. We incorporate this measure into a variational formulation for inpainting and present a numerical algorithm for optimizing it. We show that our method can be applied to complete a perspectively distorted texture as well as to automatically inpaint one view of a scene using other view of the same scene as a source. We present experimental results both for gray and color images, and a comparison with some exemplar-based image inpainting methods., The first, second and fourth authors acknowledge partial support by the MINECO/FEDER project with reference TIN2015-70410-C2-1-R , the MICINN project with reference MTM2012- 30772, and by GRC reference 2014 SGR 1301, Generalitat de Catalunya. The second and third authors were partly founded by the Centre National dEtudes Spatiales (CNES, MISS Project), BPIFrance and R´egion Ile de France, in the framework of the FUI 18 Plein Phare project, the European Research Council (advanced grant Twelve Labours n246961), the Office of Naval research (ONR grant N00014-14-1-0023), and ANRDGA project ANR-12-ASTR-0035.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A multiclass anisotropic Mumford-Shah functional for segmentation of d-dimensional vectorial images

  • Garamendi, Juan Francisco
  • Schiavi, Emanuele
Comunicació presentada a la 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), celebrada del 27 de febrer l'1 de març de 2017 a Porto, Portugal., We present a general model for multi-class segmentation of multi-channel digital images. It is based on the minimization of an anisotropic version of the Mumford-Shah energy functional in the class of piecewise constant functions. In the framework of geometric measure theory we use the concept of common interphases between regions (classes) and the value of the jump discontinuities of the (weak) solution between adjacent regions in order to define a minimal partition energy functional. The resulting problem is non-smooth and non-convex. Non-smoothness is dealt with highlighting the relationship of the proposed model with the well known Rudin, Osher and Fatemi model for image denoising when piecewise constant solutions (i.e partitions) are considered. Non-convexity is tackled with an optimal threshold of the ROF solution which we which generalize to multi-channel images through a probabilistic clustering. The optimal solution is then computed with a fixed point iteration. The result ing algorithm is described and results are presented showing the successful application of the method to Light Field (LF) images., The authors acknowledge partial support by TIN2015-70410-C2-1-R (MINECO/FEDER, UE) and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Semantic image inpainting through improved Wasserstein generative adversarial networks

  • Vitoria, Patricia
  • Sintes Marcos, Joan
  • Ballester, Coloma
Comunicació presentada a: VISAPP 2019, celebrada a Praga, República Txeca, del 25 al 27 de febrer de 2019, Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the missing content conditioned by the available data. It takes into account powerful contextual and perceptual content inherent in the image itself. The benefits include the ability to recover large regions by accumulating semantic information even it is not fully present in the damaged image. Experiments show that the presented method obtains qualitative and quantitative top-tier results in different experimental situations and also achieves accurate photo-realism comparable to state-of-the-art works., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015-70410-C2-1 and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Visual music transcription of clarinet video recordings trained with audio-based labelled data

  • Gómez Gutiérrez, Emilia, 1975-
  • Arias Martínez, Pablo
  • Zinemanas, Pablo
  • Haro Ortega, Gloria
Comunicació presentada a la 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), celebrada els dies 22 a 29 d'octubre de 2017 a Venècia, Itàlia., Automatic transcription is a well-known task in the music information retrieval (MIR) domain, and consists on the computation of a symbolic music representation (e.g. MIDI) from an audio recording. In this work, we address the automatic transcription of video recordings when the audio modality is missing or it does not have enough quality, and thus analyze the visual information. We focus on the clarinet which is played by opening/closing a set of holes and keys. We propose a method for automatic visual note estimation by detecting the fingertips of the player and measuring their displacement with respect to the holes and keys of the clarinet. To this aim, we track the clarinet and determine its position on every frame. The relative positions of the fingertips are used as features of a machine learning algorithm trained for note pitch classification. For that purpose, a dataset is built in a semiautomatic way by estimating pitch information from audio signals in an existing collection of 4.5 hours of video recordings from six different songs performed by nine different players. Our results confirm the difficulty of performing visual vs audio automatic transcription mainly due to motion blur and occlusions that cannot be solved with a single view., This work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502); the Spanish research projects CASAS (TIN2015-70816-R, MINECO/FEDER, UE) and MOTECAVI (TIN2015-70410-C2-1-R, MINECO/FEDER, UE); BPIFrance and Région Île de France in the framework of the FUI 18 Plein Phare project; the Office of Naval research by grant N00014-17-1-2552, ANR-DGA project ANR-12-ASTR-0035; and ANR-14-CE27-001 (MIRIAM).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

An analysis and speedup of the FALDOI method for Optical Flow estimation

  • Gamonal, Ferran P.
  • Ballester, Coloma
  • Haro Ortega, Gloria
  • Meinhardt Llopis, Enric
  • Palomares, Roberto P.
We present a detailed analysis of FALDOI, a large displacement optical flow method proposed by P. Palomares et al. This method requires a set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from the matches. It follows a two-step minimization method at the finest scale which is very robust to the outliers of the sparse matcher and can capture large displacements of small objects. The results shown in the original paper consistently outperformed the coarse-to-fine approaches and achieved good qualitative and quantitative performance on the standard optical flow benchmarks. In this paper we revise the proposed method and the changes done to significantly reduce its execution time while reporting nearly the same accuracy. Finally, we also compare it against the current state-of-the-art to assess its performance., The authors acknowledge support by TIN2015-70410-C2-1-R(MINECO/FEDER, UE) project and by H2020-MSCA-RISE-2017 project, reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

From occlusion to global depth order, a monocular approach

  • Rezaeirowshan, Babak
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al 11th International Joint Conference (VISIGRAPP), celebrat del 27 al 29 de febrer de 2016 a Roma, Itàlia., Estimating 3D structure of the scene from a single image remains a challenging problem in computer vision. This paper proposes a novel approach to obtain a global depth order of objects by incorporating monocular perceptual cues such as T-junctions and object boundary convexity, which are local indicators of occlusions, together with physical cues, namely ground contact points. The proposed combination of these local cues complement each other and creates a more thorough partial depth order relationship. The different partial orders are then robustly aggregated using a Markov random chain approximation to obtain the most plausible global depth order. Experiments show that the proposed method excels in comparison to state of the art methods., The authors acknowledge partial support by the MINECO/FEDER project with reference TIN2015-70410-C2-1-R, the MICINN project with reference MTM2012-30772, and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

End-to-end sound source separation conditioned on instrument labels

  • Slizovskaia, Olga
  • Kim, Leo
  • Haro Ortega, Gloria
  • Gómez Gutiérrez, Emilia, 1975-
Comunicació presentada a: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing celebrat de 12 al 17 de maig de 2019 a Brighton, Regne Unit., Can we perform an end-to-end music source separation with a variable number of sources using a deep learning model? This paper presents an extension of the Wave-UNet [1] model which allows end-to-end monaural source separation with a non-fixed number of sources. Furthermore, we propose multiplicative conditioning with instrument labels at the bottleneck of the Wave-U-Net and show its effect on the separation results. This approach can be further extended to other types of conditioning such as audio-visual source separation and score-informed source separation., This work has received funding from the Maria de Maeztu Programme (MDM-2015-0502), ERC Innovation Programme (grant 770376, TROMPA), and MINECO/FEDER UE project (TIN2015-70410-C2-1-R).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Affine non-local means image denoising

  • Fedorov, Vadim
  • Ballester, Coloma
This work presents an extension of the Non-Local Means denoising method, that effectively exploits the affine invariant self-similarities present in images of real scenes. Our method provides a better image denoising result by grounding on the fact that in many occasions similar patches exist in the image but have undergone a transformation. The proposal uses an affine invariant patch similarity measure that performs an appropriate patch comparison by automatically and intrinsically adapting the size and shape of the patches. As a result, more similar patches are found and appropriately used. We show that this image denoising method achieves top-tier performance in terms of PSNR, outperforming consistently the results of the regular Non-Local Means, and that it provides state-of-the-art qualitative results., The authors acknowledge partial support by MICINN project, reference MTM2012-30772, by TIN2015-70410-C2- 1-R (MINECO/FEDER, UE), and by GRC reference 2014 SGR 1301, Generalitat de Catalunya.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

An affine invariant patch similarity

  • Fedorov, Vadim
  • Ballester, Coloma
Image and video comparison is often approached by comparing patches of visual information. In this work we present a detailed description and implementation of an affine invariant patch simi- larity measure that performs an appropriate patch comparison by automatically and intrinsically adapting the size and shape of the patches. We also describe the complete implementation of the proposed iterative algorithm for computation of those shape-adaptive patches around each point in the image domain., The authors acknowledge partial support by MINECO/FEDER UE project, reference TIN2015- 70410-C2-1-R, by GRC reference 2014 SGR 1301 Generalitat de Catalunya, and by H2020-MSCARISE-2017 project, reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Musical instrument recognition in user-generated videos using a multimodal convolutional neural network architecture

  • Slizovskaia, Olga
  • Gómez Gutiérrez, Emilia, 1975-
  • Haro Ortega, Gloria
Comunicació presentada a la International Conference on Multimedia Retrieval celebrada del 6 al 9 de juny de 2017 a Bucarest, Romania., This paper presents a method for recognizing musical instruments in user-generated videos. Musical instrument recognition from music signals is a well-known task in the music information retrieval (MIR) field, where current approaches rely on the analysis of the good-quality audio material. This work addresses a real-world scenario with several research challenges, i.e. the analysis of user-generated videos that are varied in terms of recording conditions and quality and may contain multiple instruments sounding simultaneously and background noise. Our approach does not only focus on the analysis of audio information, but we exploit the multimodal information embedded in the audio and visual domains. In order to do so, we develop a Convolutional Neural Network (CNN) architecture which combines learned representations from both modalities at a late fusion stage. Our approach is trained and evaluated on two large-scale video datasets: YouTube-8M and FCVID. The proposed architectures demonstrate state-of-the-art results in audio and video object recognition, provide additional robustness to missing modalities, and remains computationally cheap to train., is work is partly supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502), the CASAS Spanish research project (TIN2015-70816-R), and project TIN2015-70410-C2- 1-R (MINECO/FEDER, UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X GPU used for this research.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Automatic flare spot artifacts detection and removal in photographs

  • Vitoria, Patricia
  • Ballester, Coloma
Flare spot is one type of are artifact caused by a number of conditions, frequently provoked by one or more high-luminance sources within or close to the camera field of view. When light rays coming from a high-luminance source reach the front element of a camera, it can produce intra-reections within camera elements that emerge at the film plane forming non-image information or are on the captured image. Even though preventive mechanisms are used, artifacts can appear. In this paper, we propose a robust computational method to automatically detect and remove are spot artifacts. Our contribution is threefold: firstly, we propose a characterization which is based on intrinsic properties that a are spot is likely to satisfy; secondly, we define a new confidence measure able to select are spots among the candidates; and, finally, a method to accurately determine the are region is given. Then, the detected artifacts are removed by using exemplar-based inpainting. We show that our algorithm achieve top-tier quantitative and qualitative performance., The authors acknowledge partial support by MINECO/FEDER UE project, with reference TIN2015-70410-C2-1-R, and by H2020-MSCA-RISE-2017 project with reference 777826 NoMADS.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Motion inpainting by an image-based geodesic AMLE method

  • Oliver, Maria
  • Raad, Lara
  • Ballester, Coloma
  • Haro Ortega, Gloria
Comunicació presentada al congrés 25th IEEE International Conference on Image Processing (ICIP) celebrat del 7 al 10 d'octubre de 2018 a Atenes, Grècia., This work presents an automatic method for optical flow inpainting. Given a video, each frame domain is endowed with a Riemannian metric based on the video pixel values. The missing optical flow is recovered by solving the Absolutely Minimizing Lipschitz Extension (AMLE) partial differential equation on the Riemannian manifold. An efficient numerical algorithm is proposed using eikonal operators for nonlinear elliptic partial differential equations on a finite graph. The choice of the metric is discussed and the method is applied to optical flow inpainting and sparse-to-dense optical flow estimation, achieving top-tier performance in terms of End-Point-Error (EPE)., This work is partially supported by MINECO/FEDER UE project, with reference TIN2015-70410-C2-1-R and by RISE project NoMADS.