Pasar al contenido principal

APRENDIZAJE PROFUNDO MULTIMODAL ESTRUCTURADO

RTI2018-102285-A-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos I+D+i Retos de Investigación
Año convocatoria 2018
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario CENTRO DE VISION POR COMPUTADOR

Resultados relacionados

Resultados totales (Incluyendo duplicados): 7
Encontrada(s) 1 página(s)

Dipòsit Digital de Documents de la UAB
Publicación Capítulo o parte de un libro (BookPart)

Slimmable Compressive Autoencoders for Practical Neural Image Compression

  • Yang, Fei|||0000-0003-4099-6511
  • Herranz, Luis|||0000-0002-7022-3395
  • Cheng, Yongmei
  • Mozerov, Mikhail|||0000-0002-1117-743X
Neural image compression leverages deep neural networks to outperform traditional image codecs in rate-distortion performance. However, the resulting models are also heavy, computationally demanding and generally optimized for a single rate, limiting their practical use. Focusing on practical image compression, we propose slimmable compressive autoencoders (SlimCAEs), where rate (R) and distortion (D) are jointly optimized for different capacities. Once trained, encoders and decoders can be executed at different capacities, leading to different rates and complexities. We show that a successful implementation of SlimCAEs requires suitable capacity-specific RD tradeoffs. Our experiments show that SlimCAEs are highly flexible models that provide excellent rate-distortion performance, variable rate, and dynamic adjustment of memory, computational cost and latency, thus addressing the main requirements of practical image compression.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Mix and Match Networks, Cross-Modal Alignment for Zero-Pair Image-to-Image Translation

  • Wang, Yaxing|||0000-0002-6055-7164
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
This paper addresses the problem of inferring unseen cross-modal image-to-image translations between multiple modalities. We assume that only some of the pairwise translations have been seen (i.e. trained) and infer the remaining unseen translations (where training pairs are not available). We propose mix and match networks, an approach where multiple encoders and decoders are aligned in such a way that the desired translation can be obtained by simply cascading the source encoder and the target decoder, even when they have not interacted during the training stage (i.e. unseen). The main challenge lies in the alignment of the latent representations at the bottlenecks of encoder-decoder pairs. We propose an architecture with several tools to encourage alignment, including autoencoders and robust side information and latent consistency losses. We show the benefits of our approach in terms of effectiveness and scalability compared with other pairwise image-to-image translation approaches. We also propose zero-pair cross-modal image translation, a challenging setting where the objective is inferring semantic segmentation from depth (and vice-versa) without explicit segmentation-depth pairs, and only from two (disjoint) segmentation-RGB and depth-RGB training sets. We observe that a certain part of the shared information between unseen modalities might not be reachable, so we further propose a variant that leverages pseudo-pairs which allows us to exploit this shared information between the unseen modalities.
Dipòsit Digital de Documents de la UAB
Publicación Capítulo o parte de un libro (BookPart)

MineGAN, Effective Knowledge Transfer from GANs to Target Domains with Few Images

  • Wang, Yaxing|||0000-0002-6055-7164
  • Gonzalez-Garcia, Abel
  • Berga, David
  • Herranz, Luis|||0000-0002-7022-3395
  • Khan, Fahad Shahbaz|||0000-0002-4263-3143
  • Weijer, Joost van de|||0000-0002-9656-9706
One of the attractive characteristics of deep neural networks is their ability to transfer knowledge obtained in one domain to other related domains. As a result, high-quality networks can be trained in domains with relatively little training data. This property has been extensively studied for discriminative networks but has received significantly less attention for generative models. Given the often enormous effort required to train GANs, both computationally as well as in the dataset collection, the re-use of pretrained GANs is a desirable objective. We propose a novel knowledge transfer method for generative models based on mining the knowledge that is most beneficial to a specific target domain, either from a single or multiple pretrained GANs. This is done using a miner network that identifies which part of the generative distribution of each pretrained GAN outputs samples closest to the target domain. Mining effectively steers GAN sampling towards suitable regions of the latent space, which facilitates the posterior finetuning and avoids pathologies of other methods such as mode collapse and lack of flexibility. We perform experiments on several complex datasets using various GAN architectures (BigGAN, Progressive GAN) and show that the proposed method, called MineGAN, effectively transfers knowledge to domains with few target images, outperforming existing methods. In addition, MineGAN can successfully transfer knowledge from multiple pretrained GANs. Our code is available at: \url{https://github.com/yaxingwang/MineGAN}.
RUC. Repositorio da Universidade da Coruña
Publicación Artículo científico (JournalArticle)

Variable Rate Deep Image Compression With Modulated Autoencoder

  • Yang, Fei
  • Herranz, Luis
  • Van de Weijer, Joost
  • Iglesias-Guitian, Jose A.
  • López, Antonio M.
  • Mozerov, Mikhail G.
This version of the paper has been accepted for publication. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., [Abstract]: Variable rate is a requirement for flexible and adaptable image and video compression. However, deep image compression methods (DIC) are optimized for a single fixed rate-distortion (R-D) tradeoff. While this can be addressed by training multiple models for different tradeoffs, the memory requirements increase proportionally to the number of models. Scaling the bottleneck representation of a shared autoencoder can provide variable rate compression with a single shared autoencoder. However, the R-D performance using this simple mechanism degrades in low bitrates, and also shrinks the effective range of bitrates. To address these limitations, we formulate the problem of variable R-D optimization for DIC, and propose modulated autoencoders (MAEs), where the representations of a shared autoencoder are adapted to the specific R-D tradeoff via a modulation network. Jointly training this modulated autoencoder and the modulation network provides an effective way to navigate the R-D operational curve. Our experiments show that the proposed method can achieve almost the same R-D performance of independent models with significantly fewer parameters, This work was supported in part by Audi Electronics Venture GmbH. The Computer Vision Center (CVC) was supported by the Generalitat de Catalunya CERCA Program and its ACCIO agency. The work of F. Yang was supported by the Chinese Scholarship Council under Grant 201706290127. The work of L. Herranz was supported by the Spanish project RTI2018-102285-A-I00. The work of J. van de Weijer was supported by the Spanish project TIN2016-79717-R. The work of A. M. López and J. A. I. Guitián was supported by the project TIN2017-88709-R (MINECO/AEI/FEDER, UE). The work of A. M. López was also supported by the ICREA Academia programme. The work of J. A. I. Guitián and L. Herranz was supported by the EU’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie Grant 665919
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Main product detection with graph networks for fashion

  • Yazici, Vacit Oguz|||0000-0002-8878-2929
  • Yu, L.
  • Ramisa, A.
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
Computer vision has established a foothold in the online fashion retail industry. Main product detection is a crucial step of vision-based fashion product feed parsing pipelines, focused on identifying the bounding boxes that contain the product being sold in the gallery of images of the product page. The current state-of-the-art approach does not leverage the relations between regions in the image, and treats images of the same product independently, therefore not fully exploiting visual and product contextual information. In this paper, we propose a model that incorporates Graph Convolutional Networks (GCN) that jointly represent all detected bounding boxes in the gallery as nodes. We show that the proposed method is better than the state-of-the-art, especially, when we consider the scenario where title-input is missing at inference time and for cross-dataset evaluation, our method outperforms previous approaches by a large margin.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Distributed learning and inference with compressed images

  • Katakol, Sudeep|||0000-0002-4421-9819
  • Elbarashy, Basem
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
Modern computer vision requires processing large amounts of data, both while training the model and/or during inference, once the model is deployed. Scenarios where images are captured and processed in physically separated locations are increasingly common (e.g. autonomous vehicles, cloud computing, smartphones). In addition, many devices suffer from limited resources to store or transmit data (e.g. storage space, channel capacity). In these scenarios, lossy image compression plays a crucial role to effectively increase the number of images collected under such constraints. However, lossy compression entails some undesired degradation of the data that may harm the performance of the downstream analysis task at hand, since important semantic information may be lost in the process. Moreover, we may only have compressed images at training time but are able to use original images at inference time (i.e. test), or vice versa, and in such a case, the downstream model suffers from covariate shift. In this paper, we analyze this phenomenon, with a special focus on vision-based perception for autonomous driving as a paradigmatic scenario. We see that loss of semantic information and covariate shift do indeed exist, resulting in a drop in performance that depends on the compression rate. In order to address the problem, we propose dataset restoration, based on image restoration with generative adversarial networks (GANs). Our method is agnostic to both the particular image compression method and the downstream task; and has the advantage of not adding additional cost to the deployed models, which is particularly important in resource-limited devices. The presented experiments focus on semantic segmentation as a challenging use case, cover a broad range of compression rates and diverse datasets, and show how our method is able to significantly alleviate the negative effects of compression on the downstream visual task.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Variable rate deep image compression with modulated autoencoder

  • Yang, Fei|||0000-0003-4099-6511
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • Iglesias-Guitian, Jose A.|||0000-0002-0817-1010
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Mozerov, Mikhail|||0000-0002-1117-743X
Variable rate is a requirement for flexible and adaptable image and video compression. However, deep image compression methods (DIC) are optimized for a single fixed rate-distortion (R-D) tradeoff. While this can be addressed by training multiple models for different tradeoffs, the memory requirements increase proportionally to the number of models. Scaling the bottleneck representation of a shared autoencoder can provide variable rate compression with a single shared autoencoder. However, the R-D performance using this simple mechanism degrades in low bitrates, and also shrinks the effective range of bitrates. To address these limitations, we formulate the problem of variable R-D optimization for DIC, and propose modulated autoencoders (MAEs), where the representations of a shared autoencoder are adapted to the specific R-D tradeoff via a modulation network. Jointly training this modulated autoencoder and the modulation network provides an effective way to navigate the R-D operational curve. Our experiments show that the proposed method can achieve almost the same R-D performance of independent models with significantly fewer parameters.