Pasar al contenido principal

Multimodal deep learning applied to computer vision and multimedia

RYC2019-027020-I

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de Promoción del Talento y su Empleabilidad en I+D+i
Subprograma Subprograma Estatal de Incorporación
Convocatoria Ramón y Cajal
Año convocatoria 2019
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario CENTRO DE VISION POR COMPUTADOR

Resultados relacionados

Resultados totales (Incluyendo duplicados): 10
Encontrada(s) 1 página(s)

Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Trust Your Good Friends, Source-Free Domain Adaptation by Reciprocal Neighborhood Clustering

  • Yang, Shiqi|||0000-0002-4141-7190
  • Wang, Yaxing|||0000-0002-6055-7164
  • Weijer, Joost van de|||0000-0002-9656-9706
  • Herranz, Luis|||0000-0002-7022-3395
  • Jui, Shangling|||0000-0002-1047-4264
  • Yang, Jian|||0000-0003-4800-832X
Domain adaptation (DA) aims to alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g., due to data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain in the absence of source data. Our method is based on the observation that target data, which might not align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity. We observe that higher affinity should be assigned to reciprocal neighbors. To aggregate information with more context, we consider expanded neighborhoods with small affinity values. Furthermore, we consider the density around each target sample, which can alleviate the negative impact of potential outliers. In the experimental results we verify that the inherent structure of the target features is an important source of information for domain adaptation. We demonstrate that this local structure can be efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of-the-art performance on several 2D image and 3D point cloud recognition datasets.
Dipòsit Digital de Documents de la UAB
Publicación Capítulo o parte de un libro (BookPart)

Slimmable Compressive Autoencoders for Practical Neural Image Compression

  • Yang, Fei|||0000-0003-4099-6511
  • Herranz, Luis|||0000-0002-7022-3395
  • Cheng, Yongmei
  • Mozerov, Mikhail|||0000-0002-1117-743X
Neural image compression leverages deep neural networks to outperform traditional image codecs in rate-distortion performance. However, the resulting models are also heavy, computationally demanding and generally optimized for a single rate, limiting their practical use. Focusing on practical image compression, we propose slimmable compressive autoencoders (SlimCAEs), where rate (R) and distortion (D) are jointly optimized for different capacities. Once trained, encoders and decoders can be executed at different capacities, leading to different rates and complexities. We show that a successful implementation of SlimCAEs requires suitable capacity-specific RD tradeoffs. Our experiments show that SlimCAEs are highly flexible models that provide excellent rate-distortion performance, variable rate, and dynamic adjustment of memory, computational cost and latency, thus addressing the main requirements of practical image compression.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Casting a BAIT for offline and online source-free domain adaptation

  • Yang, Shiqi|||0000-0002-4141-7190
  • Wang, Yaxing|||0000-0002-6055-7164
  • Herranz, Luis|||0000-0002-7022-3395
  • Jui, Shangling|||0000-0002-1047-4264
  • van de Weijer, Joost
We address the source-free domain adaptation (SFDA) problem, where only the source model is available during adaptation to the target domain. We consider two settings: the offline setting where all target data can be visited multiple times (epochs) to arrive at a prediction for each target sample, and the online setting where the target data needs to be directly classified upon arrival. Inspired by diverse classifier based domain adaptation methods, in this paper we introduce a second classifier, but with another classifier head fixed. When adapting to the target domain, the additional classifier initialized from source classifier is expected to find misclassified features. Next, when updating the feature extractor, those features will be pushed towards the right side of the source decision boundary, thus achieving source-free domain adaptation. Experimental results show that the proposed method achieves competitive results for offline SFDA on several benchmark datasets compared with existing DA and SFDA methods, and our method surpasses by a large margin other SFDA methods under online source-free domain adaptation setting.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Leveraging Color Naming for Image Enhancement

  • Serrano Lozano, David|||0009-0009-1224-8453
  • Herranz, Luis|||0000-0002-7022-3395
  • Brown, Michael S.|||0000-0002-9840-0795
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
Enhancing images to make them visually appealing is a persistent challenge in computer vision. Many deep-learning methods train models on paired datasets to replicate expert editing styles. However, these approaches struggle with two key issues: (1) interpretability and (2) a parametrization suitable for user adjustments. To address these challenges, we present NamedCurves+, an approach inspired by the concept of Color Naming, a universal set of familiar colors widely used in software tools for intuitive editing. Our method integrates color names into a learning-based framework, enabling global adjustments for each named color through tone curves. To address local image variations, we incorporate a transformer block that captures spatial dependencies, enabling context-aware edits across the image. NamedCurves+ enhances the retouching process's interpretability and supports user interaction, allowing flexible modifications of individual tone curves to refine the retouched image according to personal preferences. Extensive experiments on tasks such as image retouching, tone mapping, and exposure correction demonstrate that NamedCurves+ outperforms state-of-the-art methods. Notably, our approach is both explainable, as the tone curves explicitly represent how each color name contributes to the enhancement, and interactive, allowing users to customize the retouching process and achieve results tailored to their liking. Source code and models will be publicly available at: https://namedcurves.github.io.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Main product detection with graph networks for fashion

  • Yazici, Vacit Oguz|||0000-0002-8878-2929
  • Yu, L.
  • Ramisa, A.
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
Computer vision has established a foothold in the online fashion retail industry. Main product detection is a crucial step of vision-based fashion product feed parsing pipelines, focused on identifying the bounding boxes that contain the product being sold in the gallery of images of the product page. The current state-of-the-art approach does not leverage the relations between regions in the image, and treats images of the same product independently, therefore not fully exploiting visual and product contextual information. In this paper, we propose a model that incorporates Graph Convolutional Networks (GCN) that jointly represent all detected bounding boxes in the gallery as nodes. We show that the proposed method is better than the state-of-the-art, especially, when we consider the scenario where title-input is missing at inference time and for cross-dataset evaluation, our method outperforms previous approaches by a large margin.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Distributed learning and inference with compressed images

  • Katakol, Sudeep|||0000-0002-4421-9819
  • Elbarashy, Basem
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
Modern computer vision requires processing large amounts of data, both while training the model and/or during inference, once the model is deployed. Scenarios where images are captured and processed in physically separated locations are increasingly common (e.g. autonomous vehicles, cloud computing, smartphones). In addition, many devices suffer from limited resources to store or transmit data (e.g. storage space, channel capacity). In these scenarios, lossy image compression plays a crucial role to effectively increase the number of images collected under such constraints. However, lossy compression entails some undesired degradation of the data that may harm the performance of the downstream analysis task at hand, since important semantic information may be lost in the process. Moreover, we may only have compressed images at training time but are able to use original images at inference time (i.e. test), or vice versa, and in such a case, the downstream model suffers from covariate shift. In this paper, we analyze this phenomenon, with a special focus on vision-based perception for autonomous driving as a paradigmatic scenario. We see that loss of semantic information and covariate shift do indeed exist, resulting in a drop in performance that depends on the compression rate. In order to address the problem, we propose dataset restoration, based on image restoration with generative adversarial networks (GANs). Our method is agnostic to both the particular image compression method and the downstream task; and has the advantage of not adding additional cost to the deployed models, which is particularly important in resource-limited devices. The presented experiments focus on semantic segmentation as a challenging use case, cover a broad range of compression rates and diverse datasets, and show how our method is able to significantly alleviate the negative effects of compression on the downstream visual task.
Dipòsit Digital de Documents de la UAB
Publicación Capítulo o parte de un libro (BookPart)

Burst perception-distortion tradeoff, analysis and evaluation

  • Xue, Danna|||0000-0002-4280-706X
  • Herranz, Luis|||0000-0002-7022-3395
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Zhang, Yanning|||0000-0002-2977-8057
Burst image restoration attempts to effectively utilize the complementary cues appearing in sequential images to produce a high-quality image. Most current methods use all the available images to obtain the reconstructed image. However, using more images for burst restoration is not always the best option regarding reconstruction quality and efficiency, as the images acquired by handheld imaging devices suffer from degradation and misalignment caused by the camera noise and shake. In this paper, we extend the perception-distortion tradeoff theory by introducing multiple-frame information. We propose the area of the unattainable region as a new metric for perception-distortion tradeoff evaluation and comparison. Based on this metric, we analyse the performance of burst restoration from the perspective of the perception-distortion tradeoff under both aligned bursts and misaligned bursts situations. Our analysis reveals the importance of inter-frame alignment for burst restoration and shows that the optimal burst length for the restoration model depends both on the degree of degradation and misalignment.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Improving the perception of low-light enhanced images

  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Finlayson, Graham D.|||0000-0003-4040-0554
  • Herranz, Luis|||0000-0002-7022-3395
Improving images captured under low-light conditions has become an important topic in computational color imaging, as it has a wide range of applications. Most current methods are either based on handcrafted features or on end-to-end training of deep neural networks that mostly focus on minimizing some distortion metric -such as PSNR or SSIM-on a set of training images. However, the minimization of distortion metrics does not mean that the results are optimal in terms of perception (i.e. perceptual quality). As an example, the perception-distortion trade-off states that, close to the optimal results, improving distortion results in worsening perception. This means that current low-light image enhancement methods -that focus on distortion minimization- cannot be optimal in the sense of obtaining a good image in terms of perception errors. In this paper, we propose a post-processing approach in which, given the original low-light image and the result of a specific method, we are able to obtain a result that resembles as much as possible that of the original method, but, at the same time, giving an improvement in the perception of the final image. More in detail, our method follows the hypothesis that in order to minimally modify the perception of an input image, any modification should be a combination of a local change in the shading across a scene and a global change in illumination color. We demonstrate the ability of our method quantitatively using perceptual blind image metrics such as BRISQUE, NIQE, or UNIQUE, and through user preference tests.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Integrating High-Level Features for Consistent Palette-based Multi-image Recoloring

  • Xue, Danna|||0000-0002-4280-706X
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Herranz, Luis|||0000-0002-7022-3395
  • Zhang, Yanning|||0000-0002-2977-8057
  • Brown, Michael S.|||0000-0002-9840-0795
Achieving visually consistent colors across multiple images is important when images are used in photo albums, websites, and brochures. Unfortunately, only a handful of methods address multi-image color consistency compared to one-to-one color transfer techniques. Furthermore, existing methods do not incorporate high-level features that can assist graphic designers in their work. To address these limitations, we introduce a framework that builds upon a previous palette-based color consistency method and incorporates three high-level features: white balance, saliency, and color naming. We show how these features overcome the limitations of the prior multi-consistency workflow and showcase the user-friendly nature of our framework.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Palette-Based Color Harmonization via Color Naming

  • Xue, Danna|||0000-0002-4280-706X
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Herranz, Luis|||0000-0002-7022-3395
  • Zhang, Yanning|||0000-0002-2977-8057
  • Brown, Michael S.|||0000-0002-9840-0795
Color harmony refers to combinations of colors that look pleasing together. We present a novel strategy to harmonize an image's colors using color-palette manipulation and color naming. Palette-based color manipulation is a method that extracts a few colors to represent the image. Modifying the palette colors modifies the color appearance of the image. A color-naming model is a mechanism to categorize colors into a fixed number of basic color terms. Working from a color-naming model, we derive a set of prototype colors and demonstrate that mapping an image's extracted color palette to the nearest prototype colors effectively harmonizes the image's colors. This straightforward approach yields visually compelling, outperforming more complex color harmony methods.