Pasar al contenido principal

DEL CINE A LOS MODELOS DE VISION, Y VICEVERSA

PGC2018-099651-B-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de Generación de Conocimiento y Fortalecimiento Científico y Tecnológico del Sistema de I+D+i
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D (Excelencia / Generación de Conocimiento)
Año convocatoria 2018
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 30
Encontrada(s) 1 página(s)

Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, Syed Waqas|||0000-0002-7198-0187
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Bertalmío, Marcelo|||0000-0002-1023-8325
Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette of the display device where the content will be shown, while preserving the artistic intent of the original content's creator. In particular, in the cinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamut mapping algorithms. In this article, we propose a novel framework that is based on vision science models, performs both gamut reduction and gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-art methods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamut mapping problem.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Physical-based optimization for non-physical image dehazing methods

  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Finlayson, Graham D.|||0000-0003-4040-0554
  • Bertalmío, Marcelo|||0000-0002-1023-8325
Images captured under hazy conditions (e.g. fog, air pollution) usually present faded colors and loss of contrast. To improve their visibility, a process called image dehazing can be applied. Some of the most successful image dehazing algorithms are based on image processing methods but do not follow any physical image formation model, which limits their performance. In this paper, we propose a post-processing technique to alleviate this handicap by enforcing the original method to be consistent with a popular physical model for image formation under haze. Our results improve upon those of the original methods qualitatively and according to several metrics, and they have also been validated via psychophysical experiments. These results are particularly striking in terms of avoiding over-saturation and reducing color artifacts, which are the most common shortcomings faced by image dehazing methods.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Physical-based optimization for non-physical image dehazing methods

  • Vazquez-Corral, Javier
  • Finlayson, Graham D.
  • Bertalmío, Marcelo
Images captured under hazy conditions (e.g. fog, air pollution) usually present faded colors and loss of contrast. To improve their visibility, a process called image dehazing can be applied. Some of the most successful image dehazing algorithms are based on image processing methods but do not follow any physical image formation model, which limits their performance. In this paper, we propose a post-processing technique to alleviate this handicap by enforcing the original method to be consistent with a popular physical model for image formation under haze. Our results improve upon those of the original methods qualitatively and according to several metrics, and they have also been validated via psychophysical experiments. These results are particularly striking in terms of avoiding over-saturation and reducing color artifacts, which are the most common shortcomings faced by image dehazing methods., Horizon 2020 Framework Programme (761544, 780470); Engineering and Physical Sciences Research Council (EP/028730, EP/M001768); Spanish Government MINECO and Feder Fund (PGC2018-099651-B-I00).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Photorealistic style transfer for video

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
This work proposes an efficient example-based photorealistic style transfer algorithm for video. Given a source unprocessed video and a reference image color graded by an artist, we describe an automatic algorithm to transfer the visual style of the reference onto the source footage. Our approach builds upon the color transfer methods based on the statistical properties of images. These methods are fast and of low-complexity, therefore suitable for real-time implementations. Our contribution is to adapt those methods to be used for unprocessed video from cinema cameras, and optionally to incorporate regions of interest previously selected by the user, affecting the final color transfer. The resulting videos are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time. Results are free of spatio-temporal artifacts and the method outperforms diverse state-of-the-art methods according to observer preference experiments., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Cortical-inspired Wilson–Cowan-type equations for orientation-dependent contrast perception modelling

  • Bertalmío, Marcelo
  • Calatroni, Luca
  • Franceschi, Valentina
  • Franceschiello, Benedetta
  • Prandi, Dario
We consider the evolution model proposed in Bertalmío (Front Comput Neurosci 8:71, 2014), Bertalmío et al. (IEEE Trans Image Process 16(4):1058–1072, 2007) to describe illusory contrast perception phenomena induced by surrounding orientations. Firstly, we highlight its analogies and differences with the widely used Wilson–Cowan equations (Wilson and Cowan in BioPhys J 12(1):1–24, 1972), mainly in terms of efficient representation properties. Then, in order to explicitly encode local directional information, we exploit the model of the primary visual cortex (V1) proposed in Citti and Sarti (J Math Imaging Vis 24(3):307–326, 2006) and largely used over the last years for several image processing problems (Duits and Franken in Q Appl Math 68(2):255–292, 2010; Prandi and Gauthier in A semidiscrete version of the Petitot model as a plausible model for anthropomorphic image reconstruction and pattern recognition. SpringerBriefs in Mathematics, Springer, Cham, 2017; Franceschiello et al. in J Math Imaging Vis 60(1):94–108, 2018). The resulting model is thus defined in the space of positions and orientation, and it is capable of describing assimilation and contrast visual bias at the same time. We report several numerical tests showing the ability of the model to reproduce, in particular, orientation-dependent phenomena such as grating induction and a modified version of the Poggendorff illusion. For this latter example, we empirically show the existence of a set of threshold parameters differentiating from inpainting to perception-type reconstructions and describing long-range connectivity between different hypercolumns in V1., The authors acknowledge the anonymous referees for their suggestions which improved significantly the quality of their manuscript. M. B. acknowledges the support of the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 761544 (Project HDR4EU) and under Grant Agreement No. 780470 (Project SAUCE), and of the Spanish government and FEDER Fund, Grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). L. C., V. F. and D. P. acknowledge the support of a public grant overseen by the French National Research Agency (ANR) as part of the Investissement d’avenir program, through the iCODE project funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02 and of the research project LiftME funded by INS2I, CNRS. V. F. acknowledges the support received from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant No. 794592 and from the INdAM project Problemi isoperimetrici in spazi Euclidei e non. V. F. and D. P. also acknowledge the support of ANR-15-CE40-0018 project SRGI - Sub-Riemannian Geometry and Interactions. B. F. acknowledges the support of the Fondation Asile des Aveugles.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Matching visual induction effects on screens of different size

  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Mathieu, Elise
  • Bertalmío, Marcelo
In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen–size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images., Funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). JVC received funding from the project PID2019-109628RJ-I00/AEI/10.13039/501100011033 by the Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) of the Spanish government.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Color illusions also deceive CNNs for low-level vision tasks: analysis and implications

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
  • Malo, Jesús
The study of visual illusions has proven to be a very useful approach in vision science. In this work we start by showing that, while convolutional neural networks (CNNs) trained for low-level visual tasks in natural images may be deceived by brightness and color illusions, some network illusions can be inconsistent with the perception of humans. Next, we analyze where these similarities and differences may come from. On one hand, the proposed linear eigenanalysis explains the overall similarities: in simple CNNs trained for tasks like denoising or deblurring, the linear version of the network has center-surround receptive fields, and global transfer functions are very similar to the human achromatic and chromatic contrast sensitivity functions in human-like opponent color spaces. These similarities are consistent with the long-standing hypothesis that considers low-level visual illusions as a by-product of the optimization to natural environments. Specifically, here human-like features emerge from error minimization. On the other hand, the observed differences must be due to the behavior of the human visual system not explained by the linear approximation. However, our study also shows that more ‘flexible’ network architectures, with more layers and a higher degree of nonlinearity, may actually have a worse capability of reproducing visual illusions. This implies, in line with other works in the vision science literature, a word of caution on using CNNs to study human vision: on top of the intrinsic limitations of the L+NL formulation of artificial networks to model vision, the nonlinear behavior of flexible architectures may easily be markedly different from that of the visual system., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). The work of AM was supported by the Spanish government under Grant FJCI-2017–31758. JM has been supported by the Spanish government under the MINECO grant Ref. DPI2017-89867 and by the Generalitat Velanciana grant Ref. GrisoliaP-2019-035. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

The persistent influence of viewing environment illumination color on displayed image appearance

  • Canham, Trevor
  • Bertalmío, Marcelo
Comunicació presentada a la IS&T London Imaging Meeting, celebrada online del 29 de setembre a l'1 d'octubre de 2020., Chromatic adaptation considering competing influences from emissive displays and ambient illumination is a little studied topic in the context of color management in proportion to its influence on displayed image appearance. An experiment was conducted to identify the degree to which observers adapt to the white point of natural images on an emissive display versus the color of ambient illumination in the room. The responses of observers had no significant difference from those of a previous experiment which was conducted with roughly the same procedure and conditions on a mobile display with a significantly smaller viewing angle. A model is proposed to predict the degree of adaptation values reported by observers. This model has a form such that it can be re-optimized to fit additional data sets for different viewing scenarios and can be used in conjunction with a number of chromatic adaptation transforms., The authors would like to thank all observers for their participation, in particular the colorists from Deluxe Spain who generously lent their time and spectrophotometer to the experiment. This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Color stabilization for multi-camera light-field imaging

  • Vu-Thanh, Olivier
  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Gil Rodríguez, Raquel
  • Bertalmío, Marcelo
By capturing a more complete rendition of scene light than standard 2D cameras, light-field technology represents an important step towards closing the gap between live action cinematography and computer graphics. Light-field cameras accomplish this by simultaneously capturing the same scene under different angular configurations, providing directional information that allows for a multitude of post-production effects. Among the practical challenges related to capturing multiple images simultaneously, a very important problem is the fact that the different images do not perfectly match in terms of color, which severely complicates all further processing. In this work we adapt and extend to the light-field scenario a color stabilization method previously proposed for standard multi-camera shoots, and demonstrate experimentally that it provides an improvement over the state-of-the-art techniques for light-field imaging., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

  • Cyriac, Praveen
  • Canham, Trevor
  • Kane, David
  • Bertalmío, Marcelo
Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). We’re very grateful to Albert Pascual, Brett Harrison, Stephane Cattan and everyone at Deluxe-Spain, Alejandro Matus and everyone at Moonlight Barcelona, for their help in fine-tuning and validating our method.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Contrast sensitivity functions in autoencoders

  • Li, Qiang
  • Gomez-Villa, Alex
  • Bertalmío, Marcelo
  • Malo, Jesús
Three decades ago, Atick et al. suggested that human frequency sensitivity may emerge from the enhancement required for a more efficient analysis of retinal images. Here we reassess the relevance of low-level vision tasks in the explanation of the contrast sensitivity functions (CSFs) in light of 1) the current trend of using artificial neural networks for studying vision, and 2) the current knowledge of retinal image representations. As a first contribution, we show that a very popular type of convolutional neural networks (CNNs), called autoencoders, may develop human-like CSFs in the spatiotemporal and chromatic dimensions when trained to perform some basic low-level vision tasks (like retinal noise and optical blur removal), but not others (like chromatic) adaptation or pure reconstruction after simple bottlenecks). As an illustrative example, the best CNN (in the considered set of simple architectures for enhancement of the retinal signal) reproduces the CSFs with a root mean square error of 11% of the maximum sensitivity. As a second contribution, we provide experimental evidence of the fact that, for some functional goals (at low abstraction level), deeper CNNs that are better in reaching the quantitative goal are actually worse in replicating human-like phenomena (such as the CSFs). This low-level result (for the explored networks) is not necessarily in contradiction with other works that report advantages of deeper nets in modeling higher level vision goals. However, in line with a growing body of literature, our results suggests another word of caution about CNNs in vision science because the use of simplified units or unrealistic architectures in goal optimization may be a limitation for the modeling and understanding of human vision., Partially funded by these grants from GVA/AEI/FEDER/EU: MICINN DPI2017- 89867-C2-2-R, MICINN PID2020-118071GB-I00, and GVA Grisolía-P/2019/035 (for JM and QL), and MICINN PGC2018-099651-B-I00 (for A.G.V. and M.B.).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Retinal noise emulation: a novel artistic tool for cinema that also improves compression efficiency

  • Zabaleta, Itziar
  • Cámara, Mateo
  • Díaz, César
  • Canham, Trevor
  • García, Narciso
  • Bertalmío, Marcelo
In cinema it is standard practice to improve the appearance of images by adding noise that simulates film grain. This is computationally very costly, so it is only done in post-production and not on the set. It is also limiting because the artists are not able to really experiment with the noise nor introduce novel looks. Furthermore, video compression requires a higher bit rate when the source material has film grain or any other type of high frequency texture. In this work, we introduce a method for adding texture to digital cinema that aims to solve these problems. The proposed algorithm is based on modeling retinal noise, with which the images processed by our method have a natural appearance. This “retinal grain” serves a double purpose. One is aesthetic, as it has parameters that allow to vary widely the resulting texture appearance, which make it an artistic tool for cinematographers. Results are validated through psychophysical experiments in which observers, including cinema professionals, prefer our method over film grain synthesis methods from academia and the industry. The other purpose of the retinal noise emulation method is to improve the quality of compressed video by masking compression artifacts, which allows to lower the encoding bit rate while preserving image quality, and to improve image quality while keeping the bit rate fixed. The effectiveness of our approach for improving coding efficiency, with average bit rate savings of 22.5%, has been validated through psychophysical experiments using professional cinema content shot in 4K, color-graded and where the amount of retinal noise was selected by a motion picture specialist based solely on aesthetic preference., This work was supported in part by the European Union through the Horizon 2020 Research and Innovation Programme (Project HDR4EU) under Grant 761544 and (Project SAUCE) Grant 780470, and in part by the Spanish Government, Ministerio de Economía, Industria y Competitividad (AEI/FEDER), under Project TEC2016-75981 (IVME) and Project PGC2018-099651-B-I00.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Non-experts or experts? Statistical analyses of MOS using DSIS method

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), celebrada a Barcelona (Espanya) del 4 a 8 de maig de 2020., In image quality assessments, the results of subjective evaluation experiments that use the double-stimulus impairment scale (DSIS) method are often expressed in terms of the mean opinion score (MOS), which is the average score of all subjects for each test condition. Some MOS values are used to derive image quality criteria, and it has been assumed that it is preferable to perform tests with non-expert subjects rather than with experts. In this study, we analyze the results of several subjective evaluation experiments using the DSIS method. Our first contribution is to discuss the statistical meaning of the MOS values, which has not been previously addressed in the literature. Second, our results show that, contrary to the established belief, there are advantages when performing subjective tests with experts, in that they allow experiments to be performed with fewer subjects, and to better determine the lower threshold of image quality., This work received partial funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE) and from the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Photorealistic style transfer for video

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
11 pags., 14 figs., This work proposes an efficient example-based photorealistic style transfer algorithm for video. Given a source unprocessed video and a reference image color graded by an artist, we describe an automatic algorithm to transfer the visual style of the reference onto the source footage. Our approach builds upon the color transfer methods based on the statistical properties of images. These methods are fast and of low-complexity, therefore suitable for real-time implementations. Our contribution is to adapt those methods to be used for unprocessed video from cinema cameras, and optionally to incorporate regions of interest previously selected by the user, affecting the final color transfer. The resulting videos are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time. Results are free of spatio-temporal artifacts and the method outperforms diverse state-of-the-art methods according to observer preference experiments., This work has received funding from the European Union’s Hori- zon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE)
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

  • Cyriac, Praveen
  • Canham, Trevor
  • Kane, David
  • Bertalmío, Marcelo
27 pags., 16 figs., Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, S.W.
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
14 pags., 14 figs., Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette of the display device where the content will be shown, while preserving the artistic intent of the original content’s creator. In particular, in the cinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamut mapping algorithms. In this article, we propose a novel framework that is based on vision science models, performs both gamut reduction and gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-art methods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamut mapping problem., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant agreement number 761544 (project HDR4EU) and under Grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Contrast sensitivity functions in autoencoders

  • Li, Qiang
  • Gómez-Villa, Alex
  • Bertalmío, Marcelo
  • Malo, Jesús
45 pags., 24 figs., 4 tabs., 3 apps., The human contrast sensitivity function (CSF) characterizes the psychophysical response to visual gratings of different frequency (Campbell & Robson, 1968). Filter characterizations in the Fourier domain are complete only for linear, shift-invariant systems. Human vision certainly is more complicated than that, however, this simple measure of the bandwidth of the system is still of paramount significance in biological vision: the CSF filter is an image-computable model that roughly describes the kind of visual information that is available for humans (Watson & Ahumada, 2016). Moreover, although it is defined for threshold conditions, there are many examples that illustrate the relevance of the CSF in more general situations (Watson et al., 1986; Watson & Malo, 2002; Watson & Ahumada, 2005), so it has shaped image engineering over decades (Mannos & Sakrison, 1974; Hunt, 1975; Wallace, 1992; Taubman & Marcellin, 2001). This theoretical and practical relevance motivated the measurement of CSFs, not only for spatial gratings (Campbell & Robson, 1968), but also for moving gratings (Kelly, 1979), chromatic gratings (Mullen, 1985), spatiotemporal chromatic gratings (Díez-Ajenjo et al., 2011), at different luminance levels (Wuerger et al., 2020), and for alternative basis of the image space (Malo et al., 1997), Partially funded by these grants from GVA/AEI/FEDER/EU: MICINN DPI2017-89867-C2-2-R, MICINN PID2020-118071GB-I00, and GVA Grisolía-P/2019/035 (for JM and QL), and MICINN PGC2018-099651-B-I00 (for A.G.V. and M.B.)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Color stabilization for multi-camera light-field imaging

  • Vu-Thanh, Olivier
  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Gil Rodríguez, Raquel
  • Bertalmío, Marcelo
By capturing a more complete rendition of scene light than standard 2D cameras, light-field technology represents an important step towards closing the gap between live action cinematography and computer graphics. Light-field cameras accomplish this by simultaneously capturing the same scene under different angular configurations, providing directional information that allows for a multitude of post-production effects. Among the practical challenges related to capturing multiple images simultaneously, a very important problem is the fact that the different images do not perfectly match in terms of color, which severely complicates all further processing. In this work we adapt and extend to the light-field scenario a color stabilization method previously proposed for standard multi-camera shoots, and demonstrate experimentally that it provides an improvement over the state-of-the-art techniques for light-field imaging., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Physical-based optimization for non-physical image dehazing methods

  • Vazquez-Corral, Javier
  • Finlayson, Graham D.
  • Bertalmío, Marcelo
Images captured under hazy conditions (e.g. fog, air pollution) usually present faded colors and loss of contrast. To improve their visibility, a process called image dehazing can be applied. Some of the most successful image dehazing algorithms are based on image processing methods but do not follow any physical image formation model, which limits their performance. In this paper, we propose a post-processing technique to alleviate this handicap by enforcing the original method to be consistent with a popular physical model for image formation under haze. Our results improve upon those of the original methods qualitatively and according to several metrics, and they have also been validated via psychophysical experiments. These results are particularly striking in terms of avoiding over-saturation and reducing color artifacts, which are the most common shortcomings faced by image dehazing methods., Horizon 2020 Framework Programme (761544, 780470); Engineering and Physical Sciences Research Council (EP/028730, EP/M001768); Spanish Government MINECO and Feder Fund (PGC2018-099651-B-I00).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Matching visual induction effects on screens of different size

  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Mathieu, Elise
  • Bertalmío, Marcelo
In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen–size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images., Funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). JVC received funding from the project PID2019-109628RJ-I00/AEI/10.13039/501100011033 by the Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) of the Spanish government.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Photorealistic style transfer for video

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
This work proposes an efficient example-based photorealistic style transfer algorithm for video. Given a source unprocessed video and a reference image color graded by an artist, we describe an automatic algorithm to transfer the visual style of the reference onto the source footage. Our approach builds upon the color transfer methods based on the statistical properties of images. These methods are fast and of low-complexity, therefore suitable for real-time implementations. Our contribution is to adapt those methods to be used for unprocessed video from cinema cameras, and optionally to incorporate regions of interest previously selected by the user, affecting the final color transfer. The resulting videos are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time. Results are free of spatio-temporal artifacts and the method outperforms diverse state-of-the-art methods according to observer preference experiments., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Cortical-inspired Wilson–Cowan-type equations for orientation-dependent contrast perception modelling

  • Bertalmío, Marcelo
  • Calatroni, Luca
  • Franceschi, Valentina
  • Franceschiello, Benedetta
  • Prandi, Dario
We consider the evolution model proposed in Bertalmío (Front Comput Neurosci 8:71, 2014), Bertalmío et al. (IEEE Trans Image Process 16(4):1058–1072, 2007) to describe illusory contrast perception phenomena induced by surrounding orientations. Firstly, we highlight its analogies and differences with the widely used Wilson–Cowan equations (Wilson and Cowan in BioPhys J 12(1):1–24, 1972), mainly in terms of efficient representation properties. Then, in order to explicitly encode local directional information, we exploit the model of the primary visual cortex (V1) proposed in Citti and Sarti (J Math Imaging Vis 24(3):307–326, 2006) and largely used over the last years for several image processing problems (Duits and Franken in Q Appl Math 68(2):255–292, 2010; Prandi and Gauthier in A semidiscrete version of the Petitot model as a plausible model for anthropomorphic image reconstruction and pattern recognition. SpringerBriefs in Mathematics, Springer, Cham, 2017; Franceschiello et al. in J Math Imaging Vis 60(1):94–108, 2018). The resulting model is thus defined in the space of positions and orientation, and it is capable of describing assimilation and contrast visual bias at the same time. We report several numerical tests showing the ability of the model to reproduce, in particular, orientation-dependent phenomena such as grating induction and a modified version of the Poggendorff illusion. For this latter example, we empirically show the existence of a set of threshold parameters differentiating from inpainting to perception-type reconstructions and describing long-range connectivity between different hypercolumns in V1., The authors acknowledge the anonymous referees for their suggestions which improved significantly the quality of their manuscript. M. B. acknowledges the support of the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 761544 (Project HDR4EU) and under Grant Agreement No. 780470 (Project SAUCE), and of the Spanish government and FEDER Fund, Grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). L. C., V. F. and D. P. acknowledge the support of a public grant overseen by the French National Research Agency (ANR) as part of the Investissement d’avenir program, through the iCODE project funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02 and of the research project LiftME funded by INS2I, CNRS. V. F. acknowledges the support received from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant No. 794592 and from the INdAM project Problemi isoperimetrici in spazi Euclidei e non. V. F. and D. P. also acknowledge the support of ANR-15-CE40-0018 project SRGI - Sub-Riemannian Geometry and Interactions. B. F. acknowledges the support of the Fondation Asile des Aveugles.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

The persistent influence of viewing environment illumination color on displayed image appearance

  • Canham, Trevor
  • Bertalmío, Marcelo
Comunicació presentada a la IS&T London Imaging Meeting, celebrada online del 29 de setembre a l'1 d'octubre de 2020., Chromatic adaptation considering competing influences from emissive displays and ambient illumination is a little studied topic in the context of color management in proportion to its influence on displayed image appearance. An experiment was conducted to identify the degree to which observers adapt to the white point of natural images on an emissive display versus the color of ambient illumination in the room. The responses of observers had no significant difference from those of a previous experiment which was conducted with roughly the same procedure and conditions on a mobile display with a significantly smaller viewing angle. A model is proposed to predict the degree of adaptation values reported by observers. This model has a form such that it can be re-optimized to fit additional data sets for different viewing scenarios and can be used in conjunction with a number of chromatic adaptation transforms., The authors would like to thank all observers for their participation, in particular the colorists from Deluxe Spain who generously lent their time and spectrophotometer to the experiment. This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Non-experts or experts? Statistical analyses of MOS using DSIS method

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), celebrada a Barcelona (Espanya) del 4 a 8 de maig de 2020., In image quality assessments, the results of subjective evaluation experiments that use the double-stimulus impairment scale (DSIS) method are often expressed in terms of the mean opinion score (MOS), which is the average score of all subjects for each test condition. Some MOS values are used to derive image quality criteria, and it has been assumed that it is preferable to perform tests with non-expert subjects rather than with experts. In this study, we analyze the results of several subjective evaluation experiments using the DSIS method. Our first contribution is to discuss the statistical meaning of the MOS values, which has not been previously addressed in the literature. Second, our results show that, contrary to the established belief, there are advantages when performing subjective tests with experts, in that they allow experiments to be performed with fewer subjects, and to better determine the lower threshold of image quality., This work received partial funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE) and from the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Contrast sensitivity functions in autoencoders

  • Li, Qiang
  • Gomez-Villa, Alex
  • Bertalmío, Marcelo
  • Malo, Jesús
Three decades ago, Atick et al. suggested that human frequency sensitivity may emerge from the enhancement required for a more efficient analysis of retinal images. Here we reassess the relevance of low-level vision tasks in the explanation of the contrast sensitivity functions (CSFs) in light of 1) the current trend of using artificial neural networks for studying vision, and 2) the current knowledge of retinal image representations. As a first contribution, we show that a very popular type of convolutional neural networks (CNNs), called autoencoders, may develop human-like CSFs in the spatiotemporal and chromatic dimensions when trained to perform some basic low-level vision tasks (like retinal noise and optical blur removal), but not others (like chromatic) adaptation or pure reconstruction after simple bottlenecks). As an illustrative example, the best CNN (in the considered set of simple architectures for enhancement of the retinal signal) reproduces the CSFs with a root mean square error of 11% of the maximum sensitivity. As a second contribution, we provide experimental evidence of the fact that, for some functional goals (at low abstraction level), deeper CNNs that are better in reaching the quantitative goal are actually worse in replicating human-like phenomena (such as the CSFs). This low-level result (for the explored networks) is not necessarily in contradiction with other works that report advantages of deeper nets in modeling higher level vision goals. However, in line with a growing body of literature, our results suggests another word of caution about CNNs in vision science because the use of simplified units or unrealistic architectures in goal optimization may be a limitation for the modeling and understanding of human vision., Partially funded by these grants from GVA/AEI/FEDER/EU: MICINN DPI2017- 89867-C2-2-R, MICINN PID2020-118071GB-I00, and GVA Grisolía-P/2019/035 (for JM and QL), and MICINN PGC2018-099651-B-I00 (for A.G.V. and M.B.).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

  • Cyriac, Praveen
  • Canham, Trevor
  • Kane, David
  • Bertalmío, Marcelo
Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). We’re very grateful to Albert Pascual, Brett Harrison, Stephane Cattan and everyone at Deluxe-Spain, Alejandro Matus and everyone at Moonlight Barcelona, for their help in fine-tuning and validating our method.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Color illusions also deceive CNNs for low-level vision tasks: analysis and implications

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
  • Malo, Jesús
The study of visual illusions has proven to be a very useful approach in vision science. In this work we start by showing that, while convolutional neural networks (CNNs) trained for low-level visual tasks in natural images may be deceived by brightness and color illusions, some network illusions can be inconsistent with the perception of humans. Next, we analyze where these similarities and differences may come from. On one hand, the proposed linear eigenanalysis explains the overall similarities: in simple CNNs trained for tasks like denoising or deblurring, the linear version of the network has center-surround receptive fields, and global transfer functions are very similar to the human achromatic and chromatic contrast sensitivity functions in human-like opponent color spaces. These similarities are consistent with the long-standing hypothesis that considers low-level visual illusions as a by-product of the optimization to natural environments. Specifically, here human-like features emerge from error minimization. On the other hand, the observed differences must be due to the behavior of the human visual system not explained by the linear approximation. However, our study also shows that more ‘flexible’ network architectures, with more layers and a higher degree of nonlinearity, may actually have a worse capability of reproducing visual illusions. This implies, in line with other works in the vision science literature, a word of caution on using CNNs to study human vision: on top of the intrinsic limitations of the L+NL formulation of artificial networks to model vision, the nonlinear behavior of flexible architectures may easily be markedly different from that of the visual system., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). The work of AM was supported by the Spanish government under Grant FJCI-2017–31758. JM has been supported by the Spanish government under the MINECO grant Ref. DPI2017-89867 and by the Generalitat Velanciana grant Ref. GrisoliaP-2019-035. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Retinal noise emulation: a novel artistic tool for cinema that also improves compression efficiency

  • Zabaleta, Itziar
  • Cámara, Mateo
  • Díaz, César
  • Canham, Trevor
  • García, Narciso
  • Bertalmío, Marcelo
In cinema it is standard practice to improve the appearance of images by adding noise that simulates film grain. This is computationally very costly, so it is only done in post-production and not on the set. It is also limiting because the artists are not able to really experiment with the noise nor introduce novel looks. Furthermore, video compression requires a higher bit rate when the source material has film grain or any other type of high frequency texture. In this work, we introduce a method for adding texture to digital cinema that aims to solve these problems. The proposed algorithm is based on modeling retinal noise, with which the images processed by our method have a natural appearance. This “retinal grain” serves a double purpose. One is aesthetic, as it has parameters that allow to vary widely the resulting texture appearance, which make it an artistic tool for cinematographers. Results are validated through psychophysical experiments in which observers, including cinema professionals, prefer our method over film grain synthesis methods from academia and the industry. The other purpose of the retinal noise emulation method is to improve the quality of compressed video by masking compression artifacts, which allows to lower the encoding bit rate while preserving image quality, and to improve image quality while keeping the bit rate fixed. The effectiveness of our approach for improving coding efficiency, with average bit rate savings of 22.5%, has been validated through psychophysical experiments using professional cinema content shot in 4K, color-graded and where the amount of retinal noise was selected by a motion picture specialist based solely on aesthetic preference., This work was supported in part by the European Union through the Horizon 2020 Research and Innovation Programme (Project HDR4EU) under Grant 761544 and (Project SAUCE) Grant 780470, and in part by the Spanish Government, Ministerio de Economía, Industria y Competitividad (AEI/FEDER), under Project TEC2016-75981 (IVME) and Project PGC2018-099651-B-I00.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Matching visual induction effects on screens of different size

  • Canham, Trevor|||0000-0003-4830-4786
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Mathieu, Elise
  • Bertalmío, Marcelo|||0000-0002-1023-8325
In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen-size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images.
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Contrast sensitivity functions in autoencoders

  • Li, Qiang
  • Gómez Villa, Alexandra|||0000-0003-0469-3425
  • Bertalmío, Marcelo|||0000-0002-1023-8325
  • Malo, Jesús|||0000-0002-5684-8591
Three decades ago, Atick et al. suggested that human frequency sensitivity may emerge from the enhancement required for a more efficient analysis of retinal images. Here we reassess the relevance of low-level vision tasks in the explanation of the contrast sensitivity functions (CSFs) in light of 1) the current trend of using artificial neural networks for studying vision, and 2) the current knowledge of retinal image representations. As a first contribution, we show that a very popular type of convolutional neural networks (CNNs), called autoencoders, may develop human-like CSFs in the spatiotemporal and chromatic dimensions when trained to perform some basic low-level vision tasks (like retinal noise and optical blur removal), but not others (like chromatic) adaptation or pure reconstruction after simple bottlenecks). As an illustrative example, the best CNN (in the considered set of simple architectures for enhancement of the retinal signal) reproduces the CSFs with a root mean square error of 11% of the maximum sensitivity. As a second contribution, we provide experimental evidence of the fact that, for some functional goals (at low abstraction level), deeper CNNs that are better in reaching the quantitative goal are actually worse in replicating human-like phenomena (such as the CSFs). This low-level result (for the explored networks) is not necessarily in contradiction with other works that report advantages of deeper nets in modeling higher level vision goals. However, in line with a growing body of literature, our results suggests another word of caution about CNNs in vision science because the use of simplified units or unrealistic architectures in goal optimization may be a limitation for the modeling and understanding of human vision.