Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 57
Encontrada(s) 1 página(s)

Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, Syed Waqas|||0000-0002-7198-0187
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Bertalmío, Marcelo|||0000-0002-1023-8325
Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette of the display device where the content will be shown, while preserving the artistic intent of the original content's creator. In particular, in the cinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamut mapping algorithms. In this article, we propose a novel framework that is based on vision science models, performs both gamut reduction and gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-art methods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamut mapping problem.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Color Matching Images with Unknown Non-Linear Encodings

  • Gil Rodríguez, Raquel|||0000-0001-5561-1990
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Bertalmío, Marcelo|||0000-0002-1023-8325
We present a color matching method that deals with different non-linear encodings. In particular, given two different views of the same scene taken by two cameras with unknown settings and internal parameters, and encoded with unknown non-linear curves, our method is able to correct the colors of one of the images making it look as if it was captured under the other camera's settings. Our method is based on treating the in-camera color processing pipeline as a concatenation of a matrix multiplication on the linear image followed by a non-linearity. This allows us to model a color stabilization transformation among the two shots by estimating a single matrix -that will contain information from both of the original images- and an extra parameter that complies with the non-linearity. The method is fast and the results have no spurious colors. It outperforms the state-of-the-art both visually and according to several metrics, and can handle HDR encodings and very challenging real-life examples.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Convolutional neural networks can be deceived by visual illusions

  • Gómez Villa, Alexandra|||0000-0003-0469-3425
  • Martín, Adrián|||0000-0001-7127-0073
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Bertalmío, Marcelo|||0000-0002-1023-8325
Visual illusions teach us that what we see is not always what is represented in the physical world. Their special nature make them a fascinating tool to test and validate any new vision model proposed. In general, current vision models are based on the concatenation of linear and non-linear operations. The similarity of this structure with the operations present in Convolutional Neural Networks (CNNs) has motivated us to study if CNNs trained for low-level visual tasks are deceived by visual illusions. In particular, we show that CNNs trained for image denoising, image deblurring, and computational color constancy are able to replicate the human response to visual illusions, and that the extent of this replication varies with respect to variation in architecture and spatial pattern size. These results suggest that in order to obtain CNNs that better replicate human behaviour, we may need to start aiming for them to better replicate visual illusions.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Physical-based optimization for non-physical image dehazing methods

  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Finlayson, Graham D.|||0000-0003-4040-0554
  • Bertalmío, Marcelo|||0000-0002-1023-8325
Images captured under hazy conditions (e.g. fog, air pollution) usually present faded colors and loss of contrast. To improve their visibility, a process called image dehazing can be applied. Some of the most successful image dehazing algorithms are based on image processing methods but do not follow any physical image formation model, which limits their performance. In this paper, we propose a post-processing technique to alleviate this handicap by enforcing the original method to be consistent with a popular physical model for image formation under haze. Our results improve upon those of the original methods qualitatively and according to several metrics, and they have also been validated via psychophysical experiments. These results are particularly striking in terms of avoiding over-saturation and reducing color artifacts, which are the most common shortcomings faced by image dehazing methods.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

The Potential of light fields in media productions

  • Trottnow, Jonas
  • Spielmann, Simon
  • Lange, Tobias
  • Chelli, Kelvin
  • Solony, Marek
  • Smrž, Pavel
  • Zemčík, Pavel
  • Aenchbacher, Weston
  • Grogan, Mairéad
  • Alain, Martin
  • Smolic, Aljosa
  • Canham, Trevor
  • Vu-Thanh, Olivier
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: The 12th ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, celebrada del 17 al 20 de novembre de 2019 a Brisbane, Austràlia., One aspect of the EU funded project SAUCE is to explore the pos-sibilities and challenges of integrating light field capturing andprocessing into media productions. A special light field camera wasbuild by Saarland University [Herfet et al.2018] and is first testedunder production conditions in the test production “Unfolding” aspart of the SAUCE project. Filmakademie Baden-Württemberg de-veloped the contentual frame, executed the post-production and pre-pared a complete previsualization. Calibration and post-processingalgorithms are developed by the Trinity College Dublin and theBrno University of Technology. This document describes challengesduring building and shooting with the light field camera array, aswell as its potential and challenges for the post-production., The project has received funding from the European Union’s Hori-zon 2020 Research and Innovation Programme under Grant Agree-ment No. 780470.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Physical-based optimization for non-physical image dehazing methods

  • Vazquez-Corral, Javier
  • Finlayson, Graham D.
  • Bertalmío, Marcelo
Images captured under hazy conditions (e.g. fog, air pollution) usually present faded colors and loss of contrast. To improve their visibility, a process called image dehazing can be applied. Some of the most successful image dehazing algorithms are based on image processing methods but do not follow any physical image formation model, which limits their performance. In this paper, we propose a post-processing technique to alleviate this handicap by enforcing the original method to be consistent with a popular physical model for image formation under haze. Our results improve upon those of the original methods qualitatively and according to several metrics, and they have also been validated via psychophysical experiments. These results are particularly striking in terms of avoiding over-saturation and reducing color artifacts, which are the most common shortcomings faced by image dehazing methods., Horizon 2020 Framework Programme (761544, 780470); Engineering and Physical Sciences Research Council (EP/028730, EP/M001768); Spanish Government MINECO and Feder Fund (PGC2018-099651-B-I00).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Photorealistic style transfer for video

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
This work proposes an efficient example-based photorealistic style transfer algorithm for video. Given a source unprocessed video and a reference image color graded by an artist, we describe an automatic algorithm to transfer the visual style of the reference onto the source footage. Our approach builds upon the color transfer methods based on the statistical properties of images. These methods are fast and of low-complexity, therefore suitable for real-time implementations. Our contribution is to adapt those methods to be used for unprocessed video from cinema cameras, and optionally to incorporate regions of interest previously selected by the user, affecting the final color transfer. The resulting videos are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time. Results are free of spatio-temporal artifacts and the method outperforms diverse state-of-the-art methods according to observer preference experiments., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Cortical-inspired Wilson–Cowan-type equations for orientation-dependent contrast perception modelling

  • Bertalmío, Marcelo
  • Calatroni, Luca
  • Franceschi, Valentina
  • Franceschiello, Benedetta
  • Prandi, Dario
We consider the evolution model proposed in Bertalmío (Front Comput Neurosci 8:71, 2014), Bertalmío et al. (IEEE Trans Image Process 16(4):1058–1072, 2007) to describe illusory contrast perception phenomena induced by surrounding orientations. Firstly, we highlight its analogies and differences with the widely used Wilson–Cowan equations (Wilson and Cowan in BioPhys J 12(1):1–24, 1972), mainly in terms of efficient representation properties. Then, in order to explicitly encode local directional information, we exploit the model of the primary visual cortex (V1) proposed in Citti and Sarti (J Math Imaging Vis 24(3):307–326, 2006) and largely used over the last years for several image processing problems (Duits and Franken in Q Appl Math 68(2):255–292, 2010; Prandi and Gauthier in A semidiscrete version of the Petitot model as a plausible model for anthropomorphic image reconstruction and pattern recognition. SpringerBriefs in Mathematics, Springer, Cham, 2017; Franceschiello et al. in J Math Imaging Vis 60(1):94–108, 2018). The resulting model is thus defined in the space of positions and orientation, and it is capable of describing assimilation and contrast visual bias at the same time. We report several numerical tests showing the ability of the model to reproduce, in particular, orientation-dependent phenomena such as grating induction and a modified version of the Poggendorff illusion. For this latter example, we empirically show the existence of a set of threshold parameters differentiating from inpainting to perception-type reconstructions and describing long-range connectivity between different hypercolumns in V1., The authors acknowledge the anonymous referees for their suggestions which improved significantly the quality of their manuscript. M. B. acknowledges the support of the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 761544 (Project HDR4EU) and under Grant Agreement No. 780470 (Project SAUCE), and of the Spanish government and FEDER Fund, Grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). L. C., V. F. and D. P. acknowledge the support of a public grant overseen by the French National Research Agency (ANR) as part of the Investissement d’avenir program, through the iCODE project funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02 and of the research project LiftME funded by INS2I, CNRS. V. F. acknowledges the support received from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant No. 794592 and from the INdAM project Problemi isoperimetrici in spazi Euclidei e non. V. F. and D. P. also acknowledge the support of ANR-15-CE40-0018 project SRGI - Sub-Riemannian Geometry and Interactions. B. F. acknowledges the support of the Fondation Asile des Aveugles.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Matching visual induction effects on screens of different size

  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Mathieu, Elise
  • Bertalmío, Marcelo
In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen–size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images., Funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). JVC received funding from the project PID2019-109628RJ-I00/AEI/10.13039/501100011033 by the Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) of the Spanish government.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline and their impact in hdr imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tonemapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., The work of the authors was supported by the European Unions Horizon 2020 research and innovation programme grant 761544 (project HDR4EU) and grant 780470 (project SAUCE) and by the Spanish government and FEDER fund grant TIN2015-71537-P (MINECO/FEDER,UE). The work of the second author was supported by the Spanish government under grant IJCI-2014-19516.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

In-camera, photorealistic style transfer for on-set automatic grading

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a la SMPTE 2018 Annual Technical Conference of the Society of Motion Picture and Television Engineers, celebrada a Los Angeles (Estats Units d'Amèrica) del 22 a 25 de octubre de 2018., In professional cinema, the intended artistic look of the movie informs the creation of a static 3D LUT that is applied on set, where further manual modifications to the image appearance are registered as 10-parameter transforms in a color decision list (CDL). The original RAW footage and its corresponding LUT and CDL are passedon to the post-production stage where the fine-tuning of the final look is performed during color grading. — In many cases, the director wants to emulate the style and look present in a reference image, e.g. a still from an existing movie, or a photograph, or a painting, or even a frame from a previously shot sequence in the current movie. The manual creation of a LUT and CDL for this purpose may require a significant amount of work from very skilled artists and technicians, while the state of the art in the academic literature offers promising but partial solutions to the photorealistic style transfer problem, with limitations regarding artifacts, speed and manual interaction. — In this paper, we propose a method that automatically transfers the style, in terms of luminance, color palette and contrast, from a reference image to the source raw footage. It consists of three separable operations: global luminance matching, global color transfer and local contrast matching. As it just takes into account the statistics of source and reference images, no training is required. The total transform is not static but adapts to the changes in the source footage. The computational complexity of the procedure is extremely low and allows for real-time implementation in-camera, for on-set monitoring. While the method is proposed as a substitute for the need to specify a LUT and a CDL, it's compatible with further refinements performed via LUTs, CDLs and grading, both on-set and in post-production. The results are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time., This work has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

In praise of artifice reloaded: caution with natural image databases in modeling vision

  • Martínez García, Marina
  • Bertalmío, Marcelo
  • Malo, Jesús
Subjective image quality databases are a major source of raw data on how the visual system works in naturalistic environments. These databases describe the sensitivity of many observers to a wide range of distortions of different nature and intensity seen on top of a variety of natural images. Data of this kind seems to open a number of possibilities for the vision scientist to check the models in realistic scenarios. However, while these natural databases are great benchmarks for models developed in some other way (e.g., by using the well-controlled artificial stimuli of traditional psychophysics), they should be carefully used when trying to fit vision models. Given the high dimensionality of the image space, it is very likely that some basic phenomena are under-represented in the database. Therefore, a model fitted on these large-scale natural databases will not reproduce these under-represented basic phenomena that could otherwise be easily illustrated with well selected artificial stimuli. In this work we study a specific example of the above statement. A standard corticalmodel using wavelets and divisive normalization tuned to reproduce subjective opinion on a large image quality dataset fails to reproduce basic cross-masking. Here we outline a solution for this problem by using artificial stimuli and by proposing a modification that makes the model easier to tune. Then, we show that the modified model is still competitive in the large-scale database. Our simulations with these artificial stimuli show that when using steerable wavelets, the conventional unit norm Gaussian kernels in divisive normalization should be multiplied by high-pass filters to reproduce basic trends in masking. Basic visual phenomena may be misrepresented in large natural image datasets but this can be solved with model-interpretable stimuli. This is an additional argument in praise of artifice in line with Rust and Movshon (2005)., This work was partially funded by the Spanish and EU FEDER fund through the MINECO/FEDER/EU grants TIN2015-71537- P and DPI2017-89867-C2-2-R; and by the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Color illusions also deceive CNNs for low-level vision tasks: analysis and implications

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
  • Malo, Jesús
The study of visual illusions has proven to be a very useful approach in vision science. In this work we start by showing that, while convolutional neural networks (CNNs) trained for low-level visual tasks in natural images may be deceived by brightness and color illusions, some network illusions can be inconsistent with the perception of humans. Next, we analyze where these similarities and differences may come from. On one hand, the proposed linear eigenanalysis explains the overall similarities: in simple CNNs trained for tasks like denoising or deblurring, the linear version of the network has center-surround receptive fields, and global transfer functions are very similar to the human achromatic and chromatic contrast sensitivity functions in human-like opponent color spaces. These similarities are consistent with the long-standing hypothesis that considers low-level visual illusions as a by-product of the optimization to natural environments. Specifically, here human-like features emerge from error minimization. On the other hand, the observed differences must be due to the behavior of the human visual system not explained by the linear approximation. However, our study also shows that more ‘flexible’ network architectures, with more layers and a higher degree of nonlinearity, may actually have a worse capability of reproducing visual illusions. This implies, in line with other works in the vision science literature, a word of caution on using CNNs to study human vision: on top of the intrinsic limitations of the L+NL formulation of artificial networks to model vision, the nonlinear behavior of flexible architectures may easily be markedly different from that of the visual system., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). The work of AM was supported by the Spanish government under Grant FJCI-2017–31758. JM has been supported by the Spanish government under the MINECO grant Ref. DPI2017-89867 and by the Generalitat Velanciana grant Ref. GrisoliaP-2019-035. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Performance evaluation of objective quality metrics on HLG-based HDR image coding

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), celebrada a Anaheim, Estats Units d'Amèrica, del 26 al 29 de novembre de 2018., We evaluate the performance of objective quality metrics for high dynamic range (HDR) image coding that uses the transfer function (TF) of the Hybrid Log-Gamma (HLG) method. Previous evaluations of objective metrics for HDR image coding have studied which of them are reliable predictors of perceived quality; however, in those tests, all the non-linear transforms used both for encoding and by the best-performing metrics are essentially very similar and based on visual perception data of detection thresholds for lightness variations. The HLG non-linearity on the other hand is very different, as it is designed for backward compatibility with standard dynamic range (SDR) displays. We test a variety of options for objective metrics, including HLG-based. Our results show that the ranking of metrics for HDR coding changes drastically depending on the TF used for compression., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER, UE)
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

A reevaluation of Whittle (1986, 1992) reveals the link between detection thresholds, discrimination thresholds, and brightness perception

  • Kane, David
  • Bertalmío, Marcelo
In 1986, Paul Whittle investigated the ability to discriminate between the luminance of two small patches viewed upon a uniform background. In 1992, Paul Whittle asked subjects to manipulate the luminance of a number of patches on a uniform background until their brightness appeared to vary from black to white with even steps. The data from the discrimination experiment almost perfectly predicted the gradient of the function obtained in the brightness experiment, indicating that the two experimental methodologies were probing the same underlying mechanism. Whittle introduced a model that was able to capture the pattern of discrimination thresholds and, in turn, the brightness data; however, there were a number of features in the data set that the model couldn't capture. In this paper, we demonstrate that the models of Kane and Bertalmío (2017) and Kingdom and Moulden (1991) may be adapted to predict all the data but only by incorporating an accurate model of detection thresholds. Additionally, we show that a divisive gain model may also capture the data but only by considering polarity-dependent, nonlinear inputs following the underlying pattern of detection thresholds. In summary, we conclude that these models provide a simple link between detection thresholds, discrimination thresholds, and brightness perception., This work has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant reference TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Photorealistic style transfer for cinema shoots

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 Colour and Visual Computing Symposium (CVCS), celebrat del 19 a 20 de setembre de 2018 a Gjøvik, Noruega, Color grading is the process of subtly mixing andadjusting the color and tonal balance of a movie to achievea specific visuallook. This manual editing task may requirea significant amount of work from very skilled artists andtechnicians. In many cases the director wants to emulate thestyle and look present in a reference image, e.g. a still froman existing movie, a photograph, or even a previously shotsequence in the current movie. In this paper we propose amethod that automatically transfers the style, in terms of tone,color palette and contrast, from a reference image to the sourceRAW image. It consists of three separable operations: globalluminance matching, global color transfer and local contrastmatching. The computational complexity of the procedureis extremely low and allows for real-time implementationin-camera. As it just takes into account the statistics of sourceand reference images, no training is required. The results arefree from artifacts and provide an excellent approximation tothe intended look, bringing savings in pre-production, shootingand post-production time., This work has received funding from the European Union’s Horizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Statistics of natural images as a function of dynamic range

  • Grimaldi, Antoine Vincent
  • Kane, David
  • Bertalmío, Marcelo
The statistics of real world images have been extensively investigated, but in virtually all cases using only low dynamic range image databases. The few studies that have considered high dynamic range (HDR) images have performed statistical analyses categorizing images as HDR according to their creation technique, and not to the actual dynamic range of the underlying scene. In this study we demonstrate, using a recent HDR dataset of natural images, that the statistics of the image as received at the camera sensor change dramatically with dynamic range, with particularly strong correlations with dynamic range being observed for the median, standard deviation, skewness, and kurtosis, while the one over frequency relationship for the power spectrum breaks down for images with a very high dynamic range, in practice making HDR images not scale invariant. Effects are also noted in the derivative statistics, the single pixel histograms, and the Haar wavelet analysis. However, we also show that after some basic early transforms occurring within the eye (light scatter, nonlinear photoreceptor response, center-surround modulation) the statistics of the resulting images become virtually independent from the dynamic range, which would allow them to be processed more efficiently by the human visual system., This work has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement Number 761544 (project HDR4EU) and under Grant Agreement Number 780470 (project SAUCE); and by the Spanish government and FEDER Fund, Grantref. TIN2015- 71537-P(MINECO/ FEDER, UE).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

The persistent influence of viewing environment illumination color on displayed image appearance

  • Canham, Trevor
  • Bertalmío, Marcelo
Comunicació presentada a la IS&T London Imaging Meeting, celebrada online del 29 de setembre a l'1 d'octubre de 2020., Chromatic adaptation considering competing influences from emissive displays and ambient illumination is a little studied topic in the context of color management in proportion to its influence on displayed image appearance. An experiment was conducted to identify the degree to which observers adapt to the white point of natural images on an emissive display versus the color of ambient illumination in the room. The responses of observers had no significant difference from those of a previous experiment which was conducted with roughly the same procedure and conditions on a mobile display with a significantly smaller viewing angle. A model is proposed to predict the degree of adaptation values reported by observers. This model has a form such that it can be re-optimized to fit additional data sets for different viewing scenarios and can be used in conjunction with a number of chromatic adaptation transforms., The authors would like to thank all observers for their participation, in particular the colorists from Deluxe Spain who generously lent their time and spectrophotometer to the experiment. This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette ofthe display device where the content will be shown, while preserving the artistic intent of the original content’s creator. In particular in thecinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamutmapping algorithms. In this paper we propose a novel framework that is based on vision science models, performs both gamut reductionand gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-artmethods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamutmapping problem., Thiswork has received funding from the European Union’sHorizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) andunder grant agreement number 780470 (project SAUCE),and by the Spanish government and FEDER Fund, grantref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J.Vazquez-Corral was supported by the Spanish governmentunder Grant IJCI-2014-19516
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Color matching images with unknown non-linear encodings

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
We present a color matching method that, given two different views of the same scene taken by two cameras with unknown settings and unknown internal parameter values, and encoded with unknown non-linear curves, is able to correct the colors of one of the images making it look as if it was captured under the other camera’s settings. Our method is based on treating the in-camera color processing pipeline as a matrix multiplication followed by a non-linearity. This allows us to model a color stabilization transformation among the two shots by estimating several parameters. The method is fast and the results have no spurious colors. It outperforms the state-of-the-art both visually and according to several metrics, and can handle very challenging real-life examples., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Practical use suggests a re-evaluation of HDR objective quality metrics

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: Eleventh International Conference on Quality of Multimedia Experience (QoMEX, 2019), celebrada a Berlín (Alemanya) del 5 al 7 de juny de 2019., Full-reference objective quality metrics for high dynamic range (HDR) images must be validated in terms of their consistency with subjective evaluation results, and previous studies have found a number of metrics that appear to correlate well with the preference of observers. However, those conclusions were based on experiments that do not correspond to the practical use of a current day HDR professional production scenario. In this work, we carry out subjective evaluation experiments for the popular HDR standards of perceptual quantization (PQ) and Hybrid Log-Gamma (HLG), with a state-of-the-art HDR reference monitor used in broadcasting and post-production, and where all the observers participating in the tests are video experts. We find that the ranking of HDR metrics is now substantially different from what was reported earlier, and also that a simple standard dynamic range (SDR) metric can be applied directly to PQ or HLG encoded signals providing excellent results that surpass those of HDR metrics., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER, UE)
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Color stabilization for multi-camera light-field imaging

  • Vu-Thanh, Olivier
  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Gil Rodríguez, Raquel
  • Bertalmío, Marcelo
By capturing a more complete rendition of scene light than standard 2D cameras, light-field technology represents an important step towards closing the gap between live action cinematography and computer graphics. Light-field cameras accomplish this by simultaneously capturing the same scene under different angular configurations, providing directional information that allows for a multitude of post-production effects. Among the practical challenges related to capturing multiple images simultaneously, a very important problem is the fact that the different images do not perfectly match in terms of color, which severely complicates all further processing. In this work we adapt and extend to the light-field scenario a color stabilization method previously proposed for standard multi-camera shoots, and demonstrate experimentally that it provides an improvement over the state-of-the-art techniques for light-field imaging., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

  • Cyriac, Praveen
  • Canham, Trevor
  • Kane, David
  • Bertalmío, Marcelo
Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). We’re very grateful to Albert Pascual, Brett Harrison, Stephane Cattan and everyone at Deluxe-Spain, Alejandro Matus and everyone at Moonlight Barcelona, for their help in fine-tuning and validating our method.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline, and their impact in HDR imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tone-mapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., This work has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Convolutional neural networks deceived by visual illusions

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada al CVPR 2019: Conference on Computer Vision and Pattern Recognition, celebrat del 16 al 20 de juny de 2019 a California, Estats Units d'Amèrica., Visual illusions teach us that what we see is not always what is represented in the physical world. Their special nature make them a fascinating tool to test and validate any new vision model proposed. In general, current vision models are based on the concatenation of linear and non-linear operations. The similarity of this structure with the operations present in Convolutional Neural Networks (CNNs) has motivated us to study if CNNs trained for low-level visual tasks are deceived by visual illusions. In particular, we show that CNNs trained for image denoising, image deblurring, and computational color constancy are able to replicate the human response to visual illusions, and that the extent of this replication varies with respect to variation in architecture and spatial pattern size. These results suggest that in order to obtain CNNs that better replicate human behaviour, we may need to start aiming for them to better replicate visual illusions., This work has received funding from the EU Horizon 2020 programme under grant agreement 761544 (project HDR4EU) and under grant agreement 780470 (project SAUCE) and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Retinal noise emulation: a novel artistic tool for cinema that also improves compression efficiency

  • Zabaleta, Itziar
  • Cámara, Mateo
  • Díaz, César
  • Canham, Trevor
  • García, Narciso
  • Bertalmío, Marcelo
In cinema it is standard practice to improve the appearance of images by adding noise that simulates film grain. This is computationally very costly, so it is only done in post-production and not on the set. It is also limiting because the artists are not able to really experiment with the noise nor introduce novel looks. Furthermore, video compression requires a higher bit rate when the source material has film grain or any other type of high frequency texture. In this work, we introduce a method for adding texture to digital cinema that aims to solve these problems. The proposed algorithm is based on modeling retinal noise, with which the images processed by our method have a natural appearance. This “retinal grain” serves a double purpose. One is aesthetic, as it has parameters that allow to vary widely the resulting texture appearance, which make it an artistic tool for cinematographers. Results are validated through psychophysical experiments in which observers, including cinema professionals, prefer our method over film grain synthesis methods from academia and the industry. The other purpose of the retinal noise emulation method is to improve the quality of compressed video by masking compression artifacts, which allows to lower the encoding bit rate while preserving image quality, and to improve image quality while keeping the bit rate fixed. The effectiveness of our approach for improving coding efficiency, with average bit rate savings of 22.5%, has been validated through psychophysical experiments using professional cinema content shot in 4K, color-graded and where the amount of retinal noise was selected by a motion picture specialist based solely on aesthetic preference., This work was supported in part by the European Union through the Horizon 2020 Research and Innovation Programme (Project HDR4EU) under Grant 761544 and (Project SAUCE) Grant 780470, and in part by the Spanish Government, Ministerio de Economía, Industria y Competitividad (AEI/FEDER), under Project TEC2016-75981 (IVME) and Project PGC2018-099651-B-I00.
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Non-experts or experts? Statistical analyses of MOS using DSIS method

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), celebrada a Barcelona (Espanya) del 4 a 8 de maig de 2020., In image quality assessments, the results of subjective evaluation experiments that use the double-stimulus impairment scale (DSIS) method are often expressed in terms of the mean opinion score (MOS), which is the average score of all subjects for each test condition. Some MOS values are used to derive image quality criteria, and it has been assumed that it is preferable to perform tests with non-expert subjects rather than with experts. In this study, we analyze the results of several subjective evaluation experiments using the DSIS method. Our first contribution is to discuss the statistical meaning of the MOS values, which has not been previously addressed in the literature. Second, our results show that, contrary to the established belief, there are advantages when performing subjective tests with experts, in that they allow experiments to be performed with fewer subjects, and to better determine the lower threshold of image quality., This work received partial funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE) and from the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

In praise of artifice reloaded: Caution with natural image databases in modeling vision

  • Martinez-Garcia, Marina
  • Bertalmío, Marcelo
  • Malo, Jesús
Subjective image quality databases are a major source of raw data on how the visual system works in naturalistic environments. These databases describe the sensitivity of many observers to a wide range of distortions of different nature and intensity seen on top of a variety of natural images. Data of this kind seems to open a number of possibilities for the vision scientist to check the models in realistic scenarios. However, while these natural databases are great benchmarks for models developed in some other way (e.g., by using the well-controlled artificial stimuli of traditional psychophysics), they should be carefully used when trying to fit vision models. Given the high dimensionality of the image space, it is very likely that some basic phenomena are under-represented in the database. Therefore, a model fitted on these large-scale natural databases will not reproduce these under-represented basic phenomena that could otherwise be easily illustrated with well selected artificial stimuli. In this work we study a specific example of the above statement. A standard cortical model using wavelets and divisive normalization tuned to reproduce subjective opinion on a large image quality dataset fails to reproduce basic cross-masking. Here we outline a solution for this problem by using artificial stimuli and by proposing a modification that makes the model easier to tune. Then, we show that the modified model is still competitive in the large-scale database. Our simulations with these artificial stimuli show that when using steerable wavelets, the conventional unit norm Gaussian kernels in divisive normalization should be multiplied by high-pass filters to reproduce basic trends in masking. Basic visual phenomena may be misrepresented in large natural image datasets but this can be solved with model-interpretable stimuli. This is an additional argument in praise of artifice in line with Rust and Movshon (2005)., This work was partially funded by the Spanish and EU FEDER fund through the MINECO/FEDER/EU grants TIN2015-71537-P and DPI2017-89867-C2-2-R; and by the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Matching visual induction effects on screens of different size

  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Mathieu, Elise
  • Bertalmío, Marcelo
22 pags., 13 figs., 3 tabs. -- This work is dedicated to the memory of Jihyun Yeonan-Kim., In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen–size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images., Funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-BI00 (MCIU/AEI/FEDER, UE). JVC received funding from the project PID2019-109628RJI00/AEI/10.13039/501100011033 by the Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) of the Spanish government., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Photorealistic style transfer for video

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
11 pags., 14 figs., This work proposes an efficient example-based photorealistic style transfer algorithm for video. Given a source unprocessed video and a reference image color graded by an artist, we describe an automatic algorithm to transfer the visual style of the reference onto the source footage. Our approach builds upon the color transfer methods based on the statistical properties of images. These methods are fast and of low-complexity, therefore suitable for real-time implementations. Our contribution is to adapt those methods to be used for unprocessed video from cinema cameras, and optionally to incorporate regions of interest previously selected by the user, affecting the final color transfer. The resulting videos are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time. Results are free of spatio-temporal artifacts and the method outperforms diverse state-of-the-art methods according to observer preference experiments., This work has received funding from the European Union’s Hori- zon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE)
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Cortical-Inspired Wilson-Cowan-Type Equations for Orientation-Dependent Contrast Perception Modelling

  • Bertalmío, Marcelo
  • Calatroni, Luca
  • Franceschi, Valentina
  • Franceschiello, Bendetta
  • Prandi, Dario
19 pags., 17 figs., We consider the evolution model proposed in Bertalmío (Front Comput Neurosci 8:71, 2014), Bertalmío et al. (IEEE Trans Image Process 16(4):1058–1072, 2007) to describe illusory contrast perception phenomena induced by surrounding orientations. Firstly, we highlight its analogies and differences with the widely used Wilson–Cowan equations (Wilson and Cowan in BioPhys J 12(1):1–24, 1972), mainly in terms of efficient representation properties. Then, in order to explicitly encode local directional information, we exploit the model of the primary visual cortex (V1) proposed in Citti and Sarti (J Math Imaging Vis 24(3):307–326, 2006) and largely used over the last years for several image processing problems (Duits and Franken in Q Appl Math 68(2):255–292, 2010; Prandi and Gauthier in A semidiscrete version of the Petitot model as a plausible model for anthropomorphic image reconstruction and pattern recognition. SpringerBriefs in Mathematics, Springer, Cham, 2017; Franceschiello et al. in J Math Imaging Vis 60(1):94–108, 2018). The resulting model is thus defined in the space of positions and orientation, and it is capable of describing assimilation and contrast visual bias at the same time. We report several numerical tests showing the ability of the model to reproduce, in particular, orientation-dependent phenomena such as grating induction and a modified version of the Poggendorff illusion. For this latter example, we empirically show the existence of a set of threshold parameters differentiating from inpainting to perception-type reconstructions and describing long-range connectivity between different hypercolumns in V1., M. B. acknowledges the support of the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 761544 (Project HDR4EU) and under Grant Agreement No. 780470 (Project SAUCE), and of the Spanish government and FEDER Fund, Grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). L. C., V. F. and D. P. acknowledge the support of a public grant overseen by the French National Research Agency (ANR) as part of the Investissement d’avenir program, through the iCODE project funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02 and of the research project LiftME funded by INS2I, CNRS. V. F. acknowledges the support received from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant No. 794592 and from the INdAM project Problemi isoperimetrici in spazi Euclidei e non. V. F. and D. P. also acknowledge the support of ANR-15-CE40- 0018 project SRGI - Sub-Riemannian Geometry and Interactions. B. F. acknowledges the support of the Fondation Asile des Aveugles.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

  • Cyriac, Praveen
  • Canham, Trevor
  • Kane, David
  • Bertalmío, Marcelo
27 pags., 16 figs., Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, S.W.
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
14 pags., 14 figs., Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette of the display device where the content will be shown, while preserving the artistic intent of the original content’s creator. In particular, in the cinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamut mapping algorithms. In this article, we propose a novel framework that is based on vision science models, performs both gamut reduction and gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-art methods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamut mapping problem., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant agreement number 761544 (project HDR4EU) and under Grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Performance evaluation of objective quality metrics on HLG-based HDR image coding

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), celebrada a Anaheim, Estats Units d'Amèrica, del 26 al 29 de novembre de 2018., We evaluate the performance of objective quality metrics for high dynamic range (HDR) image coding that uses the transfer function (TF) of the Hybrid Log-Gamma (HLG) method. Previous evaluations of objective metrics for HDR image coding have studied which of them are reliable predictors of perceived quality; however, in those tests, all the non-linear transforms used both for encoding and by the best-performing metrics are essentially very similar and based on visual perception data of detection thresholds for lightness variations. The HLG non-linearity on the other hand is very different, as it is designed for backward compatibility with standard dynamic range (SDR) displays. We test a variety of options for objective metrics, including HLG-based. Our results show that the ranking of metrics for HDR coding changes drastically depending on the TF used for compression., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER, UE)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Color stabilization for multi-camera light-field imaging

  • Vu-Thanh, Olivier
  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Gil Rodríguez, Raquel
  • Bertalmío, Marcelo
By capturing a more complete rendition of scene light than standard 2D cameras, light-field technology represents an important step towards closing the gap between live action cinematography and computer graphics. Light-field cameras accomplish this by simultaneously capturing the same scene under different angular configurations, providing directional information that allows for a multitude of post-production effects. Among the practical challenges related to capturing multiple images simultaneously, a very important problem is the fact that the different images do not perfectly match in terms of color, which severely complicates all further processing. In this work we adapt and extend to the light-field scenario a color stabilization method previously proposed for standard multi-camera shoots, and demonstrate experimentally that it provides an improvement over the state-of-the-art techniques for light-field imaging., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The Potential of light fields in media productions

  • Trottnow, Jonas
  • Spielmann, Simon
  • Lange, Tobias
  • Chelli, Kelvin
  • Solony, Marek
  • Smrž, Pavel
  • Zemčík, Pavel
  • Aenchbacher, Weston
  • Grogan, Mairéad
  • Alain, Martin
  • Smolic, Aljosa
  • Canham, Trevor
  • Vu-Thanh, Olivier
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: The 12th ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, celebrada del 17 al 20 de novembre de 2019 a Brisbane, Austràlia., One aspect of the EU funded project SAUCE is to explore the pos-sibilities and challenges of integrating light field capturing andprocessing into media productions. A special light field camera wasbuild by Saarland University [Herfet et al.2018] and is first testedunder production conditions in the test production “Unfolding” aspart of the SAUCE project. Filmakademie Baden-Württemberg de-veloped the contentual frame, executed the post-production and pre-pared a complete previsualization. Calibration and post-processingalgorithms are developed by the Trinity College Dublin and theBrno University of Technology. This document describes challengesduring building and shooting with the light field camera array, aswell as its potential and challenges for the post-production., The project has received funding from the European Union’s Hori-zon 2020 Research and Innovation Programme under Grant Agree-ment No. 780470.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Physical-based optimization for non-physical image dehazing methods

  • Vazquez-Corral, Javier
  • Finlayson, Graham D.
  • Bertalmío, Marcelo
Images captured under hazy conditions (e.g. fog, air pollution) usually present faded colors and loss of contrast. To improve their visibility, a process called image dehazing can be applied. Some of the most successful image dehazing algorithms are based on image processing methods but do not follow any physical image formation model, which limits their performance. In this paper, we propose a post-processing technique to alleviate this handicap by enforcing the original method to be consistent with a popular physical model for image formation under haze. Our results improve upon those of the original methods qualitatively and according to several metrics, and they have also been validated via psychophysical experiments. These results are particularly striking in terms of avoiding over-saturation and reducing color artifacts, which are the most common shortcomings faced by image dehazing methods., Horizon 2020 Framework Programme (761544, 780470); Engineering and Physical Sciences Research Council (EP/028730, EP/M001768); Spanish Government MINECO and Feder Fund (PGC2018-099651-B-I00).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Matching visual induction effects on screens of different size

  • Canham, Trevor
  • Vazquez-Corral, Javier
  • Mathieu, Elise
  • Bertalmío, Marcelo
In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen–size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images., Funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). JVC received funding from the project PID2019-109628RJ-I00/AEI/10.13039/501100011033 by the Ministerio de Ciencia, Innovación y Universidades (MCIU) and the Agencia Estatal de Investigación (AEI) of the Spanish government.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A reevaluation of Whittle (1986, 1992) reveals the link between detection thresholds, discrimination thresholds, and brightness perception

  • Kane, David
  • Bertalmío, Marcelo
In 1986, Paul Whittle investigated the ability to discriminate between the luminance of two small patches viewed upon a uniform background. In 1992, Paul Whittle asked subjects to manipulate the luminance of a number of patches on a uniform background until their brightness appeared to vary from black to white with even steps. The data from the discrimination experiment almost perfectly predicted the gradient of the function obtained in the brightness experiment, indicating that the two experimental methodologies were probing the same underlying mechanism. Whittle introduced a model that was able to capture the pattern of discrimination thresholds and, in turn, the brightness data; however, there were a number of features in the data set that the model couldn't capture. In this paper, we demonstrate that the models of Kane and Bertalmío (2017) and Kingdom and Moulden (1991) may be adapted to predict all the data but only by incorporating an accurate model of detection thresholds. Additionally, we show that a divisive gain model may also capture the data but only by considering polarity-dependent, nonlinear inputs following the underlying pattern of detection thresholds. In summary, we conclude that these models provide a simple link between detection thresholds, discrimination thresholds, and brightness perception., This work has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant reference TIN2015-71537-P (MINECO/FEDER,UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Photorealistic style transfer for cinema shoots

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 Colour and Visual Computing Symposium (CVCS), celebrat del 19 a 20 de setembre de 2018 a Gjøvik, Noruega, Color grading is the process of subtly mixing andadjusting the color and tonal balance of a movie to achievea specific visuallook. This manual editing task may requirea significant amount of work from very skilled artists andtechnicians. In many cases the director wants to emulate thestyle and look present in a reference image, e.g. a still froman existing movie, a photograph, or even a previously shotsequence in the current movie. In this paper we propose amethod that automatically transfers the style, in terms of tone,color palette and contrast, from a reference image to the sourceRAW image. It consists of three separable operations: globalluminance matching, global color transfer and local contrastmatching. The computational complexity of the procedureis extremely low and allows for real-time implementationin-camera. As it just takes into account the statistics of sourceand reference images, no training is required. The results arefree from artifacts and provide an excellent approximation tothe intended look, bringing savings in pre-production, shootingand post-production time., This work has received funding from the European Union’s Horizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Photorealistic style transfer for video

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
This work proposes an efficient example-based photorealistic style transfer algorithm for video. Given a source unprocessed video and a reference image color graded by an artist, we describe an automatic algorithm to transfer the visual style of the reference onto the source footage. Our approach builds upon the color transfer methods based on the statistical properties of images. These methods are fast and of low-complexity, therefore suitable for real-time implementations. Our contribution is to adapt those methods to be used for unprocessed video from cinema cameras, and optionally to incorporate regions of interest previously selected by the user, affecting the final color transfer. The resulting videos are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time. Results are free of spatio-temporal artifacts and the method outperforms diverse state-of-the-art methods according to observer preference experiments., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline and their impact in hdr imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tonemapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., The work of the authors was supported by the European Unions Horizon 2020 research and innovation programme grant 761544 (project HDR4EU) and grant 780470 (project SAUCE) and by the Spanish government and FEDER fund grant TIN2015-71537-P (MINECO/FEDER,UE). The work of the second author was supported by the Spanish government under grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Cortical-inspired Wilson–Cowan-type equations for orientation-dependent contrast perception modelling

  • Bertalmío, Marcelo
  • Calatroni, Luca
  • Franceschi, Valentina
  • Franceschiello, Benedetta
  • Prandi, Dario
We consider the evolution model proposed in Bertalmío (Front Comput Neurosci 8:71, 2014), Bertalmío et al. (IEEE Trans Image Process 16(4):1058–1072, 2007) to describe illusory contrast perception phenomena induced by surrounding orientations. Firstly, we highlight its analogies and differences with the widely used Wilson–Cowan equations (Wilson and Cowan in BioPhys J 12(1):1–24, 1972), mainly in terms of efficient representation properties. Then, in order to explicitly encode local directional information, we exploit the model of the primary visual cortex (V1) proposed in Citti and Sarti (J Math Imaging Vis 24(3):307–326, 2006) and largely used over the last years for several image processing problems (Duits and Franken in Q Appl Math 68(2):255–292, 2010; Prandi and Gauthier in A semidiscrete version of the Petitot model as a plausible model for anthropomorphic image reconstruction and pattern recognition. SpringerBriefs in Mathematics, Springer, Cham, 2017; Franceschiello et al. in J Math Imaging Vis 60(1):94–108, 2018). The resulting model is thus defined in the space of positions and orientation, and it is capable of describing assimilation and contrast visual bias at the same time. We report several numerical tests showing the ability of the model to reproduce, in particular, orientation-dependent phenomena such as grating induction and a modified version of the Poggendorff illusion. For this latter example, we empirically show the existence of a set of threshold parameters differentiating from inpainting to perception-type reconstructions and describing long-range connectivity between different hypercolumns in V1., The authors acknowledge the anonymous referees for their suggestions which improved significantly the quality of their manuscript. M. B. acknowledges the support of the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 761544 (Project HDR4EU) and under Grant Agreement No. 780470 (Project SAUCE), and of the Spanish government and FEDER Fund, Grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). L. C., V. F. and D. P. acknowledge the support of a public grant overseen by the French National Research Agency (ANR) as part of the Investissement d’avenir program, through the iCODE project funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02 and of the research project LiftME funded by INS2I, CNRS. V. F. acknowledges the support received from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant No. 794592 and from the INdAM project Problemi isoperimetrici in spazi Euclidei e non. V. F. and D. P. also acknowledge the support of ANR-15-CE40-0018 project SRGI - Sub-Riemannian Geometry and Interactions. B. F. acknowledges the support of the Fondation Asile des Aveugles.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Practical use suggests a re-evaluation of HDR objective quality metrics

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: Eleventh International Conference on Quality of Multimedia Experience (QoMEX, 2019), celebrada a Berlín (Alemanya) del 5 al 7 de juny de 2019., Full-reference objective quality metrics for high dynamic range (HDR) images must be validated in terms of their consistency with subjective evaluation results, and previous studies have found a number of metrics that appear to correlate well with the preference of observers. However, those conclusions were based on experiments that do not correspond to the practical use of a current day HDR professional production scenario. In this work, we carry out subjective evaluation experiments for the popular HDR standards of perceptual quantization (PQ) and Hybrid Log-Gamma (HLG), with a state-of-the-art HDR reference monitor used in broadcasting and post-production, and where all the observers participating in the tests are video experts. We find that the ranking of HDR metrics is now substantially different from what was reported earlier, and also that a simple standard dynamic range (SDR) metric can be applied directly to PQ or HLG encoded signals providing excellent results that surpass those of HDR metrics., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER, UE)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

The persistent influence of viewing environment illumination color on displayed image appearance

  • Canham, Trevor
  • Bertalmío, Marcelo
Comunicació presentada a la IS&T London Imaging Meeting, celebrada online del 29 de setembre a l'1 d'octubre de 2020., Chromatic adaptation considering competing influences from emissive displays and ambient illumination is a little studied topic in the context of color management in proportion to its influence on displayed image appearance. An experiment was conducted to identify the degree to which observers adapt to the white point of natural images on an emissive display versus the color of ambient illumination in the room. The responses of observers had no significant difference from those of a previous experiment which was conducted with roughly the same procedure and conditions on a mobile display with a significantly smaller viewing angle. A model is proposed to predict the degree of adaptation values reported by observers. This model has a form such that it can be re-optimized to fit additional data sets for different viewing scenarios and can be used in conjunction with a number of chromatic adaptation transforms., The authors would like to thank all observers for their participation, in particular the colorists from Deluxe Spain who generously lent their time and spectrophotometer to the experiment. This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline, and their impact in HDR imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tone-mapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., This work has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Non-experts or experts? Statistical analyses of MOS using DSIS method

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), celebrada a Barcelona (Espanya) del 4 a 8 de maig de 2020., In image quality assessments, the results of subjective evaluation experiments that use the double-stimulus impairment scale (DSIS) method are often expressed in terms of the mean opinion score (MOS), which is the average score of all subjects for each test condition. Some MOS values are used to derive image quality criteria, and it has been assumed that it is preferable to perform tests with non-expert subjects rather than with experts. In this study, we analyze the results of several subjective evaluation experiments using the DSIS method. Our first contribution is to discuss the statistical meaning of the MOS values, which has not been previously addressed in the literature. Second, our results show that, contrary to the established belief, there are advantages when performing subjective tests with experts, in that they allow experiments to be performed with fewer subjects, and to better determine the lower threshold of image quality., This work received partial funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE) and from the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Color matching images with unknown non-linear encodings

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
We present a color matching method that, given two different views of the same scene taken by two cameras with unknown settings and unknown internal parameter values, and encoded with unknown non-linear curves, is able to correct the colors of one of the images making it look as if it was captured under the other camera’s settings. Our method is based on treating the in-camera color processing pipeline as a matrix multiplication followed by a non-linearity. This allows us to model a color stabilization transformation among the two shots by estimating several parameters. The method is fast and the results have no spurious colors. It outperforms the state-of-the-art both visually and according to several metrics, and can handle very challenging real-life examples., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette ofthe display device where the content will be shown, while preserving the artistic intent of the original content’s creator. In particular in thecinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamutmapping algorithms. In this paper we propose a novel framework that is based on vision science models, performs both gamut reductionand gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-artmethods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamutmapping problem., Thiswork has received funding from the European Union’sHorizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) andunder grant agreement number 780470 (project SAUCE),and by the Spanish government and FEDER Fund, grantref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J.Vazquez-Corral was supported by the Spanish governmentunder Grant IJCI-2014-19516
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

In praise of artifice reloaded: caution with natural image databases in modeling vision

  • Martínez García, Marina
  • Bertalmío, Marcelo
  • Malo, Jesús
Subjective image quality databases are a major source of raw data on how the visual system works in naturalistic environments. These databases describe the sensitivity of many observers to a wide range of distortions of different nature and intensity seen on top of a variety of natural images. Data of this kind seems to open a number of possibilities for the vision scientist to check the models in realistic scenarios. However, while these natural databases are great benchmarks for models developed in some other way (e.g., by using the well-controlled artificial stimuli of traditional psychophysics), they should be carefully used when trying to fit vision models. Given the high dimensionality of the image space, it is very likely that some basic phenomena are under-represented in the database. Therefore, a model fitted on these large-scale natural databases will not reproduce these under-represented basic phenomena that could otherwise be easily illustrated with well selected artificial stimuli. In this work we study a specific example of the above statement. A standard corticalmodel using wavelets and divisive normalization tuned to reproduce subjective opinion on a large image quality dataset fails to reproduce basic cross-masking. Here we outline a solution for this problem by using artificial stimuli and by proposing a modification that makes the model easier to tune. Then, we show that the modified model is still competitive in the large-scale database. Our simulations with these artificial stimuli show that when using steerable wavelets, the conventional unit norm Gaussian kernels in divisive normalization should be multiplied by high-pass filters to reproduce basic trends in masking. Basic visual phenomena may be misrepresented in large natural image datasets but this can be solved with model-interpretable stimuli. This is an additional argument in praise of artifice in line with Rust and Movshon (2005)., This work was partially funded by the Spanish and EU FEDER fund through the MINECO/FEDER/EU grants TIN2015-71537- P and DPI2017-89867-C2-2-R; and by the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Statistics of natural images as a function of dynamic range

  • Grimaldi, Antoine Vincent
  • Kane, David
  • Bertalmío, Marcelo
The statistics of real world images have been extensively investigated, but in virtually all cases using only low dynamic range image databases. The few studies that have considered high dynamic range (HDR) images have performed statistical analyses categorizing images as HDR according to their creation technique, and not to the actual dynamic range of the underlying scene. In this study we demonstrate, using a recent HDR dataset of natural images, that the statistics of the image as received at the camera sensor change dramatically with dynamic range, with particularly strong correlations with dynamic range being observed for the median, standard deviation, skewness, and kurtosis, while the one over frequency relationship for the power spectrum breaks down for images with a very high dynamic range, in practice making HDR images not scale invariant. Effects are also noted in the derivative statistics, the single pixel histograms, and the Haar wavelet analysis. However, we also show that after some basic early transforms occurring within the eye (light scatter, nonlinear photoreceptor response, center-surround modulation) the statistics of the resulting images become virtually independent from the dynamic range, which would allow them to be processed more efficiently by the human visual system., This work has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement Number 761544 (project HDR4EU) and under Grant Agreement Number 780470 (project SAUCE); and by the Spanish government and FEDER Fund, Grantref. TIN2015- 71537-P(MINECO/ FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

  • Cyriac, Praveen
  • Canham, Trevor
  • Kane, David
  • Bertalmío, Marcelo
Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). We’re very grateful to Albert Pascual, Brett Harrison, Stephane Cattan and everyone at Deluxe-Spain, Alejandro Matus and everyone at Moonlight Barcelona, for their help in fine-tuning and validating our method.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Color illusions also deceive CNNs for low-level vision tasks: analysis and implications

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
  • Malo, Jesús
The study of visual illusions has proven to be a very useful approach in vision science. In this work we start by showing that, while convolutional neural networks (CNNs) trained for low-level visual tasks in natural images may be deceived by brightness and color illusions, some network illusions can be inconsistent with the perception of humans. Next, we analyze where these similarities and differences may come from. On one hand, the proposed linear eigenanalysis explains the overall similarities: in simple CNNs trained for tasks like denoising or deblurring, the linear version of the network has center-surround receptive fields, and global transfer functions are very similar to the human achromatic and chromatic contrast sensitivity functions in human-like opponent color spaces. These similarities are consistent with the long-standing hypothesis that considers low-level visual illusions as a by-product of the optimization to natural environments. Specifically, here human-like features emerge from error minimization. On the other hand, the observed differences must be due to the behavior of the human visual system not explained by the linear approximation. However, our study also shows that more ‘flexible’ network architectures, with more layers and a higher degree of nonlinearity, may actually have a worse capability of reproducing visual illusions. This implies, in line with other works in the vision science literature, a word of caution on using CNNs to study human vision: on top of the intrinsic limitations of the L+NL formulation of artificial networks to model vision, the nonlinear behavior of flexible architectures may easily be markedly different from that of the visual system., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant Ref. PGC2018-099651-B-I00 (MCIU/AEI/FEDER, UE). The work of AM was supported by the Spanish government under Grant FJCI-2017–31758. JM has been supported by the Spanish government under the MINECO grant Ref. DPI2017-89867 and by the Generalitat Velanciana grant Ref. GrisoliaP-2019-035. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Convolutional neural networks deceived by visual illusions

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada al CVPR 2019: Conference on Computer Vision and Pattern Recognition, celebrat del 16 al 20 de juny de 2019 a California, Estats Units d'Amèrica., Visual illusions teach us that what we see is not always what is represented in the physical world. Their special nature make them a fascinating tool to test and validate any new vision model proposed. In general, current vision models are based on the concatenation of linear and non-linear operations. The similarity of this structure with the operations present in Convolutional Neural Networks (CNNs) has motivated us to study if CNNs trained for low-level visual tasks are deceived by visual illusions. In particular, we show that CNNs trained for image denoising, image deblurring, and computational color constancy are able to replicate the human response to visual illusions, and that the extent of this replication varies with respect to variation in architecture and spatial pattern size. These results suggest that in order to obtain CNNs that better replicate human behaviour, we may need to start aiming for them to better replicate visual illusions., This work has received funding from the EU Horizon 2020 programme under grant agreement 761544 (project HDR4EU) and under grant agreement 780470 (project SAUCE) and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Retinal noise emulation: a novel artistic tool for cinema that also improves compression efficiency

  • Zabaleta, Itziar
  • Cámara, Mateo
  • Díaz, César
  • Canham, Trevor
  • García, Narciso
  • Bertalmío, Marcelo
In cinema it is standard practice to improve the appearance of images by adding noise that simulates film grain. This is computationally very costly, so it is only done in post-production and not on the set. It is also limiting because the artists are not able to really experiment with the noise nor introduce novel looks. Furthermore, video compression requires a higher bit rate when the source material has film grain or any other type of high frequency texture. In this work, we introduce a method for adding texture to digital cinema that aims to solve these problems. The proposed algorithm is based on modeling retinal noise, with which the images processed by our method have a natural appearance. This “retinal grain” serves a double purpose. One is aesthetic, as it has parameters that allow to vary widely the resulting texture appearance, which make it an artistic tool for cinematographers. Results are validated through psychophysical experiments in which observers, including cinema professionals, prefer our method over film grain synthesis methods from academia and the industry. The other purpose of the retinal noise emulation method is to improve the quality of compressed video by masking compression artifacts, which allows to lower the encoding bit rate while preserving image quality, and to improve image quality while keeping the bit rate fixed. The effectiveness of our approach for improving coding efficiency, with average bit rate savings of 22.5%, has been validated through psychophysical experiments using professional cinema content shot in 4K, color-graded and where the amount of retinal noise was selected by a motion picture specialist based solely on aesthetic preference., This work was supported in part by the European Union through the Horizon 2020 Research and Innovation Programme (Project HDR4EU) under Grant 761544 and (Project SAUCE) Grant 780470, and in part by the Spanish Government, Ministerio de Economía, Industria y Competitividad (AEI/FEDER), under Project TEC2016-75981 (IVME) and Project PGC2018-099651-B-I00.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

In-camera, photorealistic style transfer for on-set automatic grading

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a la SMPTE 2018 Annual Technical Conference of the Society of Motion Picture and Television Engineers, celebrada a Los Angeles (Estats Units d'Amèrica) del 22 a 25 de octubre de 2018., In professional cinema, the intended artistic look of the movie informs the creation of a static 3D LUT that is applied on set, where further manual modifications to the image appearance are registered as 10-parameter transforms in a color decision list (CDL). The original RAW footage and its corresponding LUT and CDL are passedon to the post-production stage where the fine-tuning of the final look is performed during color grading. — In many cases, the director wants to emulate the style and look present in a reference image, e.g. a still from an existing movie, or a photograph, or a painting, or even a frame from a previously shot sequence in the current movie. The manual creation of a LUT and CDL for this purpose may require a significant amount of work from very skilled artists and technicians, while the state of the art in the academic literature offers promising but partial solutions to the photorealistic style transfer problem, with limitations regarding artifacts, speed and manual interaction. — In this paper, we propose a method that automatically transfers the style, in terms of luminance, color palette and contrast, from a reference image to the source raw footage. It consists of three separable operations: global luminance matching, global color transfer and local contrast matching. As it just takes into account the statistics of source and reference images, no training is required. The total transform is not static but adapts to the changes in the source footage. The computational complexity of the procedure is extremely low and allows for real-time implementation in-camera, for on-set monitoring. While the method is proposed as a substitute for the need to specify a LUT and a CDL, it's compatible with further refinements performed via LUTs, CDLs and grading, both on-set and in post-production. The results are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time., This work has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Matching visual induction effects on screens of different size

  • Canham, Trevor|||0000-0003-4830-4786
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Mathieu, Elise
  • Bertalmío, Marcelo|||0000-0002-1023-8325
In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen-size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images.