Pasar al contenido principal

MODELOS DE VISION DE CONTRASTE Y COLOR PARA CINE

TIN2015-71537-P

•
Nombre agencia financiadora Ministerio de Economía y Competitividad
Acrónimo agencia financiadora MINECO
Programa Programa Estatal de Fomento de la Investigación Científica y Técnica de Excelencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D (Excelencia / Generación de Conocimiento)
Año convocatoria 2015
Plan Estatal 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Unidad de gestión Secretaría de Estado de Educación, Formación Profesional y Universidades
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT
Identificador persistente http://dx.doi.org/10.13039/501100003329

Resultados relacionados

Resultados totales (Incluyendo duplicados): 59
Encontrada(s) 1 página(s)

Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Color Matching Images with Unknown Non-Linear Encodings

  • Gil Rodríguez, Raquel|||0000-0001-5561-1990
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Bertalmío, Marcelo|||0000-0002-1023-8325
We present a color matching method that deals with different non-linear encodings. In particular, given two different views of the same scene taken by two cameras with unknown settings and internal parameters, and encoded with unknown non-linear curves, our method is able to correct the colors of one of the images making it look as if it was captured under the other camera's settings. Our method is based on treating the in-camera color processing pipeline as a concatenation of a matrix multiplication on the linear image followed by a non-linearity. This allows us to model a color stabilization transformation among the two shots by estimating a single matrix -that will contain information from both of the original images- and an extra parameter that complies with the non-linearity. The method is fast and the results have no spurious colors. It outperforms the state-of-the-art both visually and according to several metrics, and can handle HDR encodings and very challenging real-life examples.
Dipòsit Digital de Documents de la UAB
Publicación Capítulo o parte de un libro (BookPart)

Convolutional neural networks can be deceived by visual illusions

  • Gómez Villa, Alexandra|||0000-0003-0469-3425
  • Martín, Adrián|||0000-0001-7127-0073
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Bertalmío, Marcelo|||0000-0002-1023-8325
Visual illusions teach us that what we see is not always what is represented in the physical world. Their special nature make them a fascinating tool to test and validate any new vision model proposed. In general, current vision models are based on the concatenation of linear and non-linear operations. The similarity of this structure with the operations present in Convolutional Neural Networks (CNNs) has motivated us to study if CNNs trained for low-level visual tasks are deceived by visual illusions. In particular, we show that CNNs trained for image denoising, image deblurring, and computational color constancy are able to replicate the human response to visual illusions, and that the extent of this replication varies with respect to variation in architecture and spatial pattern size. These results suggest that in order to obtain CNNs that better replicate human behaviour, we may need to start aiming for them to better replicate visual illusions.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

System gamma as a function of image- and monitor-dynamic range

  • Kane, David
  • Bertalmío, Marcelo
System gamma is the end-to-end exponent that describes the relationship between the relative luminance values at capture and the reproduced image. The system gamma preferred by subjects is known to vary with the background luminance condition and the image in question. We confirm the previous two findings using an image database with both high and low dynamic range images (from 102 to 107), but also find that the preferred system gamma varies with the dynamic range of the monitor (CRT, LCD, or OLED). We find that the preferred system gamma can be predicted in all conditions and for all images by a simple model that searches for the value that best flattens the lightness distribution, where lightness is modeled as a power law of onscreen luminance. To account for the data, the exponent must vary with the viewing conditions. The method presented allows the inference of lightness perception in natural scenes without direct measurement and makes testable predictions for how lightness perception varies with the viewing condition and the distribution of luminance values in a scene. The data from this paper has been made available online., This work was supported by the European Research Council, Starting (Grant ref. 306337), the Spanish government (grant ref. TIN2015-71537-P), and by the Icrea Academia Award.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Analysis of retinal and cortical components of Retinex algorithms

  • Yeonan-Kim, Jihyun
  • Bertalmío, Marcelo
Following Land and McCann’s first proposal of the Retinex theory, numerous Retinex algorithms that differ considerably both algorithmically and functionally have been developed. We clarify the relationships among various Retinex families by associating their spatial processing structures to the neural organizations in the retina and the primary visual cortex in the brain. Some of the Retinex algorithms have a retina-like processing structure (Land’s designator idea and NASA Retinex), and some show a close connection with the cortical structures in the primary visual area of the brain (two-dimensional L&M Retinex). A third group of Retinexes (the variational Retinex) manifests an explicit algorithmic relation to Wilson–Cowan’s physiological model. We intend to overview these three groups of Retinexes with the frame of reference in the biological visual mechanisms., This work was supported by the European Research Council, Starting Grant Ref. 306337, by the Spanish government and FEDER Fund, grant Ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The authors declare that no conflicting interests exist.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Gamut mapping for visual attention retargeting

  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Visual attention retargeting attempts to modify an image such that the viewer’s attention is directed to specific regions. Goals include highlighting a particular object or hiding possible problems in the image. In this work, we show that we can pose the visual retargeting problem in terms of gamut mapping. In short, visual attention retargeting can be achieved by performing gamut extension in those regions that we want to highlight and gamut reduction in the other regions., Authors are supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014- 19516.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Gamut reduction through local saturation reduction

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award. The work of Javier Vazquez-Corral was supported by the Spanish government grant IJCI-2014-19516.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Retinal lateral inhibition provides the biological basis of long-range spatial induction

  • Yeonan-Kim, Jihyun
  • Bertalmío, Marcelo
Retinal lateral inhibition is one of the conventional efficient coding mechanisms in the visual system that is produced by interneurons that pool signals over a neighborhood of presynaptic feedforward cells and send inhibitory signals back to them. Thus, the receptive-field (RF) of a retinal ganglion cell has a center-surround receptive-field (RF) profile that is classically represented as a difference-of-Gaussian (DOG) adequate for efficient spatial contrast coding. The DOG RF profile has been attributed to produce the psychophysical phenomena of brightness induction, in which the perceived brightness of an object is affected by that of its vicinity, either shifting away from it (brightness contrast) or becoming more similar to it (brightness assimilation) depending on the size of the surfaces surrounding the object. While brightness contrast can be modeled using a DOG with a narrow surround, brightness assimilation requires a wide suppressive surround. Early retinal studies determined that the suppressive surround of a retinal ganglion cell is narrow (< 100–300 μm; ‘classic RF’), which led researchers to postulate that brightness assimilation must originate at some post-retinal, possibly cortical, stage where long-range interactions are feasible. However, more recent studies have reported that the retinal interneurons also exhibit a spatially wide component (> 500–1000 μm). In the current study, we examine the effect of this wide interneuron RF component in two biophysical retinal models and show that for both of the retinal models it explains the long-range effect evidenced in simultaneous brightness induction phenomena and that the spatial extent of this long-range effect of the retinal model responses matches that of perceptual data. These results suggest that the retinal lateral inhibition mechanism alone can regulate local as well as long-range spatial induction through the narrow and wide RF components of retinal interneurons, arguing against the existing view that spatial induction is operated by two separate local vs. long-range mechanisms., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline and their impact in hdr imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tonemapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., The work of the authors was supported by the European Unions Horizon 2020 research and innovation programme grant 761544 (project HDR4EU) and grant 780470 (project SAUCE) and by the Spanish government and FEDER fund grant TIN2015-71537-P (MINECO/FEDER,UE). The work of the second author was supported by the Spanish government under grant IJCI-2014-19516.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

In-camera, photorealistic style transfer for on-set automatic grading

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a la SMPTE 2018 Annual Technical Conference of the Society of Motion Picture and Television Engineers, celebrada a Los Angeles (Estats Units d'Amèrica) del 22 a 25 de octubre de 2018., In professional cinema, the intended artistic look of the movie informs the creation of a static 3D LUT that is applied on set, where further manual modifications to the image appearance are registered as 10-parameter transforms in a color decision list (CDL). The original RAW footage and its corresponding LUT and CDL are passedon to the post-production stage where the fine-tuning of the final look is performed during color grading. — In many cases, the director wants to emulate the style and look present in a reference image, e.g. a still from an existing movie, or a photograph, or a painting, or even a frame from a previously shot sequence in the current movie. The manual creation of a LUT and CDL for this purpose may require a significant amount of work from very skilled artists and technicians, while the state of the art in the academic literature offers promising but partial solutions to the photorealistic style transfer problem, with limitations regarding artifacts, speed and manual interaction. — In this paper, we propose a method that automatically transfers the style, in terms of luminance, color palette and contrast, from a reference image to the source raw footage. It consists of three separable operations: global luminance matching, global color transfer and local contrast matching. As it just takes into account the statistics of source and reference images, no training is required. The total transform is not static but adapts to the changes in the source footage. The computational complexity of the procedure is extremely low and allows for real-time implementation in-camera, for on-set monitoring. While the method is proposed as a substitute for the need to specify a LUT and a CDL, it's compatible with further refinements performed via LUTs, CDLs and grading, both on-set and in post-production. The results are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time., This work has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Perceptually-based gamut extension algorithm for emerging wide color gamut display and projection technologies

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: SMPTE 2016 Annual Technical Conference and Exhibition, celebrada a Los Angeles, Estats Units d'Amèrica, del 25 al 27 d'octubre de 2016., While wide color gamut (WCG) capabilities are a key element of emerging display and projection technologies, at present most image content is recorded using standards such as DCI-P3 for cinema or BT.709 for TV that have a reduced color gamut. Therefore, there is a need for gamut extension methods that process regular content and allow to appreciate the full color potential of new displays, improving user experience. We present a gamut extension algorithm that is based on visual perception models and explicitly takes into account low chromatic colors such as skin tones. It produces results that look natural, are free of artifacts of any kind, and outperform the state of the art. The method is fast, allowing for operation interaction if needed., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Performance evaluation of objective quality metrics on HLG-based HDR image coding

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), celebrada a Anaheim, Estats Units d'Amèrica, del 26 al 29 de novembre de 2018., We evaluate the performance of objective quality metrics for high dynamic range (HDR) image coding that uses the transfer function (TF) of the Hybrid Log-Gamma (HLG) method. Previous evaluations of objective metrics for HDR image coding have studied which of them are reliable predictors of perceived quality; however, in those tests, all the non-linear transforms used both for encoding and by the best-performing metrics are essentially very similar and based on visual perception data of detection thresholds for lightness variations. The HLG non-linearity on the other hand is very different, as it is designed for backward compatibility with standard dynamic range (SDR) displays. We test a variety of options for objective metrics, including HLG-based. Our results show that the ranking of metrics for HDR coding changes drastically depending on the TF used for compression., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER, UE)
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

A reevaluation of Whittle (1986, 1992) reveals the link between detection thresholds, discrimination thresholds, and brightness perception

  • Kane, David
  • Bertalmío, Marcelo
In 1986, Paul Whittle investigated the ability to discriminate between the luminance of two small patches viewed upon a uniform background. In 1992, Paul Whittle asked subjects to manipulate the luminance of a number of patches on a uniform background until their brightness appeared to vary from black to white with even steps. The data from the discrimination experiment almost perfectly predicted the gradient of the function obtained in the brightness experiment, indicating that the two experimental methodologies were probing the same underlying mechanism. Whittle introduced a model that was able to capture the pattern of discrimination thresholds and, in turn, the brightness data; however, there were a number of features in the data set that the model couldn't capture. In this paper, we demonstrate that the models of Kane and Bertalmío (2017) and Kingdom and Moulden (1991) may be adapted to predict all the data but only by incorporating an accurate model of detection thresholds. Additionally, we show that a divisive gain model may also capture the data but only by considering polarity-dependent, nonlinear inputs following the underlying pattern of detection thresholds. In summary, we conclude that these models provide a simple link between detection thresholds, discrimination thresholds, and brightness perception., This work has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant reference TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Visual information flow in Wilson–Cowan networks

  • Gómez Villa, Alexander
  • Bertalmío, Marcelo
  • Malo, Jesús
In this paper, we study the communication efficiency of a psychophysically tuned cascade of Wilson–Cowan and divisive normalization layers that simulate the retina-V1 pathway. This is the first analysis of Wilson–Cowan networks in terms of multivariate total correlation. The parameters of the cortical model have been derived through the relation between the steady state of the Wilson–Cowan model and the divisive normalization model. The communication efficiency has been analyzed in two ways: First, we provide an analytical expression for the reduction of the total correlation among the responses of a V1-like population after the application of the Wilson–Cowan interaction. Second, we empirically study the efficiency with visual stimuli and statistical tools that were not available before 1) we use a recent, radiometrically calibrated, set of natural scenes, and 2) we use a recent technique to estimate the multivariate total correlation in bits from sets of visual responses, which only involves univariate operations, thus giving better estimates of the redundancy. The theoretical and the empirical results show that, although this cascade of layers was not optimized for statistical independence in any way, the redundancy between the responses gets substantially reduced along the neural pathway. Specifically, we show that 1) the efficiency of a Wilson–Cowan network is similar to its equivalent divisive normalization model; 2) while initial layers (Von Kries adaptation and Weber-like brightness) contribute to univariate equalization, and the bigger contributions to the reduction in total correlation come from the computation of nonlinear local contrast and the application of local oriented filters; and 3) psychophysically tuned models are more efficient (reduce more total correlation) in the more populated regions of the luminance-contrast plane. These results are an alternative confirmation of the efficient coding hypothesis for the Wilson–Cowan systems, and, from an applied perspective, they suggest that neural field models could be an option in image coding to perform image compression., This work was partially funded by the Spanish Government and EU FEDER fund through the MINECO grants TIN2015-71537-P and DPI2017-89867-C2-2-R; and by the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Photorealistic style transfer for cinema shoots

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 Colour and Visual Computing Symposium (CVCS), celebrat del 19 a 20 de setembre de 2018 a Gjøvik, Noruega, Color grading is the process of subtly mixing andadjusting the color and tonal balance of a movie to achievea specific visuallook. This manual editing task may requirea significant amount of work from very skilled artists andtechnicians. In many cases the director wants to emulate thestyle and look present in a reference image, e.g. a still froman existing movie, a photograph, or even a previously shotsequence in the current movie. In this paper we propose amethod that automatically transfers the style, in terms of tone,color palette and contrast, from a reference image to the sourceRAW image. It consists of three separable operations: globalluminance matching, global color transfer and local contrastmatching. The computational complexity of the procedureis extremely low and allows for real-time implementationin-camera. As it just takes into account the statistics of sourceand reference images, no training is required. The results arefree from artifacts and provide an excellent approximation tothe intended look, bringing savings in pre-production, shootingand post-production time., This work has received funding from the European Union’s Horizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Statistics of natural images as a function of dynamic range

  • Grimaldi, Antoine Vincent
  • Kane, David
  • Bertalmío, Marcelo
The statistics of real world images have been extensively investigated, but in virtually all cases using only low dynamic range image databases. The few studies that have considered high dynamic range (HDR) images have performed statistical analyses categorizing images as HDR according to their creation technique, and not to the actual dynamic range of the underlying scene. In this study we demonstrate, using a recent HDR dataset of natural images, that the statistics of the image as received at the camera sensor change dramatically with dynamic range, with particularly strong correlations with dynamic range being observed for the median, standard deviation, skewness, and kurtosis, while the one over frequency relationship for the power spectrum breaks down for images with a very high dynamic range, in practice making HDR images not scale invariant. Effects are also noted in the derivative statistics, the single pixel histograms, and the Haar wavelet analysis. However, we also show that after some basic early transforms occurring within the eye (light scatter, nonlinear photoreceptor response, center-surround modulation) the statistics of the resulting images become virtually independent from the dynamic range, which would allow them to be processed more efficiently by the human visual system., This work has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement Number 761544 (project HDR4EU) and under Grant Agreement Number 780470 (project SAUCE); and by the Spanish government and FEDER Fund, Grantref. TIN2015- 71537-P(MINECO/ FEDER, UE).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette ofthe display device where the content will be shown, while preserving the artistic intent of the original content’s creator. In particular in thecinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamutmapping algorithms. In this paper we propose a novel framework that is based on vision science models, performs both gamut reductionand gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-artmethods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamutmapping problem., Thiswork has received funding from the European Union’sHorizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) andunder grant agreement number 780470 (project SAUCE),and by the Spanish government and FEDER Fund, grantref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J.Vazquez-Corral was supported by the Spanish governmentunder Grant IJCI-2014-19516
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Color-matching shots from different cameras having unknown gamma or logarithmic encoding curves

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: SMPTE 2017 Annual Technical Conference and Exhibition, celebrat del 23 al 26 d'octubre de 2017 a Los Angeles, Estats Units d'Amèrica., In cinema and TV it is quite usual to have to work with footage coming from several cameras, which show noticeable color differences among them even if they are all the same model. In TV broadcasts, technicians work in camera control units so as to ensure color consistency when cutting from one camera to another. In cinema post-production, colorists need to manually colormatch images coming from different sources. Aiming to help perform this task automatically, the Academy Color Encoding System (ACES) introduced a color management framework to work within the same color space and be able to use different cameras and displays; however, the ACES pipeline requires to have the cameras characterized previously, and therefore does not allow to work ‘in the wild’, a situation which is very common. We present a color stabilization method that, given two images of the same scene taken by two cameras with unknown settings and unknown internal parameter values, and encoded with unknown non-linear curves (logarithmic or gamma), is able to correct the colors of one of the images making it look as if it was captured with the other camera. Our method is based on treating the in-camera color processing pipeline as a combination of a 3x3 matrix followed by a non-linearity, which allows us to model a color stabilization transformation among two shots as a linear-nonlinear function with several parameters. We find corresponding points between the two images, compute the error (color difference) over them, and determine the transformation parameters that minimize this error, all automatically without any user input. The method is fast and the results have no spurious colors or spatio-temporal artifacts of any kind. It outperforms the state of the art both visually and according to several metrics, and can handle very challenging real-life examples., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Fusion-based variational image dehazing

  • Galdran, Adrian
  • Vazquez-Corral, Javier
  • Pardo, David
  • Bertalmío, Marcelo
We propose a novel image-dehazing technique based on the minimization of two energy functionals and a fusion scheme to combine the output of both optimizations. The proposed fusion-based variational image-dehazing (FVID) method is a spatially varying image enhancement process that first minimizes a previously proposed variational formulation that maximizes contrast and saturation on the hazy input. The iterates produced by this minimization are kept, and a second energy that shrinks faster intensity values of well-contrasted regions is minimized, allowing to generate a set of difference-of-saturation (DiffSat) maps by observing the shrinking rate. The iterates produced in the first minimization are then fused with these DiffSat maps to produce a haze-free version of the degraded input. The FVID method does not rely on a physical model from which to estimate a depth map, nor it needs a training stage on a database of human-labeled examples. Experimental results on a wide set of hazy images demonstrate that FVID better preserves the image structure on nearby regions that are less affected by fog, and it is successfully compared with other current methods in the task of removing haze degradation from faraway regions., The work of J. Vazquez-Corral and M. Bertalmío was supported by the ERC Starting Grant 306337, by the FEDER Fund under Grant TIN2015-71537-P(MINECO/FEDER,UE), by the ICREA Academia Award, and by the Spanish government under Grant IJCI-2014-19516. The work of D. Pardo was supported by the Spanish government under Grant MTM2013-40824-P, by the BCAM Severo Ochoa accreditation of excellence SEV-2013-0323, and the Basque Government CRG Grant IT649-13.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Color matching images with unknown non-linear encodings

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
We present a color matching method that, given two different views of the same scene taken by two cameras with unknown settings and unknown internal parameter values, and encoded with unknown non-linear curves, is able to correct the colors of one of the images making it look as if it was captured under the other camera’s settings. Our method is based on treating the in-camera color processing pipeline as a matrix multiplication followed by a non-linearity. This allows us to model a color stabilization transformation among the two shots by estimating several parameters. The method is fast and the results have no spurious colors. It outperforms the state-of-the-art both visually and according to several metrics, and can handle very challenging real-life examples., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Automatic, fast and perceptually accurate gamut mapping based on vision science models

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: SMPTE 2017 Annual Technical Conference and Exhibition, celebrat del 23 al 26 d'octubre de 2017 a Los Angeles, Estats Units d'Amèrica., Gamut mapping transforms colors of the original (image or video) content to the color palette of the display device with the simultaneous goals of (a) reproducing content accurately while preserving the artistic intent of the original content’s creator and (b) exploiting the full color rendering potential of the target display device. The rapid advancement in display technologies has created a pressing need to develop automatic and fast gamut mapping algorithms that can deal with imagery intended for both conventional and emerging displays. In this paper, we propose a novel framework based on retinal and color perception models from vision science that offers a functionality to perform both gamut reduction and gamut extension, while preserving hue and taking into account the analysis of the colors of the input image. We evaluate the performance of the proposed framework visually and by using a perceptually-based error metric, according to which the gamut-mapped results of our framework outperform those of the state-of-the-art methods., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award. The work of Javier Vazquez-Corral was supported by the Spanish government grant IJCI-2014-19516.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Spatial gamut mapping among non-inclusive gamuts

  • Bertalmío, Marcelo
  • Vazquez-Corral, Javier
Gamut mapping transforms the color gamut of an image to that of a target device. Two cases are usually considered: gamut reduction (target gamut smaller than source gamut), and gamut extension (target gamut larger than the source gamut). Less attention is devoted to the more general case, when neither gamut is fully included in the other. In this work we unify and expand two recent methods for gamut extension and reduction, so as to simultaneously perform both forms of gamut mapping in di erent regions of the same image without introducing color artifacts or halos. We demonstrate the usefulness of this approach for the traditional gamut mapping problem, and also how the proposed method can be used to adapt the color palette of an image so that it is closer to that of a given reference image. Results are compared with the state-of-the-art and validated through user tests and objective metrics., This work is supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Practical use suggests a re-evaluation of HDR objective quality metrics

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: Eleventh International Conference on Quality of Multimedia Experience (QoMEX, 2019), celebrada a Berlín (Alemanya) del 5 al 7 de juny de 2019., Full-reference objective quality metrics for high dynamic range (HDR) images must be validated in terms of their consistency with subjective evaluation results, and previous studies have found a number of metrics that appear to correlate well with the preference of observers. However, those conclusions were based on experiments that do not correspond to the practical use of a current day HDR professional production scenario. In this work, we carry out subjective evaluation experiments for the popular HDR standards of perceptual quantization (PQ) and Hybrid Log-Gamma (HLG), with a state-of-the-art HDR reference monitor used in broadcasting and post-production, and where all the observers participating in the tests are video experts. We find that the ranking of HDR metrics is now substantially different from what was reported earlier, and also that a simple standard dynamic range (SDR) metric can be applied directly to PQ or HLG encoded signals providing excellent results that surpass those of HDR metrics., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER, UE)
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

A geometric model of brightness perception and its application to color images correction

  • Batard, Thomas
  • Bertalmío, Marcelo
Human perception involves many features like contours, shapes, textures, and colors to name a few. Whereas several geometric models for contours, shapes and textures perception have been proposed, the geometry of color perception has received very little attention, possibly due to the fact that our perception of colors is still not fully understood. Nonetheless, there exists a class of mathematical models, gathered under the name Retinex, that aim at modeling the color perception of an image, that are inspired by psychophysical/physiological knowledge about color perception, and that can geometrically be viewed as the averaging of perceptual distances between image pixels. Some of the Retinex models turn out to be associated to an efficient image processing technique for the correction of camera output images. The aim of this paper is to show that this image processing technique can be improved by including more properties of the human visual system. To that purpose, we first present a generalization of the perceptual distance between image pixels by considering the parallel transport map associated to a covariant derivative on a vector bundle, and from which can be derived a new image processing model for color images correction. Then, we show that the family of covariant derivatives constructed in [T. Batard and N. Sochen, J. Math. Imaging Vision, 48(3) (2014), pp. 517-543] can model some color appearance phenomena related to brightness perception. Finally, we conduct experiments in which we show that the image processing techniques induced by these covariant derivatives outperform the original approach., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Derivatives and inverse of cascaded linear + nonlinear neural models

  • Martínez García, Marina
  • Cyriac, Praveen
  • Batard, Thomas
  • Bertalmío, Marcelo
  • Malo, Jesús
In vision science, cascades of Linear+Nonlinear transforms are very successful in modeling a number of perceptual experiences. However, the conventional literature is usually too focused on only describing the forward input-output transform. Instead, in this work we present the mathematics of such cascades beyond the forward transform, namely the Jacobian matrices and the inverse. The fundamental reason for this analytical treatment is that it offers useful analytical insight into the psychophysics, the physiology, and the function of the visual system. For instance, we show how the trends of the sensitivity (volume of the discrimination regions) and the adaptation of the receptive fields can be identified in the expression of the Jacobian w.r.t. the stimulus. This matrix also tells us which regions of the stimulus space are encoded more efficiently in multi-information terms. The Jacobian w.r.t. the parameters shows which aspects of the model have bigger impact in the response, and hence their relative relevance. The analytic inverse implies conditions for the response and model parameters to ensure appropriate decoding. From the experimental and applied perspective, (a) the Jacobian w.r.t. the stimulus is necessary in new experimental methods based on the synthesis of visual stimuli with interesting geometrical properties, (b) the Jacobian matrices w.r.t. the parameters are convenient to learn the model from classical experiments or alternative goal optimization, and (c) the inverse is a promising model-based alternative to blind machine-learning methods for neural decoding that do not include meaningful biological information. The theory is checked by building and testing a vision model that actually follows a modular Linear+Nonlinear program. Our illustrative derivable and invertible model consists of a cascade of modules that account for brightness, contrast, energy masking, and wavelet masking. To stress the generality of this modular setting we show examples where some of the canonical Divisive Normalization modules are substituted by equivalent modules such as the Wilson-Cowan interaction model (at the V1 cortex) or a tone-mapping model (at the retina)., This work was partially funded by the Spanish Ministerio de Economia y Competitividad projects CICYT TEC2013-50520-EXP and CICYT BFU2014-59776-R, by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015- 71537-P(MINECO/FEDER,UE), 1021, and by the ICREA Academia Award.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Local denoising based on curvature smoothing can visually outperform non-local methods on photographs with actual noise

  • Ghimpeteanu, Gabriela
  • Kane, David
  • Batard, Thomas
  • Levine, Stacey
  • Bertalmío, Marcelo
Comunicació presentada a: IEEE International Conference on Image Processing 2016, celebrada del 25 l 28 de setembre de 2016 a Phoenix, Estats Units d'Amèrica., We propose a fast, local denoising method where the Euclidean curvature of the noisy image is approximated in a regularizing manner and a clean image is reconstructed from this smoothed curvature. User preference tests show that when denoising real photographs with actual noise our method produces results with the same visual quality as the more sophisticated, nonlocal algorithms Non-local Means and BM3D, but at a fraction of their computational cost. These tests also highlight the limitations of objective image quality metrics like PSNR and SSIM, which correlate poorly with user preference., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government, grant ref. TIN2015-71537-P, and by the Icrea Academia Award.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline, and their impact in HDR imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tone-mapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., This work has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Convolutional neural networks deceived by visual illusions

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada al CVPR 2019: Conference on Computer Vision and Pattern Recognition, celebrat del 16 al 20 de juny de 2019 a California, Estats Units d'Amèrica., Visual illusions teach us that what we see is not always what is represented in the physical world. Their special nature make them a fascinating tool to test and validate any new vision model proposed. In general, current vision models are based on the concatenation of linear and non-linear operations. The similarity of this structure with the operations present in Convolutional Neural Networks (CNNs) has motivated us to study if CNNs trained for low-level visual tasks are deceived by visual illusions. In particular, we show that CNNs trained for image denoising, image deblurring, and computational color constancy are able to replicate the human response to visual illusions, and that the extent of this replication varies with respect to variation in architecture and spatial pattern size. These results suggest that in order to obtain CNNs that better replicate human behaviour, we may need to start aiming for them to better replicate visual illusions., This work has received funding from the EU Horizon 2020 programme under grant agreement 761544 (project HDR4EU) and under grant agreement 780470 (project SAUCE) and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Improved high dynamic range video coding with a nonlinearity based on natural image statistics

  • Sugito, Yasuko
  • Cyriac, Praveen
  • Kane, David
  • Bertalmío, Marcelo
High Dynamic Range (HDR) technologies support the capture and presentation of a wider range of luminance values than conventional systems. An important element of video processing is the transfer function which should emulate human perception and this needs to be revisited for HDR content and displays. In the paper, we adapt a nonlinearity designed for the tone-mapping problem to the problem of video coding. We test the nonlinearity using the Motion Picture Experts Group methodology and find it can outperform existing methods in terms of HDR video quality measure., This work was partially supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Log-encoding estimation for color stabilization of cinematic footage

  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: IEEE International Conference on Image Processing 2016, celebrada del 25 l 28 de setembre de 2016 a Phoenix, Estats Units d'Amèrica., We propose a method for the color stabilization of cinema shots coming from different cameras that use unknown logarithmic encoding curves. The log-encoding curves are approximated by a concatenation of gamma-curves, whose values are accurately computed using image matches. The color stabilization procedure, based on the generic color processing pipeline of a digital camera, can be performed after the estimation of the encoding curves, and it also requires the existence of image matches. Our work can be applied in different scenarios such as multi-camera shoots, native-3D cinema, or color grading in post-production., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government, grant ref. TIN2015-71537-P, and by the Icrea Academia Award.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

In praise of artifice reloaded: Caution with natural image databases in modeling vision

  • Martinez-Garcia, Marina
  • Bertalmío, Marcelo
  • Malo, Jesús
Subjective image quality databases are a major source of raw data on how the visual system works in naturalistic environments. These databases describe the sensitivity of many observers to a wide range of distortions of different nature and intensity seen on top of a variety of natural images. Data of this kind seems to open a number of possibilities for the vision scientist to check the models in realistic scenarios. However, while these natural databases are great benchmarks for models developed in some other way (e.g., by using the well-controlled artificial stimuli of traditional psychophysics), they should be carefully used when trying to fit vision models. Given the high dimensionality of the image space, it is very likely that some basic phenomena are under-represented in the database. Therefore, a model fitted on these large-scale natural databases will not reproduce these under-represented basic phenomena that could otherwise be easily illustrated with well selected artificial stimuli. In this work we study a specific example of the above statement. A standard cortical model using wavelets and divisive normalization tuned to reproduce subjective opinion on a large image quality dataset fails to reproduce basic cross-masking. Here we outline a solution for this problem by using artificial stimuli and by proposing a modification that makes the model easier to tune. Then, we show that the modified model is still competitive in the large-scale database. Our simulations with these artificial stimuli show that when using steerable wavelets, the conventional unit norm Gaussian kernels in divisive normalization should be multiplied by high-pass filters to reproduce basic trends in masking. Basic visual phenomena may be misrepresented in large natural image datasets but this can be solved with model-interpretable stimuli. This is an additional argument in praise of artifice in line with Rust and Movshon (2005)., This work was partially funded by the Spanish and EU FEDER fund through the MINECO/FEDER/EU grants TIN2015-71537-P and DPI2017-89867-C2-2-R; and by the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Derivatives and inverse of cascaded linear +nonlinear neural models

  • Martinez-Garcia, Marina
  • Cyriac, P.
  • Batard, T.
  • Bertalmío, Marcelo
  • Malo, Jesús
In vision science, cascades of Linear+Nonlinear transforms are very successful in modeling a number of perceptual experiences. However, the conventional literature is usually too focused on only describing the forward input-output transform. Instead, in this work we present the mathematics of such cascades beyond the forward transform, namely the Jacobian matrices and the inverse. The fundamental reason for this analytical treatment is that it offers useful analytical insight into the psychophysics, the physiology, and the function of the visual system. For instance, we show how the trends of the sensitivity (volume of the discrimination regions) and the adaptation of the receptive fields can be identified in the expression of the Jacobian w.r.t. the stimulus. This matrix also tells us which regions of the stimulus space are encoded more efficiently in multi-information terms. The Jacobian w.r.t. the parameters shows which aspects of the model have bigger impact in the response, and hence their relative relevance. The analytic inverse implies conditions for the response and model parameters to ensure appropriate decoding. From the experimental and applied perspective, (a) the Jacobian w.r.t. the stimulus is necessary in new experimental methods based on the synthesis of visual stimuli with interesting geometrical properties, (b) the Jacobian matrices w.r.t. the parameters are convenient to learn the model from classical experiments or alternative goal optimization, and (c) the inverse is a promising model-based alternative to blind machine-learning methods for neural decoding that do not include meaningful biological information. The theory is checked by building and testing a vision model that actually follows a modular Linear+Nonlinear program. Our illustrative derivable and invertible model consists of a cascade of modules that account for brightness, contrast, energy masking, and wavelet masking. To stress the generality of this modular setting we show examples where some of the canonical Divisive Normalization modules are substituted by equivalent modules such as the Wilson-Cowan interaction model (at the V1 cortex) or a tone-mapping model (at the retina)., This work was partially funded by the Spanish Ministerio de Economia y Competitividad projects CICYT TEC2013-50520-EXP and CICYT BFU2014-59776-R, by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), 1021, and by the ICREA Academia Award., Peer reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Performance evaluation of objective quality metrics on HLG-based HDR image coding

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), celebrada a Anaheim, Estats Units d'Amèrica, del 26 al 29 de novembre de 2018., We evaluate the performance of objective quality metrics for high dynamic range (HDR) image coding that uses the transfer function (TF) of the Hybrid Log-Gamma (HLG) method. Previous evaluations of objective metrics for HDR image coding have studied which of them are reliable predictors of perceived quality; however, in those tests, all the non-linear transforms used both for encoding and by the best-performing metrics are essentially very similar and based on visual perception data of detection thresholds for lightness variations. The HLG non-linearity on the other hand is very different, as it is designed for backward compatibility with standard dynamic range (SDR) displays. We test a variety of options for objective metrics, including HLG-based. Our results show that the ranking of metrics for HDR coding changes drastically depending on the TF used for compression., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER, UE)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Spatial gamut mapping among non-inclusive gamuts

  • Bertalmío, Marcelo
  • Vazquez-Corral, Javier
Gamut mapping transforms the color gamut of an image to that of a target device. Two cases are usually considered: gamut reduction (target gamut smaller than source gamut), and gamut extension (target gamut larger than the source gamut). Less attention is devoted to the more general case, when neither gamut is fully included in the other. In this work we unify and expand two recent methods for gamut extension and reduction, so as to simultaneously perform both forms of gamut mapping in di erent regions of the same image without introducing color artifacts or halos. We demonstrate the usefulness of this approach for the traditional gamut mapping problem, and also how the proposed method can be used to adapt the color palette of an image so that it is closer to that of a given reference image. Results are compared with the state-of-the-art and validated through user tests and objective metrics., This work is supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A reevaluation of Whittle (1986, 1992) reveals the link between detection thresholds, discrimination thresholds, and brightness perception

  • Kane, David
  • Bertalmío, Marcelo
In 1986, Paul Whittle investigated the ability to discriminate between the luminance of two small patches viewed upon a uniform background. In 1992, Paul Whittle asked subjects to manipulate the luminance of a number of patches on a uniform background until their brightness appeared to vary from black to white with even steps. The data from the discrimination experiment almost perfectly predicted the gradient of the function obtained in the brightness experiment, indicating that the two experimental methodologies were probing the same underlying mechanism. Whittle introduced a model that was able to capture the pattern of discrimination thresholds and, in turn, the brightness data; however, there were a number of features in the data set that the model couldn't capture. In this paper, we demonstrate that the models of Kane and Bertalmío (2017) and Kingdom and Moulden (1991) may be adapted to predict all the data but only by incorporating an accurate model of detection thresholds. Additionally, we show that a divisive gain model may also capture the data but only by considering polarity-dependent, nonlinear inputs following the underlying pattern of detection thresholds. In summary, we conclude that these models provide a simple link between detection thresholds, discrimination thresholds, and brightness perception., This work has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant reference TIN2015-71537-P (MINECO/FEDER,UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Photorealistic style transfer for cinema shoots

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a: 2018 Colour and Visual Computing Symposium (CVCS), celebrat del 19 a 20 de setembre de 2018 a Gjøvik, Noruega, Color grading is the process of subtly mixing andadjusting the color and tonal balance of a movie to achievea specific visuallook. This manual editing task may requirea significant amount of work from very skilled artists andtechnicians. In many cases the director wants to emulate thestyle and look present in a reference image, e.g. a still froman existing movie, a photograph, or even a previously shotsequence in the current movie. In this paper we propose amethod that automatically transfers the style, in terms of tone,color palette and contrast, from a reference image to the sourceRAW image. It consists of three separable operations: globalluminance matching, global color transfer and local contrastmatching. The computational complexity of the procedureis extremely low and allows for real-time implementationin-camera. As it just takes into account the statistics of sourceand reference images, no training is required. The results arefree from artifacts and provide an excellent approximation tothe intended look, bringing savings in pre-production, shootingand post-production time., This work has received funding from the European Union’s Horizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline and their impact in hdr imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tonemapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., The work of the authors was supported by the European Unions Horizon 2020 research and innovation programme grant 761544 (project HDR4EU) and grant 780470 (project SAUCE) and by the Spanish government and FEDER fund grant TIN2015-71537-P (MINECO/FEDER,UE). The work of the second author was supported by the Spanish government under grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Color-matching shots from different cameras having unknown gamma or logarithmic encoding curves

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: SMPTE 2017 Annual Technical Conference and Exhibition, celebrat del 23 al 26 d'octubre de 2017 a Los Angeles, Estats Units d'Amèrica., In cinema and TV it is quite usual to have to work with footage coming from several cameras, which show noticeable color differences among them even if they are all the same model. In TV broadcasts, technicians work in camera control units so as to ensure color consistency when cutting from one camera to another. In cinema post-production, colorists need to manually colormatch images coming from different sources. Aiming to help perform this task automatically, the Academy Color Encoding System (ACES) introduced a color management framework to work within the same color space and be able to use different cameras and displays; however, the ACES pipeline requires to have the cameras characterized previously, and therefore does not allow to work ‘in the wild’, a situation which is very common. We present a color stabilization method that, given two images of the same scene taken by two cameras with unknown settings and unknown internal parameter values, and encoded with unknown non-linear curves (logarithmic or gamma), is able to correct the colors of one of the images making it look as if it was captured with the other camera. Our method is based on treating the in-camera color processing pipeline as a combination of a 3x3 matrix followed by a non-linearity, which allows us to model a color stabilization transformation among two shots as a linear-nonlinear function with several parameters. We find corresponding points between the two images, compute the error (color difference) over them, and determine the transformation parameters that minimize this error, all automatically without any user input. The method is fast and the results have no spurious colors or spatio-temporal artifacts of any kind. It outperforms the state of the art both visually and according to several metrics, and can handle very challenging real-life examples., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Practical use suggests a re-evaluation of HDR objective quality metrics

  • Sugito, Yasuko
  • Bertalmío, Marcelo
Comunicació presentada a: Eleventh International Conference on Quality of Multimedia Experience (QoMEX, 2019), celebrada a Berlín (Alemanya) del 5 al 7 de juny de 2019., Full-reference objective quality metrics for high dynamic range (HDR) images must be validated in terms of their consistency with subjective evaluation results, and previous studies have found a number of metrics that appear to correlate well with the preference of observers. However, those conclusions were based on experiments that do not correspond to the practical use of a current day HDR professional production scenario. In this work, we carry out subjective evaluation experiments for the popular HDR standards of perceptual quantization (PQ) and Hybrid Log-Gamma (HLG), with a state-of-the-art HDR reference monitor used in broadcasting and post-production, and where all the observers participating in the tests are video experts. We find that the ranking of HDR metrics is now substantially different from what was reported earlier, and also that a simple standard dynamic range (SDR) metric can be applied directly to PQ or HLG encoded signals providing excellent results that surpass those of HDR metrics., This work has partially received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER, UE)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Automatic, fast and perceptually accurate gamut mapping based on vision science models

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: SMPTE 2017 Annual Technical Conference and Exhibition, celebrat del 23 al 26 d'octubre de 2017 a Los Angeles, Estats Units d'Amèrica., Gamut mapping transforms colors of the original (image or video) content to the color palette of the display device with the simultaneous goals of (a) reproducing content accurately while preserving the artistic intent of the original content’s creator and (b) exploiting the full color rendering potential of the target display device. The rapid advancement in display technologies has created a pressing need to develop automatic and fast gamut mapping algorithms that can deal with imagery intended for both conventional and emerging displays. In this paper, we propose a novel framework based on retinal and color perception models from vision science that offers a functionality to perform both gamut reduction and gamut extension, while preserving hue and taking into account the analysis of the colors of the input image. We evaluate the performance of the proposed framework visually and by using a perceptually-based error metric, according to which the gamut-mapped results of our framework outperform those of the state-of-the-art methods., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award. The work of Javier Vazquez-Corral was supported by the Spanish government grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Issues with common assumptions about the camera pipeline, and their impact in HDR imaging from multiple exposures

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tone-mapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics., This work has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Fusion-based variational image dehazing

  • Galdran, Adrian
  • Vazquez-Corral, Javier
  • Pardo, David
  • Bertalmío, Marcelo
We propose a novel image-dehazing technique based on the minimization of two energy functionals and a fusion scheme to combine the output of both optimizations. The proposed fusion-based variational image-dehazing (FVID) method is a spatially varying image enhancement process that first minimizes a previously proposed variational formulation that maximizes contrast and saturation on the hazy input. The iterates produced by this minimization are kept, and a second energy that shrinks faster intensity values of well-contrasted regions is minimized, allowing to generate a set of difference-of-saturation (DiffSat) maps by observing the shrinking rate. The iterates produced in the first minimization are then fused with these DiffSat maps to produce a haze-free version of the degraded input. The FVID method does not rely on a physical model from which to estimate a depth map, nor it needs a training stage on a database of human-labeled examples. Experimental results on a wide set of hazy images demonstrate that FVID better preserves the image structure on nearby regions that are less affected by fog, and it is successfully compared with other current methods in the task of removing haze degradation from faraway regions., The work of J. Vazquez-Corral and M. Bertalmío was supported by the ERC Starting Grant 306337, by the FEDER Fund under Grant TIN2015-71537-P(MINECO/FEDER,UE), by the ICREA Academia Award, and by the Spanish government under Grant IJCI-2014-19516. The work of D. Pardo was supported by the Spanish government under Grant MTM2013-40824-P, by the BCAM Severo Ochoa accreditation of excellence SEV-2013-0323, and the Basque Government CRG Grant IT649-13.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Retinal lateral inhibition provides the biological basis of long-range spatial induction

  • Yeonan-Kim, Jihyun
  • Bertalmío, Marcelo
Retinal lateral inhibition is one of the conventional efficient coding mechanisms in the visual system that is produced by interneurons that pool signals over a neighborhood of presynaptic feedforward cells and send inhibitory signals back to them. Thus, the receptive-field (RF) of a retinal ganglion cell has a center-surround receptive-field (RF) profile that is classically represented as a difference-of-Gaussian (DOG) adequate for efficient spatial contrast coding. The DOG RF profile has been attributed to produce the psychophysical phenomena of brightness induction, in which the perceived brightness of an object is affected by that of its vicinity, either shifting away from it (brightness contrast) or becoming more similar to it (brightness assimilation) depending on the size of the surfaces surrounding the object. While brightness contrast can be modeled using a DOG with a narrow surround, brightness assimilation requires a wide suppressive surround. Early retinal studies determined that the suppressive surround of a retinal ganglion cell is narrow (< 100–300 μm; ‘classic RF’), which led researchers to postulate that brightness assimilation must originate at some post-retinal, possibly cortical, stage where long-range interactions are feasible. However, more recent studies have reported that the retinal interneurons also exhibit a spatially wide component (> 500–1000 μm). In the current study, we examine the effect of this wide interneuron RF component in two biophysical retinal models and show that for both of the retinal models it explains the long-range effect evidenced in simultaneous brightness induction phenomena and that the spatial extent of this long-range effect of the retinal model responses matches that of perceptual data. These results suggest that the retinal lateral inhibition mechanism alone can regulate local as well as long-range spatial induction through the narrow and wide RF components of retinal interneurons, arguing against the existing view that spatial induction is operated by two separate local vs. long-range mechanisms., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Perceptually-based gamut extension algorithm for emerging wide color gamut display and projection technologies

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: SMPTE 2016 Annual Technical Conference and Exhibition, celebrada a Los Angeles, Estats Units d'Amèrica, del 25 al 27 d'octubre de 2016., While wide color gamut (WCG) capabilities are a key element of emerging display and projection technologies, at present most image content is recorded using standards such as DCI-P3 for cinema or BT.709 for TV that have a reduced color gamut. Therefore, there is a need for gamut extension methods that process regular content and allow to appreciate the full color potential of new displays, improving user experience. We present a gamut extension algorithm that is based on visual perception models and explicitly takes into account low chromatic colors such as skin tones. It produces results that look natural, are free of artifacts of any kind, and outperform the state of the art. The method is fast, allowing for operation interaction if needed., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Improved high dynamic range video coding with a nonlinearity based on natural image statistics

  • Sugito, Yasuko
  • Cyriac, Praveen
  • Kane, David
  • Bertalmío, Marcelo
High Dynamic Range (HDR) technologies support the capture and presentation of a wider range of luminance values than conventional systems. An important element of video processing is the transfer function which should emulate human perception and this needs to be revisited for HDR content and displays. In the paper, we adapt a nonlinearity designed for the tone-mapping problem to the problem of video coding. We test the nonlinearity using the Motion Picture Experts Group methodology and find it can outperform existing methods in terms of HDR video quality measure., This work was partially supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Local denoising based on curvature smoothing can visually outperform non-local methods on photographs with actual noise

  • Ghimpeteanu, Gabriela
  • Kane, David
  • Batard, Thomas
  • Levine, Stacey
  • Bertalmío, Marcelo
Comunicació presentada a: IEEE International Conference on Image Processing 2016, celebrada del 25 l 28 de setembre de 2016 a Phoenix, Estats Units d'Amèrica., We propose a fast, local denoising method where the Euclidean curvature of the noisy image is approximated in a regularizing manner and a clean image is reconstructed from this smoothed curvature. User preference tests show that when denoising real photographs with actual noise our method produces results with the same visual quality as the more sophisticated, nonlocal algorithms Non-local Means and BM3D, but at a fraction of their computational cost. These tests also highlight the limitations of objective image quality metrics like PSNR and SSIM, which correlate poorly with user preference., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government, grant ref. TIN2015-71537-P, and by the Icrea Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A geometric model of brightness perception and its application to color images correction

  • Batard, Thomas
  • Bertalmío, Marcelo
Human perception involves many features like contours, shapes, textures, and colors to name a few. Whereas several geometric models for contours, shapes and textures perception have been proposed, the geometry of color perception has received very little attention, possibly due to the fact that our perception of colors is still not fully understood. Nonetheless, there exists a class of mathematical models, gathered under the name Retinex, that aim at modeling the color perception of an image, that are inspired by psychophysical/physiological knowledge about color perception, and that can geometrically be viewed as the averaging of perceptual distances between image pixels. Some of the Retinex models turn out to be associated to an efficient image processing technique for the correction of camera output images. The aim of this paper is to show that this image processing technique can be improved by including more properties of the human visual system. To that purpose, we first present a generalization of the perceptual distance between image pixels by considering the parallel transport map associated to a covariant derivative on a vector bundle, and from which can be derived a new image processing model for color images correction. Then, we show that the family of covariant derivatives constructed in [T. Batard and N. Sochen, J. Math. Imaging Vision, 48(3) (2014), pp. 517-543] can model some color appearance phenomena related to brightness perception. Finally, we conduct experiments in which we show that the image processing techniques induced by these covariant derivatives outperform the original approach., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Color matching images with unknown non-linear encodings

  • Gil Rodríguez, Raquel
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
We present a color matching method that, given two different views of the same scene taken by two cameras with unknown settings and unknown internal parameter values, and encoded with unknown non-linear curves, is able to correct the colors of one of the images making it look as if it was captured under the other camera’s settings. Our method is based on treating the in-camera color processing pipeline as a matrix multiplication followed by a non-linearity. This allows us to model a color stabilization transformation among the two shots by estimating several parameters. The method is fast and the results have no spurious colors. It outperforms the state-of-the-art both visually and according to several metrics, and can handle very challenging real-life examples., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Vision models for wide color gamut imaging in cinema

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette ofthe display device where the content will be shown, while preserving the artistic intent of the original content’s creator. In particular in thecinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamutmapping algorithms. In this paper we propose a novel framework that is based on vision science models, performs both gamut reductionand gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-artmethods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamutmapping problem., Thiswork has received funding from the European Union’sHorizon 2020 research and innovation programme undergrant agreement number 761544 (project HDR4EU) andunder grant agreement number 780470 (project SAUCE),and by the Spanish government and FEDER Fund, grantref. TIN2015-71537-P (MINECO/FEDER,UE). The work of J.Vazquez-Corral was supported by the Spanish governmentunder Grant IJCI-2014-19516
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Log-encoding estimation for color stabilization of cinematic footage

  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada a: IEEE International Conference on Image Processing 2016, celebrada del 25 l 28 de setembre de 2016 a Phoenix, Estats Units d'Amèrica., We propose a method for the color stabilization of cinema shots coming from different cameras that use unknown logarithmic encoding curves. The log-encoding curves are approximated by a concatenation of gamma-curves, whose values are accurately computed using image matches. The color stabilization procedure, based on the generic color processing pipeline of a digital camera, can be performed after the estimation of the encoding curves, and it also requires the existence of image matches. Our work can be applied in different scenarios such as multi-camera shoots, native-3D cinema, or color grading in post-production., This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government, grant ref. TIN2015-71537-P, and by the Icrea Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

System gamma as a function of image- and monitor-dynamic range

  • Kane, David
  • Bertalmío, Marcelo
System gamma is the end-to-end exponent that describes the relationship between the relative luminance values at capture and the reproduced image. The system gamma preferred by subjects is known to vary with the background luminance condition and the image in question. We confirm the previous two findings using an image database with both high and low dynamic range images (from 102 to 107), but also find that the preferred system gamma varies with the dynamic range of the monitor (CRT, LCD, or OLED). We find that the preferred system gamma can be predicted in all conditions and for all images by a simple model that searches for the value that best flattens the lightness distribution, where lightness is modeled as a power law of onscreen luminance. To account for the data, the exponent must vary with the viewing conditions. The method presented allows the inference of lightness perception in natural scenes without direct measurement and makes testable predictions for how lightness perception varies with the viewing condition and the distribution of luminance values in a scene. The data from this paper has been made available online., This work was supported by the European Research Council, Starting (Grant ref. 306337), the Spanish government (grant ref. TIN2015-71537-P), and by the Icrea Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Gamut mapping for visual attention retargeting

  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Visual attention retargeting attempts to modify an image such that the viewer’s attention is directed to specific regions. Goals include highlighting a particular object or hiding possible problems in the image. In this work, we show that we can pose the visual retargeting problem in terms of gamut mapping. In short, visual attention retargeting can be achieved by performing gamut extension in those regions that we want to highlight and gamut reduction in the other regions., Authors are supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government FEDER Fund, grant ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The work of J. Vazquez-Corral was supported by the Spanish government under Grant IJCI-2014- 19516.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Visual information flow in Wilson–Cowan networks

  • Gómez Villa, Alexander
  • Bertalmío, Marcelo
  • Malo, Jesús
In this paper, we study the communication efficiency of a psychophysically tuned cascade of Wilson–Cowan and divisive normalization layers that simulate the retina-V1 pathway. This is the first analysis of Wilson–Cowan networks in terms of multivariate total correlation. The parameters of the cortical model have been derived through the relation between the steady state of the Wilson–Cowan model and the divisive normalization model. The communication efficiency has been analyzed in two ways: First, we provide an analytical expression for the reduction of the total correlation among the responses of a V1-like population after the application of the Wilson–Cowan interaction. Second, we empirically study the efficiency with visual stimuli and statistical tools that were not available before 1) we use a recent, radiometrically calibrated, set of natural scenes, and 2) we use a recent technique to estimate the multivariate total correlation in bits from sets of visual responses, which only involves univariate operations, thus giving better estimates of the redundancy. The theoretical and the empirical results show that, although this cascade of layers was not optimized for statistical independence in any way, the redundancy between the responses gets substantially reduced along the neural pathway. Specifically, we show that 1) the efficiency of a Wilson–Cowan network is similar to its equivalent divisive normalization model; 2) while initial layers (Von Kries adaptation and Weber-like brightness) contribute to univariate equalization, and the bigger contributions to the reduction in total correlation come from the computation of nonlinear local contrast and the application of local oriented filters; and 3) psychophysically tuned models are more efficient (reduce more total correlation) in the more populated regions of the luminance-contrast plane. These results are an alternative confirmation of the efficient coding hypothesis for the Wilson–Cowan systems, and, from an applied perspective, they suggest that neural field models could be an option in image coding to perform image compression., This work was partially funded by the Spanish Government and EU FEDER fund through the MINECO grants TIN2015-71537-P and DPI2017-89867-C2-2-R; and by the European Union’s Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Analysis of retinal and cortical components of Retinex algorithms

  • Yeonan-Kim, Jihyun
  • Bertalmío, Marcelo
Following Land and McCann’s first proposal of the Retinex theory, numerous Retinex algorithms that differ considerably both algorithmically and functionally have been developed. We clarify the relationships among various Retinex families by associating their spatial processing structures to the neural organizations in the retina and the primary visual cortex in the brain. Some of the Retinex algorithms have a retina-like processing structure (Land’s designator idea and NASA Retinex), and some show a close connection with the cortical structures in the primary visual area of the brain (two-dimensional L&M Retinex). A third group of Retinexes (the variational Retinex) manifests an explicit algorithmic relation to Wilson–Cowan’s physiological model. We intend to overview these three groups of Retinexes with the frame of reference in the biological visual mechanisms., This work was supported by the European Research Council, Starting Grant Ref. 306337, by the Spanish government and FEDER Fund, grant Ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The authors declare that no conflicting interests exist.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Derivatives and inverse of cascaded linear + nonlinear neural models

  • Martínez García, Marina
  • Cyriac, Praveen
  • Batard, Thomas
  • Bertalmío, Marcelo
  • Malo, Jesús
In vision science, cascades of Linear+Nonlinear transforms are very successful in modeling a number of perceptual experiences. However, the conventional literature is usually too focused on only describing the forward input-output transform. Instead, in this work we present the mathematics of such cascades beyond the forward transform, namely the Jacobian matrices and the inverse. The fundamental reason for this analytical treatment is that it offers useful analytical insight into the psychophysics, the physiology, and the function of the visual system. For instance, we show how the trends of the sensitivity (volume of the discrimination regions) and the adaptation of the receptive fields can be identified in the expression of the Jacobian w.r.t. the stimulus. This matrix also tells us which regions of the stimulus space are encoded more efficiently in multi-information terms. The Jacobian w.r.t. the parameters shows which aspects of the model have bigger impact in the response, and hence their relative relevance. The analytic inverse implies conditions for the response and model parameters to ensure appropriate decoding. From the experimental and applied perspective, (a) the Jacobian w.r.t. the stimulus is necessary in new experimental methods based on the synthesis of visual stimuli with interesting geometrical properties, (b) the Jacobian matrices w.r.t. the parameters are convenient to learn the model from classical experiments or alternative goal optimization, and (c) the inverse is a promising model-based alternative to blind machine-learning methods for neural decoding that do not include meaningful biological information. The theory is checked by building and testing a vision model that actually follows a modular Linear+Nonlinear program. Our illustrative derivable and invertible model consists of a cascade of modules that account for brightness, contrast, energy masking, and wavelet masking. To stress the generality of this modular setting we show examples where some of the canonical Divisive Normalization modules are substituted by equivalent modules such as the Wilson-Cowan interaction model (at the V1 cortex) or a tone-mapping model (at the retina)., This work was partially funded by the Spanish Ministerio de Economia y Competitividad projects CICYT TEC2013-50520-EXP and CICYT BFU2014-59776-R, by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015- 71537-P(MINECO/FEDER,UE), 1021, and by the ICREA Academia Award.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Statistics of natural images as a function of dynamic range

  • Grimaldi, Antoine Vincent
  • Kane, David
  • Bertalmío, Marcelo
The statistics of real world images have been extensively investigated, but in virtually all cases using only low dynamic range image databases. The few studies that have considered high dynamic range (HDR) images have performed statistical analyses categorizing images as HDR according to their creation technique, and not to the actual dynamic range of the underlying scene. In this study we demonstrate, using a recent HDR dataset of natural images, that the statistics of the image as received at the camera sensor change dramatically with dynamic range, with particularly strong correlations with dynamic range being observed for the median, standard deviation, skewness, and kurtosis, while the one over frequency relationship for the power spectrum breaks down for images with a very high dynamic range, in practice making HDR images not scale invariant. Effects are also noted in the derivative statistics, the single pixel histograms, and the Haar wavelet analysis. However, we also show that after some basic early transforms occurring within the eye (light scatter, nonlinear photoreceptor response, center-surround modulation) the statistics of the resulting images become virtually independent from the dynamic range, which would allow them to be processed more efficiently by the human visual system., This work has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement Number 761544 (project HDR4EU) and under Grant Agreement Number 780470 (project SAUCE); and by the Spanish government and FEDER Fund, Grantref. TIN2015- 71537-P(MINECO/ FEDER, UE).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Gamut reduction through local saturation reduction

  • Zamir, Syed Waqas
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
This work was supported by the European Research Council, Starting Grant ref. 306337, by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE), and by the Icrea Academia Award. The work of Javier Vazquez-Corral was supported by the Spanish government grant IJCI-2014-19516.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Convolutional neural networks deceived by visual illusions

  • Gómez Villa, Alexander
  • Martín, Adrian
  • Vazquez-Corral, Javier
  • Bertalmío, Marcelo
Comunicació presentada al CVPR 2019: Conference on Computer Vision and Pattern Recognition, celebrat del 16 al 20 de juny de 2019 a California, Estats Units d'Amèrica., Visual illusions teach us that what we see is not always what is represented in the physical world. Their special nature make them a fascinating tool to test and validate any new vision model proposed. In general, current vision models are based on the concatenation of linear and non-linear operations. The similarity of this structure with the operations present in Convolutional Neural Networks (CNNs) has motivated us to study if CNNs trained for low-level visual tasks are deceived by visual illusions. In particular, we show that CNNs trained for image denoising, image deblurring, and computational color constancy are able to replicate the human response to visual illusions, and that the extent of this replication varies with respect to variation in architecture and spatial pattern size. These results suggest that in order to obtain CNNs that better replicate human behaviour, we may need to start aiming for them to better replicate visual illusions., This work has received funding from the EU Horizon 2020 programme under grant agreement 761544 (project HDR4EU) and under grant agreement 780470 (project SAUCE) and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

In-camera, photorealistic style transfer for on-set automatic grading

  • Zabaleta, Itziar
  • Bertalmío, Marcelo
Comunicació presentada a la SMPTE 2018 Annual Technical Conference of the Society of Motion Picture and Television Engineers, celebrada a Los Angeles (Estats Units d'Amèrica) del 22 a 25 de octubre de 2018., In professional cinema, the intended artistic look of the movie informs the creation of a static 3D LUT that is applied on set, where further manual modifications to the image appearance are registered as 10-parameter transforms in a color decision list (CDL). The original RAW footage and its corresponding LUT and CDL are passedon to the post-production stage where the fine-tuning of the final look is performed during color grading. — In many cases, the director wants to emulate the style and look present in a reference image, e.g. a still from an existing movie, or a photograph, or a painting, or even a frame from a previously shot sequence in the current movie. The manual creation of a LUT and CDL for this purpose may require a significant amount of work from very skilled artists and technicians, while the state of the art in the academic literature offers promising but partial solutions to the photorealistic style transfer problem, with limitations regarding artifacts, speed and manual interaction. — In this paper, we propose a method that automatically transfers the style, in terms of luminance, color palette and contrast, from a reference image to the source raw footage. It consists of three separable operations: global luminance matching, global color transfer and local contrast matching. As it just takes into account the statistics of source and reference images, no training is required. The total transform is not static but adapts to the changes in the source footage. The computational complexity of the procedure is extremely low and allows for real-time implementation in-camera, for on-set monitoring. While the method is proposed as a substitute for the need to specify a LUT and a CDL, it's compatible with further refinements performed via LUTs, CDLs and grading, both on-set and in post-production. The results are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time., This work has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement number 761544 (project HDR4EU) and under grant agreement number 780470 (project SAUCE), and by the Spanish government and FEDER Fund, grant ref. TIN2015-71537-P (MINECO/FEDER,UE).
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

Enhancing spatio-chromatic representation with more-than-three color coding for image description

  • Rafegas Fonoll, Ivet
  • Vazquez-Corral, Javier|||0000-0003-0414-7096
  • Benavente i Vidal, Robert|||0000-0001-9819-4445
  • Vanrell i Martorell, Maria Isabel|||0000-0002-1567-9293
  • Álvarez Fernandez, Susana|||0000-0002-1376-2034
The extraction of spatio-chromatic features from color images is usually performed independently on each color channel. Usual 3D color spaces, such as RGB, present a high inter-channel correlation for natural images. This correlation can be reduced using color-opponent representations, but the spatial structure of regions with small color differences is not fully captured in two generic Red-Green and Blue-Yellow channels. To overcome these problems, we propose new color coding that is adapted to the specific content of each image. Our proposal is based on two steps: (a) setting the number of channels to the number of distinctive colors we find in each image (avoiding the problem of channel correlation), and (b) building a channel representation that maximizes contrast differences within each color channel (avoiding the problem of low local contrast). We call this approach more-than-three color coding (MTT) to emphasize the fact that the number of channels is adapted to the image content. The higher the color complexity of an image, the more channels can be used to represent it. Here we select distinctive colors as the most predominant in the image, which we call color pivots, and we build the new color coding strategy using these color pivots as a basis. To evaluate the proposed approach, we measure the efficiency in an image categorization task. We show how a generic descriptor improves performance at the description level when applied to the MTT coding.