Pasar al contenido principal

DIAGNOSTICO TEMPRANO DE ANOMALIAS CRANIOFACIALES EN NEONATOS

PID2020-114083GB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2020
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POMPEU FABRA CCT

Resultados relacionados

Resultados totales (Incluyendo duplicados): 20
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

BabyFM: towards accurate 3D baby facial models using spectral decomposition and asymmetry swapping

  • Morales Muñoz, Maria Araceli
  • Alomar Adrover, Antònia
  • Porras Pérez, Antonio Reyes
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
In this paper, we present the first publicly available 3D statistical facial shape model of babies, the Baby Face Model (BabyFM). Constructing a model of the facial geometry of babies entails specific challenges, such as occlusions, extreme and uncontrollable expressions, and data shortage. We address these challenges by proposing (1) a non-template dependent method that jointly estimates a 3D facial baby-specific template and the point-to-point correspondences; (2) a novel method to establish correspondences based on the spectral decomposition of the Laplace Beltrami Operator, which provides a more robust theoretical foundation than state-of-the-art methods; and (3) an asymmetry-swapping strategy to alleviate the shortage of large scale datasets by decoupling the identity-related and the asymmetry-related shape deformation fields. The latter leads to a data augmentation technique that we integrate within the Gaussian Process Morphable Model framework, providing a simple way of combining synthetic or sample covariance functions. We exhaustively evaluate each stage of our method and demonstrate that (1) when aiming at the 3D facial geometry of a baby, a specific model of babies is needed, since the pre-built publicly available models constructed with adults or older children are not able to accurately represent the facial shape of babies; (2) our spectral approach improves correspondences accuracy with respect to state-of-the-art-methods; and (3) the proposed data augmentation technique enhances the robustness of the BabyFM., This work is partly supported by MICIU/AEI/10.13039/501100011033/ under the project grant PID2020-114083GB-I00 and PRE2021-097544 scholarship, the ICREA Academia programme and the NIH Eunice Kennedy Shriver National Institute of Child Health & Human Development grant R42 HD08171203.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Audio-visual gated-sequenced neural networks for affect recognition

  • Aspandi, Decky
  • Sukno, Federico Mateo
  • Schuller, Björn
  • Binefa i Valls, Xavier
The interest in automatic emotion recognition and the larger field of Affective Computing has recently gained momentum. The current emergence of large, video-based affect datasets offering rich multi-modal inputs facilitates the development of deep learning-based models for automatic affect analysis that currently holds the state of the art. However, recent approaches to process these modalities cannot fully exploit them due to the use of oversimplified fusion schemes. Furthermore, the efficient use of temporal information inherent to these huge data are also largely unexplored hindering their potential progress. In this work, we propose a multi-modal, sequence-based neural network with gating mechanisms for Valence and Arousal based affect recognition. Our model consists of three major networks: Firstly, a latent-feature generator that extracts compact representations from both modalities that have been artificially degraded to add robustness. Secondly, a multi-task discriminator that estimates both input identity and a first step emotion quadrant estimation. Thirdly, a sequence-based predictor with attention and gating mechanisms that effectively merges both modalities and uses this information through sequence modelling. In our experiments on the SEMAINE and SEWA affect datasets, we observe the impact of both proposed methods with progressive increase in accuracy. We further show in our ablation studies how the internal attention weight and gating coefficient impact our models’ estimates quality. Finally, we demonstrate state of the art accuracy through comparisons with current alternatives on both datasets., This work is partly supported by the Spanish Ministry of Science and Innovation under project grant PID2020-114083GB-I00, and the donation bahi2018-19 to the CMTech at UPF. Further funding has been received from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 826506 (sustAGE) along with the UDeco project by the German BMBF-KMU Innovativ program.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

PulseFormer: continuous remote heart rate measurement through zoomed time-spectral attention

  • Comas Martínez, Joaquim
  • Ruiz Ovejero, Adrià
  • Sukno, Federico Mateo
Despite the recent advances in remote heart rate measurement, most improvements primarily focus on recovering the rPPG signal, often overlooking the inherent challenges of estimating heart rate (HR) from the derived signal. Furthermore, most existing methods adopt the average HR per video to assess model performance, thus relying on rather large temporal windows to produce a single estimate; this hampers their applicability to scenarios in which the continuous monitoring of a patient's physiological status is crucial. Besides, this evaluation approach can also lead to biased performance assessments due to low continuous precision, as it considers only the mean value of the entire video. In this paper, we present the PulseFormer, a novel continuous deep estimator for remote HR. Our proposed method utilizes a time-frequency attention block that leverages the enhanced resolution properties of the Chirp-Z Transform (CZT) to accurately estimate HR from the recovered low-resolution signal using a reduced temporal window size. We validate the effectiveness of our model on the large-scale Vision-for-Vitals (V4V) benchmark, designed for continuous physiological signals estimation from facial videos. The results reveal outstanding frame-to-frame HR estimation capabilities, establishing the proposed approach as a robust and versatile estimator that could be used with any rPPG method., This work was supported in part by the eSCANFace Project funded by the Spanish Ministry of Science and Innovation under Grant PID2020-114083GB-I00.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Deep pulse-signal magnification for remote heart rate estimation in compressed videos

  • Comas Martínez, Joaquim
  • Ruiz Ovejero, Adrià
  • Sukno, Federico Mateo
Recent advancements in data-driven approaches for remote photoplethysmography (rPPG) have significantly improved the accuracy of remote heart rate estimation. However, the performance of such approaches worsens considerably under video compression, which is nevertheless necessary to store and transmit video data efficiently. In this paper, we investigate the impact of video compression on the recovery of physiological signals from camera-based recordings. To mitigate the negative effects of compression on rPPG estimation, we propose a novel two-stage training strategy. This approach incorporates a pulse-signal magnification transformation, which adapts compressed video data into an uncompressed domain, where the rPPG signal is amplified. We validate the effectiveness of our model through comprehensive evaluations on two publicly available datasets, UCLA-rPPG and UBFC-rPPG, assessing both intra- and cross-database performance across various compression rates. Additionally, we assess the robustness of our approach on two additional highly compressed and widely-used datasets, MAHNOB-HCI and COHFACE, which reveal outstanding heart rate estimation results., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation.
Repositorio Digital de la UPF
Publicación Contribución de congreso (ConferenceOutput)

Efficient remote photoplethysmography with temporal derivative modules and time-shift invariant loss

  • Comas, Joaquim
  • Ruiz Ovejero, Adrià
  • Sukno, Federico Mateo
Comunicació presentada a 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), celebrat del 19 al 20 de juny de 2022 a Nova Orleans, Estats Units., We present a lightweight neural model for remote heart rate estimation focused on the efficient spatio-temporal learning of facial photoplethysmography (PPG) based on i) modelling of PPG dynamics by combinations of multiple convolutional derivatives, and ii) increased flexibility of the model to learn possible offsets between the facial video PPG and the ground truth. PPG dynamics are modelled by a Temporal Derivative Module (TDM) constructed by the incremental aggregation of multiple convolutional derivatives, emulating a Taylor series expansion up to the desired order. Robustness to ground truth offsets is handled by the introduction of TALOS (Temporal Adaptive LOcation Shift), a new temporal loss to train learning-based models. We verify the effectiveness of our model by reporting accuracy and efficiency metrics on the public PURE and UBFC-rPPG datasets. Compared to existing models, our approach shows competitive heart rate estimation accuracy with a much lower number of parameters and lower computational cost., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Unlocking 3D baby face photogrammetry: multi-view babymorph reconstruction from uncalibrated photographs

  • Alomar Adrover, Antònia
  • Piella Fenoy, Gemma
  • Mantilla-Rivas, Esperanza
  • Tapp, Austin
  • Porras, Antonio R.
  • Rubio, Ricardo
  • Maya-Enero, Silvia
  • Sukno, Federico Mateo
  • Linguraru, Marius George
Craniofacial anomalies are important diagnostic markers in early life. Recent studies emphasize the value of 3D imaging for extracting robust facial features that are potential indicators of disease. However, widespread availability of 3D scanning devices in hospitals remains a challenge and this type of technology may not be available in limited-resource settings. For this reason, we present a new approach to generate precise baby 3D face reconstructions from multiple uncalibrated 2D photographs acquired with a smartphone camera.The novel multi-view transformer network presented takes as input three uncalibrated photographs of the baby in frontal, left, and right pose. It then maps these images to a previously learned latent space that captures the baby's 3D facial morphology, using a 2D vision transformer encoder. Subsequently, the estimated 3D geometry is recovered by decoding the latent vector using a 3D graph convolutional network decoder. Our network demonstrates a normalized mean error of 3.29% and a root mean square error of 2.62 mm between reconstructed and true 3D faces in the baby test dataset. These outcomes are comparable with those reported by both single-view and multi-view 2D-3D reconstruction state-of-the-art errors in adult models.To conclude, the presented Multi-view BabyMorph generates highly accurate 3D baby facial reconstructions from uncalibrated photographs, simplifying the process of 3D photogrammetry. This innovation expands access to advanced baby diagnosis tools, particularly in resource-limited settings., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Deep learnable spectral decomposition of 3D baby faces

  • Zappe, Michael
  • Alomar Adrover, Antònia
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
In this paper, we introduce a novel, deep 3D morphable model for meshes with common triangulation. Specifically, we apply it to reconstruct baby faces. The proposed algorithm is simple, adaptable, and specifically targeted to perform well on small datasets. We combine Graph-Laplacian based spectral decomposition with a learnable, transformer-like component. The decomposition matrices are applied as skip-connections, providing our architecture with a prior that encodes both local and global information of the underlying mesh structure. The learnable component does not make any domain-specific assumptions and can override the prior, if necessary. This flexibility also allows our model to perform well on larger datasets. We further modify the decomposition matrices to create deeper versions of this architecture and introduce a data augmentation strategy: flipping and rotations are applied to the deviations from the mean, rather than directly to the samples. In our experiments, we compare the reconstruction error of the proposed architecture against the state of the art, examine the effect of data augmentation across a small baby face dataset and a larger adult dataset and inspect our model's capabilities to generate new samples from the encoded distribution. We show that our method outperforms current baby face models, as well as state of the art 3D morphable models, especially on the raw data. Additionally, we demonstrate that the proposed data augmentation substantially improves existing models., This work is partly supported by the Spanish Ministry of Science and Innovation under project grant PID2020-114083GB-I00, PID2024-161292OB-I00 and the NIH Eunice Kennedy Shriver National Institute of Child Health and Human Development grant R42 HD08171203. G. Piella is supported by ICREA under the ICREA Academia programme.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Accuracy and repeatability of fetal facial measurements in 3D ultrasound: A longitudinal study

  • González-Aranceta, Nerea
  • Alomar Adrover, Antònia
  • Rubio, Ricardo
  • Maya-Enero, Silvia
  • Payà Panadés, Antonio
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
Objective: Fetal face measurements in prenatal ultrasound can aid in identifying craniofacial abnormalities in the developing fetus. However, the accuracy and reliability of ultrasound measurements can be affected by factors such as fetal position, image quality, and the sonographer's expertise. This study assesses the accuracy and reliability of fetal facial measurements in prenatal ultrasound. Additionally, the temporal evolution of measurements is studied, comparing prenatal and postnatal measurements. Methods: Three different experts located up to 23 facial landmarks in 49 prenatal 3D ultrasound scans from normal Caucasian fetuses at weeks 20, 26, and 35 of gestation. Intra- and inter-observer variability was obtained. Postnatal facial measurements were also obtained at 15 days and 1 month postpartum. Results: Most facial landmarks exhibited low errors, with overall intra- and inter-observer errors of 1.01 mm and 1.60 mm, respectively. Landmarks on the nose were found to be the most reliable, while the most challenging ones were those located on the ears and eyes. Overall, scans obtained at 26 weeks of gestation presented the best trade-off between observer variability and landmark visibility. The temporal evolution of the measurements revealed that the lower face area had the highest rate of growth throughout the latest stages of pregnancy. Conclusions: Craniofacial landmarks can be evaluated using 3D fetal ultrasound, especially those located on the nose, mouth, and chin. Despite its limitations, this study provides valuable insights into prenatal and postnatal biometric changes over time, which could aid in developing predictive models for postnatal measurements based on prenatal data., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation. A. Alomar was also supported by the Spanish Ministry of Science and Innovation under the PRE2021-097544 scholarship, and G. Piella was supported by ICREA under the ICREA Academia program.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Reconstruction of the fetus face from three-dimensional ultrasound using a newborn face statistical shape model

  • Alomar Adrover, Antònia
  • Morales, Araceli
  • Vellvé, Kilian
  • Porras, Antonio R.
  • Crispi, Fatima
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
Background and objective: The fetal face is an essential source of information in the assessment of congenital malformations and neurological anomalies. Disturbance in early stages of development can lead to a wide range of effects, from subtle changes in facial and neurological features to characteristic facial shapes observed in craniofacial syndromes. Three-dimensional ultrasound (3D US) can provide more detailed information about the facial morphology of the fetus than the conventional 2D US, but its use for pre-natal diagnosis is challenging due to imaging noise, fetal movements, limited field-of-view, low soft-tissue contrast, and occlusions. Methods: In this paper, we propose the use of a novel statistical morphable model of newborn faces, the BabyFM, for fetal face reconstruction from 3D US images. We test the feasibility of using newborn statistics to accurately reconstruct fetal faces by fitting the regularized morphable model to the noisy 3D US images. Results: The results indicate that the reconstructions are quite accurate in the central-face and less reliable in the lateral regions (mean point-to-surface error of 2.35 mm vs 4.86 mm). The algorithm is able to reconstruct the whole facial morphology of babies from US scans while handle adverse conditions (e.g. missing parts, noisy data). Conclusions: The proposed algorithm has the potential to aid in-utero diagnosis for conditions that involve facial dysmorphology., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation, and the Eunice Kennedy Shriver National Institute of Child Health and Human Development grant R42HD081712. A. Alomar was supported by AGAUR under the FI scholarship and G. Piella was supported by ICREA under the ICREA Academia programme.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

BabyNet: reconstructing 3D faces of babies from uncalibrated photographs

  • Morales, Araceli
  • Alomar Adrover, Antònia
  • Porras Pérez, Antonio Reyes
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
We present a 3D face reconstruction system that aims at recovering the 3D facial geometry of babies from uncalibrated photographs, BabyNet. Since the 3D facial geometry of babies differs substantially from that of adults, baby-specific facial reconstruction systems are needed. BabyNet consists of two stages: 1) a 3D graph convolutional autoencoder learns a latent space of the baby 3D facial shape; and 2) a 2D encoder that maps photographs to the 3D latent space based on representative features extracted using transfer learning. In this way, using the pre-trained 3D decoder, we can recover a 3D face from 2D images. We evaluate BabyNet and show that 1) methods based on adult datasets cannot model the 3D facial geometry of babies, which proves the need for a baby-specific method, and 2) BabyNet outperforms classical model-fitting methods even when a baby-specific 3D morphable model, such as BabyFM, is used., This work is partly supported by the Spanish Ministry of Science and Innovation under project grant PID2020-114083GB-I00 and the NIH Eunice Kennedy Shriver National Institute of Child Health & Human Development grant R42 HD08171203. G. Piella is supported by ICREA under the ICREA Academia programme.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

PulseFormer: continuous remote heart rate measurement through zoomed time-spectral attention

  • Comas Martínez, Joaquim
  • Ruiz Ovejero, Adrià
  • Sukno, Federico Mateo
Despite the recent advances in remote heart rate measurement, most improvements primarily focus on recovering the rPPG signal, often overlooking the inherent challenges of estimating heart rate (HR) from the derived signal. Furthermore, most existing methods adopt the average HR per video to assess model performance, thus relying on rather large temporal windows to produce a single estimate; this hampers their applicability to scenarios in which the continuous monitoring of a patient's physiological status is crucial. Besides, this evaluation approach can also lead to biased performance assessments due to low continuous precision, as it considers only the mean value of the entire video. In this paper, we present the PulseFormer, a novel continuous deep estimator for remote HR. Our proposed method utilizes a time-frequency attention block that leverages the enhanced resolution properties of the Chirp-Z Transform (CZT) to accurately estimate HR from the recovered low-resolution signal using a reduced temporal window size. We validate the effectiveness of our model on the large-scale Vision-for-Vitals (V4V) benchmark, designed for continuous physiological signals estimation from facial videos. The results reveal outstanding frame-to-frame HR estimation capabilities, establishing the proposed approach as a robust and versatile estimator that could be used with any rPPG method., This work was supported in part by the eSCANFace Project funded by the Spanish Ministry of Science and Innovation under Grant PID2020-114083GB-I00.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Reconstruction of the fetus face from three-dimensional ultrasound using a newborn face statistical shape model

  • Alomar Adrover, Antònia
  • Morales, Araceli
  • Vellvé, Kilian
  • Porras, Antonio R.
  • Crispi, Fatima
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
Background and objective: The fetal face is an essential source of information in the assessment of congenital malformations and neurological anomalies. Disturbance in early stages of development can lead to a wide range of effects, from subtle changes in facial and neurological features to characteristic facial shapes observed in craniofacial syndromes. Three-dimensional ultrasound (3D US) can provide more detailed information about the facial morphology of the fetus than the conventional 2D US, but its use for pre-natal diagnosis is challenging due to imaging noise, fetal movements, limited field-of-view, low soft-tissue contrast, and occlusions. Methods: In this paper, we propose the use of a novel statistical morphable model of newborn faces, the BabyFM, for fetal face reconstruction from 3D US images. We test the feasibility of using newborn statistics to accurately reconstruct fetal faces by fitting the regularized morphable model to the noisy 3D US images. Results: The results indicate that the reconstructions are quite accurate in the central-face and less reliable in the lateral regions (mean point-to-surface error of 2.35 mm vs 4.86 mm). The algorithm is able to reconstruct the whole facial morphology of babies from US scans while handle adverse conditions (e.g. missing parts, noisy data). Conclusions: The proposed algorithm has the potential to aid in-utero diagnosis for conditions that involve facial dysmorphology., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation, and the Eunice Kennedy Shriver National Institute of Child Health and Human Development grant R42HD081712. A. Alomar was supported by AGAUR under the FI scholarship and G. Piella was supported by ICREA under the ICREA Academia programme.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

BabyFM: towards accurate 3D baby facial models using spectral decomposition and asymmetry swapping

  • Morales Muñoz, Maria Araceli
  • Alomar Adrover, Antònia
  • Porras Pérez, Antonio Reyes
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
In this paper, we present the first publicly available 3D statistical facial shape model of babies, the Baby Face Model (BabyFM). Constructing a model of the facial geometry of babies entails specific challenges, such as occlusions, extreme and uncontrollable expressions, and data shortage. We address these challenges by proposing (1) a non-template dependent method that jointly estimates a 3D facial baby-specific template and the point-to-point correspondences; (2) a novel method to establish correspondences based on the spectral decomposition of the Laplace Beltrami Operator, which provides a more robust theoretical foundation than state-of-the-art methods; and (3) an asymmetry-swapping strategy to alleviate the shortage of large scale datasets by decoupling the identity-related and the asymmetry-related shape deformation fields. The latter leads to a data augmentation technique that we integrate within the Gaussian Process Morphable Model framework, providing a simple way of combining synthetic or sample covariance functions. We exhaustively evaluate each stage of our method and demonstrate that (1) when aiming at the 3D facial geometry of a baby, a specific model of babies is needed, since the pre-built publicly available models constructed with adults or older children are not able to accurately represent the facial shape of babies; (2) our spectral approach improves correspondences accuracy with respect to state-of-the-art-methods; and (3) the proposed data augmentation technique enhances the robustness of the BabyFM., This work is partly supported by MICIU/AEI/10.13039/501100011033/ under the project grant PID2020-114083GB-I00 and PRE2021-097544 scholarship, the ICREA Academia programme and the NIH Eunice Kennedy Shriver National Institute of Child Health & Human Development grant R42 HD08171203.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Accuracy and repeatability of fetal facial measurements in 3D ultrasound: A longitudinal study

  • González-Aranceta, Nerea
  • Alomar Adrover, Antònia
  • Rubio, Ricardo
  • Maya-Enero, Silvia
  • Payà Panadés, Antonio
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
Objective: Fetal face measurements in prenatal ultrasound can aid in identifying craniofacial abnormalities in the developing fetus. However, the accuracy and reliability of ultrasound measurements can be affected by factors such as fetal position, image quality, and the sonographer's expertise. This study assesses the accuracy and reliability of fetal facial measurements in prenatal ultrasound. Additionally, the temporal evolution of measurements is studied, comparing prenatal and postnatal measurements. Methods: Three different experts located up to 23 facial landmarks in 49 prenatal 3D ultrasound scans from normal Caucasian fetuses at weeks 20, 26, and 35 of gestation. Intra- and inter-observer variability was obtained. Postnatal facial measurements were also obtained at 15 days and 1 month postpartum. Results: Most facial landmarks exhibited low errors, with overall intra- and inter-observer errors of 1.01 mm and 1.60 mm, respectively. Landmarks on the nose were found to be the most reliable, while the most challenging ones were those located on the ears and eyes. Overall, scans obtained at 26 weeks of gestation presented the best trade-off between observer variability and landmark visibility. The temporal evolution of the measurements revealed that the lower face area had the highest rate of growth throughout the latest stages of pregnancy. Conclusions: Craniofacial landmarks can be evaluated using 3D fetal ultrasound, especially those located on the nose, mouth, and chin. Despite its limitations, this study provides valuable insights into prenatal and postnatal biometric changes over time, which could aid in developing predictive models for postnatal measurements based on prenatal data., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation. A. Alomar was also supported by the Spanish Ministry of Science and Innovation under the PRE2021-097544 scholarship, and G. Piella was supported by ICREA under the ICREA Academia program.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

BabyNet: reconstructing 3D faces of babies from uncalibrated photographs

  • Morales, Araceli
  • Alomar Adrover, Antònia
  • Porras Pérez, Antonio Reyes
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
We present a 3D face reconstruction system that aims at recovering the 3D facial geometry of babies from uncalibrated photographs, BabyNet. Since the 3D facial geometry of babies differs substantially from that of adults, baby-specific facial reconstruction systems are needed. BabyNet consists of two stages: 1) a 3D graph convolutional autoencoder learns a latent space of the baby 3D facial shape; and 2) a 2D encoder that maps photographs to the 3D latent space based on representative features extracted using transfer learning. In this way, using the pre-trained 3D decoder, we can recover a 3D face from 2D images. We evaluate BabyNet and show that 1) methods based on adult datasets cannot model the 3D facial geometry of babies, which proves the need for a baby-specific method, and 2) BabyNet outperforms classical model-fitting methods even when a baby-specific 3D morphable model, such as BabyFM, is used., This work is partly supported by the Spanish Ministry of Science and Innovation under project grant PID2020-114083GB-I00 and the NIH Eunice Kennedy Shriver National Institute of Child Health & Human Development grant R42 HD08171203. G. Piella is supported by ICREA under the ICREA Academia programme.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Deep learnable spectral decomposition of 3D baby faces

  • Zappe, Michael
  • Alomar Adrover, Antònia
  • Linguraru, Marius George
  • Piella Fenoy, Gemma
  • Sukno, Federico Mateo
In this paper, we introduce a novel, deep 3D morphable model for meshes with common triangulation. Specifically, we apply it to reconstruct baby faces. The proposed algorithm is simple, adaptable, and specifically targeted to perform well on small datasets. We combine Graph-Laplacian based spectral decomposition with a learnable, transformer-like component. The decomposition matrices are applied as skip-connections, providing our architecture with a prior that encodes both local and global information of the underlying mesh structure. The learnable component does not make any domain-specific assumptions and can override the prior, if necessary. This flexibility also allows our model to perform well on larger datasets. We further modify the decomposition matrices to create deeper versions of this architecture and introduce a data augmentation strategy: flipping and rotations are applied to the deviations from the mean, rather than directly to the samples. In our experiments, we compare the reconstruction error of the proposed architecture against the state of the art, examine the effect of data augmentation across a small baby face dataset and a larger adult dataset and inspect our model's capabilities to generate new samples from the encoded distribution. We show that our method outperforms current baby face models, as well as state of the art 3D morphable models, especially on the raw data. Additionally, we demonstrate that the proposed data augmentation substantially improves existing models., This work is partly supported by the Spanish Ministry of Science and Innovation under project grant PID2020-114083GB-I00, PID2024-161292OB-I00 and the NIH Eunice Kennedy Shriver National Institute of Child Health and Human Development grant R42 HD08171203. G. Piella is supported by ICREA under the ICREA Academia programme.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Unlocking 3D baby face photogrammetry: multi-view babymorph reconstruction from uncalibrated photographs

  • Alomar Adrover, Antònia
  • Piella Fenoy, Gemma
  • Mantilla-Rivas, Esperanza
  • Tapp, Austin
  • Porras, Antonio R.
  • Rubio, Ricardo
  • Maya-Enero, Silvia
  • Sukno, Federico Mateo
  • Linguraru, Marius George
Craniofacial anomalies are important diagnostic markers in early life. Recent studies emphasize the value of 3D imaging for extracting robust facial features that are potential indicators of disease. However, widespread availability of 3D scanning devices in hospitals remains a challenge and this type of technology may not be available in limited-resource settings. For this reason, we present a new approach to generate precise baby 3D face reconstructions from multiple uncalibrated 2D photographs acquired with a smartphone camera.The novel multi-view transformer network presented takes as input three uncalibrated photographs of the baby in frontal, left, and right pose. It then maps these images to a previously learned latent space that captures the baby's 3D facial morphology, using a 2D vision transformer encoder. Subsequently, the estimated 3D geometry is recovered by decoding the latent vector using a 3D graph convolutional network decoder. Our network demonstrates a normalized mean error of 3.29% and a root mean square error of 2.62 mm between reconstructed and true 3D faces in the baby test dataset. These outcomes are comparable with those reported by both single-view and multi-view 2D-3D reconstruction state-of-the-art errors in adult models.To conclude, the presented Multi-view BabyMorph generates highly accurate 3D baby facial reconstructions from uncalibrated photographs, simplifying the process of 3D photogrammetry. This innovation expands access to advanced baby diagnosis tools, particularly in resource-limited settings., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Audio-visual gated-sequenced neural networks for affect recognition

  • Aspandi, Decky
  • Sukno, Federico Mateo
  • Schuller, Björn
  • Binefa i Valls, Xavier
The interest in automatic emotion recognition and the larger field of Affective Computing has recently gained momentum. The current emergence of large, video-based affect datasets offering rich multi-modal inputs facilitates the development of deep learning-based models for automatic affect analysis that currently holds the state of the art. However, recent approaches to process these modalities cannot fully exploit them due to the use of oversimplified fusion schemes. Furthermore, the efficient use of temporal information inherent to these huge data are also largely unexplored hindering their potential progress. In this work, we propose a multi-modal, sequence-based neural network with gating mechanisms for Valence and Arousal based affect recognition. Our model consists of three major networks: Firstly, a latent-feature generator that extracts compact representations from both modalities that have been artificially degraded to add robustness. Secondly, a multi-task discriminator that estimates both input identity and a first step emotion quadrant estimation. Thirdly, a sequence-based predictor with attention and gating mechanisms that effectively merges both modalities and uses this information through sequence modelling. In our experiments on the SEMAINE and SEWA affect datasets, we observe the impact of both proposed methods with progressive increase in accuracy. We further show in our ablation studies how the internal attention weight and gating coefficient impact our models’ estimates quality. Finally, we demonstrate state of the art accuracy through comparisons with current alternatives on both datasets., This work is partly supported by the Spanish Ministry of Science and Innovation under project grant PID2020-114083GB-I00, and the donation bahi2018-19 to the CMTech at UPF. Further funding has been received from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 826506 (sustAGE) along with the UDeco project by the German BMBF-KMU Innovativ program.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Efficient remote photoplethysmography with temporal derivative modules and time-shift invariant loss

  • Comas, Joaquim
  • Ruiz Ovejero, Adrià
  • Sukno, Federico Mateo
Comunicació presentada a 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), celebrat del 19 al 20 de juny de 2022 a Nova Orleans, Estats Units., We present a lightweight neural model for remote heart rate estimation focused on the efficient spatio-temporal learning of facial photoplethysmography (PPG) based on i) modelling of PPG dynamics by combinations of multiple convolutional derivatives, and ii) increased flexibility of the model to learn possible offsets between the facial video PPG and the ground truth. PPG dynamics are modelled by a Temporal Derivative Module (TDM) constructed by the incremental aggregation of multiple convolutional derivatives, emulating a Taylor series expansion up to the desired order. Robustness to ground truth offsets is handled by the introduction of TALOS (Temporal Adaptive LOcation Shift), a new temporal loss to train learning-based models. We verify the effectiveness of our model by reporting accuracy and efficiency metrics on the public PURE and UBFC-rPPG datasets. Compared to existing models, our approach shows competitive heart rate estimation accuracy with a much lower number of parameters and lower computational cost., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Deep pulse-signal magnification for remote heart rate estimation in compressed videos

  • Comas Martínez, Joaquim
  • Ruiz Ovejero, Adrià
  • Sukno, Federico Mateo
Recent advancements in data-driven approaches for remote photoplethysmography (rPPG) have significantly improved the accuracy of remote heart rate estimation. However, the performance of such approaches worsens considerably under video compression, which is nevertheless necessary to store and transmit video data efficiently. In this paper, we investigate the impact of video compression on the recovery of physiological signals from camera-based recordings. To mitigate the negative effects of compression on rPPG estimation, we propose a novel two-stage training strategy. This approach incorporates a pulse-signal magnification transformation, which adapts compressed video data into an uncompressed domain, where the rPPG signal is amplified. We validate the effectiveness of our model through comprehensive evaluations on two publicly available datasets, UCLA-rPPG and UBFC-rPPG, assessing both intra- and cross-database performance across various compression rates. Additionally, we assess the robustness of our approach on two additional highly compressed and widely-used datasets, MAHNOB-HCI and COHFACE, which reveal outstanding heart rate estimation results., This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation.