Pasar al contenido principal

MODELIZACION DE HUMANOS EN EL CONTEXTO

PID2020-120049RB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2020
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS (CSIC)

Resultados relacionados

Resultados totales (Incluyendo duplicados): 61
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Body size and depth disambiguation in multi-person reconstruction from single images

  • Ugrinovic, Nicolás
  • Ruiz Ovejero, Adrià
  • Agudo Martínez, Antonio
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Trabajo presentado en la 9th International Conference on 3D Vision, celebrada online del 1 al 3 de diciembre de 2021, We address the problem of multi-person 3D body pose and shape estimation from a single image. While this problem can be addressed by applying single-person approaches multiple times for the same scene, recent works have shown the advantages of building upon deep architectures that simultaneously reason about all people in the scene in a holistic manner by enforcing, e.g., depth order constraints or minimizing interpenetration among reconstructed bodies. However, existing approaches are still unable to capture the size variability of people caused by the inherent body scale and depth ambiguity. In this work we tackle this challenge by devising a novel optimization scheme that learns the appropriate body scale and relative camera pose, by enforcing the feet of all people to remain on the ground floor. A thorough evaluation on MuPoTS-3D and 3DPW datasets demonstrates that our approach is able to robustly estimate the body translation and shape of multiple people while retrieving their spatial arrangement, consistently improving current state-of-the-art, especially in scenes with people of very different heights., This work is partly supported by the Spanish government with the project MoHuCo PID2020- 120049RB-I00, the ERA-Net Chistera project IPALM PCI2019- 103386 and Mar´ıa de Maeztu Seal of Excellence MDM-2016- 0656. Adria Ruiz acknowledges financial support from MICINN (Spain) through the program Juan de la Cierva.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Generating attribution maps with disentangled masked backpropagation

  • Ruiz Ovejero, Adrià
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Trabajo presentado en la International Conference on Computer Vision (ICCV), celebrada de forma virtual del 11 al 17 de octubre de 2021, Attribution map visualization has arisen as one of the most effective techniques to understand the underlying inference process of Convolutional Neural Networks. In this task, the goal is to compute an score for each image pixel related to its contribution to the network output. In this paper, we introduce Disentangled Masked Backpropagation (DMBP), a novel gradient-based method that leverages on the piecewise linear nature of ReLU networks to decompose the model function into different linear mappings. This decomposition aims to disentangle the attribution maps into positive, negative and nuisance factors by learning a set of variables masking the contribution of each filter during back-propagation. A thorough evaluation over standard architectures (ResNet50 and VGG16) and benchmark datasets (PASCAL VOC and ImageNet) demonstrates that DMBP generates more visually interpretable attribution maps than previous approaches. Additionally, we quantitatively show that the maps produced by our method are more consistent with the true contribution of each pixel to the final network output., This work has been partially funded by the Spanish government with the project MoHuCo PID2020-120049RB-I00. Adria Ruiz acknowledges financial support from MICINN (Spain) through the program Juan de la Cierva.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

PhysXNet: A customizable approach for learning cloth dynamics on dressed people

  • Sánchez-Riera, Jordi
  • Pumarola, Albert
  • Moreno-Noguer, Francesc
Trabajo presentado en la International Conference on Computer Vision (ICCV), celebrada de forma virtual del 11 al 17 de octubre de 2021, We introduce PhysXNet, a learning-based approach to predict the dynamics of deformable clothes given 3D skeleton motion sequences of humans wearing these clothes. The proposed model is adaptable to a large variety of garments and changing topologies, without need of being retrained. Such simulations are typically carried out by physics engines that require manual human expertise and are subject to computationally intensive computations. PhysXNet, by contrast, is a fully differentiable deep network that at inference is able to estimate the geometry of dense cloth meshes in a matter of milliseconds, and thus, can be readily deployed as a layer of a larger deep learning architecture. This efficiency is achieved thanks to the specific parameterization of the clothes we consider, based on 3D UV maps encoding spatial garment displacements., This work is supported partly by the Spanish government under project MoHuCo PID2020-120049RB-I00, the ERANet Chistera project IPALM PCI2019-103386 and María de Maeztu Seal of Excellence MDM-2016-0656., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

SIDER: Single-image neural optimization for facial geometric detail recovery

  • Chatziagapi, Aggelina
  • Athar, ShahRukh
  • Moreno-Noguer, Francesc
  • Samaras, Dimitris
Trabajo presentado en la International Conference on Computer Vision (ICCV), celebrada de forma virtual del 11 al 17 de octubre de 2021, In this work we present Sider, a method for high-fidelity detailed 3D face reconstruction from a single image that can be trained in an unsupervised manner. Our approach combines the best from classical statistical models and recent implicit neural representations. The former is used to obtain a coarse shape prior, and the latter provides high-frequency geometric detail, by only optimizing over a photometric loss computed w.r.t. the input image. A thorough quantitative and qualitative evaluation shows that Sider outperforms current state-of-the-art by a significant margin. A limitation of our current approach is that it still cannot handle details like hair or beards and accessories such as glasses. This is because the photometric loss for these regions would require sub-pixel accuracy. In the future, we will explore alternatives for addressing this type of high-frequency details., This work is partly supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and Mar´ıa de Maeztu Seal of Excellence MDM-2016-0656. This work was also supported by a gift from Adobe, Partner University Fund 4DVision Project, and the SUNY2020 Infrastructure Transportation Security Center
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Conditional-Flow NeRF: Accurate 3D modelling with reliable uncertainty quantification

  • Shen, Jianxiong
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
  • Ruiz Ovejero, Adrià
Trabajo presentado en la 17th European Conference on Computer Vision, celebrada en Tel Aviv (Israel), los días 23 y 27 de octubre de 2022, A critical limitation of current methods based on Neural Radiance Fields (NeRF) is that they are unable to quantify the uncertainty associated with the learned appearance and geometry of the scene. This information is paramount in real applications such as medical diagnosis or autonomous driving where, to reduce potentially catastrophic failures, the confidence on the model outputs must be included into the decision-making process. In this context, we introduce Conditional-Flow NeRF (CF-NeRF), a novel probabilistic framework to incorporate uncertainty quantification into NeRF-based approaches. For this purpose, our method learns a distribution over all possible radiance fields modelling which is used to quantify the uncertainty associated with the modelled scene. In contrast to previous approaches enforcing strong constraints over the radiance field distribution, CF-NeRF learns it in a flexible and fully data-driven manner by coupling Latent Variable Modelling and Conditional Normalizing Flows. This strategy allows to obtain reliable uncertainty estimation while preserving model expressivity. Compared to previous state-of-the-art methods proposed for uncertainty quantification in NeRF, our experiments show that the proposed method achieves significantly lower prediction errors and more reliable uncertainty values for synthetic novel view and depth-map estimation., This work is supported partly by the Chinese Scholarship Council (CSC) under grant (201906120031), by the Spanish government under project MoHuCo PID2020-120049RB-I00 and the Chistera project IPALM PCI2019-103386. We also thank Nvidia for hardware donation under the GPU grant program.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Uncalibrated, unified and unsupervised specular-aware photometric stereo

  • Estévez Graña, Pablo
  • Agudo Martínez, Antonio
Trabajo presentado en el International Workshop on Towards a Complete Analysis of People: From Face and Body to Clothes (T-CAP), celebrado en el Canadá, del 23 al 27 de mayo de 2022, In this paper we present a variational approach to simultaneously recover the 3D reconstruction, reflectance, lighting and specularities of an object, all of them, from a set of RGB images. The approach works in an uncalibrated, unified and unsupervised manner, without assuming any prior knowledge of the shape geometry or training data to constrain the solution and under general lighting. To this end, the approach exploits a physically-aware image formation model that in combination with a perspective projection one and under spherical harmonics lighting gives a fully interpretable algorithm. Integrability is implicitly ensured as the shape is coded by a depth map rather than normal vectors. As a consequence, a wide variety of illumination conditions and complex geometries can be acquired. Our claims have been experimentally validated on challenging synthetic and real datasets, obtaining a good trade-off between accuracy and computational budget in comparison with competing approaches., This work has been partially supported by the Spanish Ministry of Science and Innovation under project MoHuCo PID2020-120049RB.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

An adaptable approach to learn realistic legged locomotion without examples

  • Ordoñez Apraez, Daniel Felipe
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
  • Martín, Mario
Trabajo presentado en el International Conference on Robotics and Automation (ICRA), celebrado en Philadelphia, PA (Estados Unidos), del 23 al 27 de mayo de 2022, Learning controllers that reproduce legged locomotion in nature has been a long-time goal in robotics and computer graphics. While yielding promising results, recent approaches are not yet flexible enough to be applicable to legged systems of different morphologies. This is partly because they often rely on precise motion capture references or elaborate learning environments that ensure the naturality of the emergent locomotion gaits but prevent generalization. This work proposes a generic approach for ensuring realism in locomotion by guiding the learning process with the spring-loaded inverted pendulum model as a reference. Leveraging on the exploration capacities of Reinforcement Learning (RL), we learn a control policy that fills in the information gap between the template model and full-body dynamics required to maintain stable and periodic locomotion. The proposed approach can be applied to robots of different sizes and morphologies and adapted to any RL technique and control architecture. We present experimental results showing that even in a model-free setup and with a simple reactive control architecture, the learned policies can generate realistic and energy-efficient locomotion gaits for a bipedal and a quadrupedal robot. And most importantly, this is achieved without using motion capture, strong constraints in the dynamics or kinematics of the robot, nor prescribing limb coordination. We provide supplemental videos for qualitative analysis of the naturality of the learned gaits., This work’s experiments were run at the Barcelona Supercomputing Center in collaboration with the HPAI group. This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net Chistera project IPALM PCI2019-103386.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Safari from visual signals: Recovering volumetric 3D shapes

  • Agudo Martínez, Antonio
Trabajo presentado en la International Conference on Acoustics, Speech and Signal Processing (ICASSP), celebrada en Singapur, del 23 al 27 de mayo de 2022, In this paper we propose a convex approach for recovering a detailed 3D volumetric geometry of several objects from visual signals. To this end, we first present a minimal detailed surface energy that is optimized together with a volume constraint by considering some geometrical priors, and without requiring neither additional training data nor templates in order to constrain the solution. Our problem can be efficiently solved by means of a gradient descent, and be applied for single RGB images or monocular videos even with very small rigid motions. Temporal-aware solutions and driven by point correspondences are incorporated without assuming any 2D tracking data over time. Thanks to this formulation, both rigid and non-rigid objects can be considered. We have extensively validated our approach in a wide variety of scenarios in the wild, recovering challenging type of shapes that have not been previously attempted without assuming any training data., This work has been partially supported by the Spanish Ministry of Science and Innovation under project MoHuCo PID2020-120049RB.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Enhancing egocentric 3D pose estimation with third person views

  • Dhamanaskar, Ameya
  • Dimiccoli, Mariella
  • Corona, Enric
  • Pumarola, Albert
  • Moreno-Noguer, Francesc
We propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. The main technical contribution consists of leveraging high-level features linking first- and third-views in a joint embedding space. To learn such embedding space we introduce First2Third-Pose, a new paired synchronized dataset of nearly 2000 videos depicting human activities captured from both first- and third-view perspectives. We explicitly consider spatial- and motion-domain features, combined using a semi-Siamese architecture trained in a self-supervised fashion. Experimental results demonstrate that the joint multi-view embedded space learned with our dataset is useful to extract discriminatory features from arbitrary single-view egocentric videos, with no need to perform any sort of domain adaptation or knowledge of camera parameters. An extensive evaluation demonstrates that we achieve significant improvement in egocentric 3D body pose estimation performance on two unconstrained datasets, over three supervised state-of-the-art approaches. The collected dataset and pre-trained model are available for research purposes., This work has been partially supported by projects PID2020-120049RB-I00 and PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033 and by “ESF Investing in your future”, and network RED2018-102511-T funded by MCIN/ AEI., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

NeRFLight: Fast and Light Neural Radiance Fields using a Shared Feature Grid

  • Rivas-Manzaneque, Fernando
  • Sierra-Acosta, Jorge
  • Penate-Sanchez, Adrian
  • Moreno-Noguer, Francesc
  • Ribeiro Seijas, Ángela
Trabajo presentado en la Conference on Computer Vision and Pattern Recognition (CVPR), celebrada en Vancuver (Canadá), del 17 al 24 de junio de 2023, While original Neural Radiance Fields (NeRF) have shown impressive results in modeling the appearance of a scene with compact MLP architectures, they are not able to achieve real-time rendering. This has been recently addressed by either baking the outputs of NeRF into a data structure or arranging trainable parameters in an explicit feature grid. These strategies, however, significantly increase the memory footprint of the model which prevents their deployment on bandwidth-constrained applications. In this paper, we extend the grid-based approach to achieve real-time view synthesis at more than 150 FPS using a lightweight model. Our main contribution is a novel architecture in which the density field of NeRF-based representations is split into N regions and the density is modeled using N different decoders which reuse the same feature grid. This results in a smaller grid where each feature is located in more than one spatial position, forcing them to learn a compact representation that is valid for different parts of the scene. We further reduce the size of the final model by disposing of the features symmetrically on each region, which favors feature pruning after training while also allowing smooth gradient transitions between neighboring voxels. An exhaustive evaluation demonstrates that our method achieves real-time performance and quality metrics on a pair with state-of-the-art with an improvement of more than 2× in the FPS/MB ratio., This work is partially supported by the European Union under the project FlexiGroBots (DOI: 10.3030/101017111) and by the Spanish government under projects MoHuCo (PID2020-120049RB-I00) and SmartWEED-DARWEEM (PID2020-113229RB-C43). Adrian Penate-Sanchez is supported by a Beatriz Galindo grant. The authors would like to thank Marcos Barrios for his support in the graphic design of the figures and videos included in the paper, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Morphological symmetries in robot learning

  • Ordoñez Apraez, Daniel Felipe
  • Martín, Mario
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Trabajo presentado en el Workshop on Symmetries in Robot Learning, celebrado en Daegu (Corea del Sur), del 10 al 14 de julio de 2023, This work studies the impact of morphological symmetries in learning applications in robotics. Morphological symmetries are a predominant feature in both biological and robotic systems, arising from the presence of planes/axis of symmetry in the system's morphology. This results in harmonious duplication and distribution of body parts (e.g., humans' sagittal/left-right symmetry). Morphological symmetries become a significant learning prior as they extend to symmetries in the system's dynamics, optimal control policies, and in all proprioceptive and exteroceptive measurements, related to the system's dynamics evolution \cite{ordonez2023discrete}. Exploiting these symmetries in learning applications offers several advantageous outcomes, such as the use of data augmentation to mitigate the cost and challenges of data collection, or the use of equivariant/invariant function approximation models (e.g., neural networks) to improve sample efficiency and generalization, while reducing the number of trainable parameters. Lastly, we provide a video presentation and an open access repository reproducing our experiments and allowing for rapid prototyping in robot learning applications exploiting morphological symmetries., This work’s experiments were run at the Barcelona Supercomputing Center in collaboration with the HPAI group. This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net Chistera project IPALM PCI2019-103386.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Detail-Aware Uncalibrated Photometric Stereo

  • Agudo Martínez, Antonio
Trabajo presentado en la International Conference on Acoustics, Speech and Signal Processing (ICASSP), celebrada en Rodas (Grecia), del 4 al 10 de junio de 2023, Photometric stereo is the problem of jointly inferring the 3D reconstruction, reflectance, lighting and specularities of an object from a set of visual signals. Recently, some variational, uncalibrated, unsupervised and unified formulations have provided robust solutions to the problem while reducing the prior knowledge about the shape geometry or the lighting conditions. Unfortunately, these approaches cannot still produce solutions with an ample variety of details in the shape. That is mainly due to the non-convex and non-linear nature of the problem which requires the best initialization as possible. In this context, we propose a fully interpretable formulation that combines a physically-aware image formation model under perspective projection with a minimal detail-aware initialization and that it can handle general lighting. As a result, our formulation can consider multiple scenarios composed of unknown complex geometries and lighting patterns. Experimental results on challenging synthetic and real datasets show the effectiveness of our approach to capture more fine details, outperforming state-of-the-art techniques in terms of 3D reconstruction., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Robust Wind Turbine Blade Segmentation from RGB Images in the Wild

  • Pérez-Gonzalo, Raül
  • Espersen, Andreas
  • Agudo Martínez, Antonio
Trabajo presentado en la IEEE International Conference on Image Processing (ICIP), celebradad en Kuala Lumpur (Malasia), del 8 al 11 de octubre de 2023, With the relentless growth of the wind industry, there is an imperious need to design automatic data-driven solutions for wind turbine maintenance. As structural health monitoring mainly relies on visual inspections, the first stage in any automatic solution is to identify the blade region on the image. Thus, we propose a novel segmentation algorithm that strengthens the U-Net results by a tailored loss, which pools the focal loss with a contiguity regularization term. To attain top performing results, a set of additional steps are proposed to ensure a reliable, generic, robust and efficient algorithm. First, we leverage our prior knowledge on the images by filling the holes enclosed by temporarily-classified blade pixels and by the image boundaries. Subsequently, the mislead classified pixels are successfully amended by training an on-the-fly random forest. Our algorithm demonstrates its effectiveness reaching a non-trivial 97.39% of accuracy., This work has been supported by the Innovation Fund Denmark under 2021 ID1044-0044A and by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Back to MLP: A Simple Baseline for Human Motion Prediction

  • Guo, Wen
  • Du, Yuming
  • Shen, Xi
  • Lepetit, Vincent
  • Alameda-Pineda, Xavier
  • Moreno-Noguer, Francesc
Trabajo presentado en la IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), celebrada en Waikoloa, HI (Estados Unidos), del 2 al 7 de enero de 2023, This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. State-of-the-art approaches provide good results, however, they rely on deep learning architectures of arbitrary complexity, such as Recurrent Neural Networks (RNN), Transformers or Graph Convolutional Networks (GCN), typically requiring multiple training stages and more than 2 million parameters. In this paper, we show that, after combining with a series of standard practices, such as applying Discrete Cosine Transform (DCT), predicting residual displacement of joints and optimizing velocity as an auxiliary loss, a light-weight network based on multi-layer perceptrons (MLPs) with only 0.14 million parameters can surpass the state-of-the-art performance. An exhaustive evaluation on the Human3.6M, AMASS, and 3DPW datasets shows that our method, named siMLpe, consistently outperforms all other approaches. We hope that our simple method could serve as a strong baseline for the community and allow re-thinking of the human motion prediction problem. The code is publicly available at https://github.com/dulucas/siMLPe., This research was supported by ANR-3IA MIAI (ANR- 19-P3IA-0003), ANR-JCJC ML3RI (ANR-19-CE33-0008- 01), H2020 SPRING (funded by EC under GA #871245), by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and by an Amazon Research Award. This project has received funding from the CHISTERA IPALM project., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

LayerNet: High-Resolution Semantic 3D Reconstruction of Clothed People

  • Corona, Enric
  • Alenyà, Guillem
  • Pons-Moll, Gerard
  • Moreno-Noguer, Francesc
In this article, we introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry; and LayerNet, a deep network that given a single image of a person simultaneously performs detailed 3D reconstruction of body and clothes. In contrast to existing learning-based approaches that require training specific models for each type of garment, SMPLicit can represent in a unified manner different garment topologies (e.g. from sleeveless tops to hoodies and open jackets), while controlling other properties like garment size or tightness/looseness. LayerNet follows a coarse-to-fine multi-stage strategy by first predicting smooth cloth geometries from SMPLicit, which are then refined by an image-guided displacement network that gracefully fits the body recovering high-frequency details and wrinkles. LayerNet achieves competitive accuracy in the task of 3D reconstruction against current 'garment-agnostic' state of the art for images of people in up-right positions and controlled environments, and consistently surpasses these methods on challenging body poses and uncontrolled settings. Furthermore, the semantically rich outcome of our approach is suitable for performing Virtual Try-on tasks directly on 3D, a task which, so far, has only been addressed in the 2D domain., This work is supported in part by the Spanish government with the project MoHuCo PID2020-120049RBI00, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual Humans) and German Federal Ministry of Education and Research (BMBF): Tubingen AI ¨ Center, FKZ: 01IS18039A. Gerard Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The project was made possible by funding from the Carl Zeiss Foundation. We thank NVIDIA for donating GPUs., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Generalized Nested Latent Variable Models For Lossy Coding Applied To Wind Turbine Scenarios

  • Pérez-Gonzalo, Raül
  • Espersen, Andreas
  • Agudo Martínez, Antonio
Rate-distortion optimization through neural networks has accomplished competitive results in compression efficiency and image quality. This learning-based approach seeks to minimize the compromise between compression rate and reconstructed image quality by automatically extracting and retaining crucial information, while discarding less critical details. A successful technique consists in introducing a deep hyperprior that operates within a 2-level nested latent variable model, enhancing compression by capturing complex data dependencies. This paper extends this concept by designing a generalized L-level nested generative model with a Markov chain structure. We demonstrate as L increases that a trainable prior is detrimental and explore a common dimensionality along the distinct latent variables to boost compression performance. As this structured framework can represent autoregressive coders, we outperform the hyperprior model and achieve state-of-the-art performance while reducing substantially the computational cost. Our experimental evaluation is performed on wind turbine scenarios to study its application on visual inspections., This work has been supported by the Innovation Fund Denmark under 2021 ID1044-0044A and by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

InstantAvatar: Efficient 3D Head Reconstruction via Surface Rendering

  • Canela, Antonio
  • Caselles, Pol
  • Malik, Ibrar
  • Ramon, Eduard
  • Martínez García, Jaime
  • Sánchez-Riera, Jordi
  • Triginer, Gil
  • Moreno-Noguer, Francesc
Recent advances in full-head reconstruction have been obtained by optimizing a neural field through differentiable surface or volume rendering to represent a single scene. While these techniques achieve an unprecedented accuracy, they take several minutes, or even hours, due to the expensive optimization process required. In this work, we introduce InstantAvatar, a method that recovers full-head avatars from few images (down to just one) in a few seconds on commodity hardware. In order to speed up the reconstruction process, we propose a system that combines, for the first time, a voxel-grid neural field representation with a surface renderer. Notably, a naive combination of these two techniques leads to unstable optimizations that do not converge to valid solutions. In order to overcome this limitation, we present a novel statistical model that learns a prior distribution over 3D head signed distance functions using a voxel-grid based architecture. The use of this prior model, in combination with other design choices, results into a system that achieves 3D head reconstructions with comparable accuracy as the state-of-the-art with a 100 × speed-up., This work has been partially supported by: MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033; DeeLight PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033; and the industrial doctorate 2021 DI 00102 funded by the Government of Catalonia., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

InstantGeoAvatar: Effective Geometry and Appearance Modeling of Animatable Avatars from Monocular Video

  • Budria, Alvaro
  • Lopez-Rodriguez, Adrian
  • Lorente, Òscar
  • Moreno-Noguer, Francesc
We present InstantGeoAvatar, a method for efficient and effective learning from monocular video of detailed 3D geometry and appearance of animatable implicit human avatars. Our key observation is that the optimization of a hash grid encoding to represent a signed distance function (SDF) of the human subject is fraught with instabilities and bad local minima. We thus propose a principled geometry-aware SDF regularization scheme that seamlessly fits into the volume rendering pipeline and adds negligible computational overhead. Our regularization scheme significantly outperforms previous approaches for training SDFs on hash grids. We obtain competitive results in geometry reconstruction and novel view synthesis in as little as five minutes of training time, a significant reduction from the several hours required by previous work. InstantGeoAvatar represents a significant leap forward towards achieving interactive reconstruction of virtual avatars., This work has been supported by the project MOHUCO PID2020-120049RB-I00 funded by MCIU/AEI/10.13039/501100011033, and by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

MultiPhys: Multi-Person Physics-Aware 3D Motion Estimation

  • Ugrinovic, Nicolas
  • Pan, Boxiao
  • Pavlakos, Georgios
  • Paschalidou, Despoina
  • Shen, Bokui
  • Sánchez-Riera, Jordi
  • Moreno-Noguer, Francesc
  • Guibas, Leonidas
We introduce MultiPhys, a method designed for recovering multi-person motion from monocular videos. Our focus lies in capturing coherent spatial placement between pairs of individuals across varying degrees of engagement. MultiPhys, being physically aware, exhibits robustness to jittering and occlusions, and effectively eliminates penetration issues between the two individuals. We devise a pipeline in which the motion estimated by a kinematic-based method is fed into a physics simulator in an autoregressive manner. We introduce distinct components that enable our model to harness the simulator's properties without compromising the accuracy of the kinematic estimates. This results in final motion estimates that are both kinematically coherent and physically compliant. Extensive evaluations on three challenging datasets characterized by substantial inter-person interaction show that our method significantly reduces errors associated with penetration and foot skating, while performing competitively with the state-of-the-art on motion accuracy and smoothness. Results and code can be found in our project page., Despoina Paschalidou is supported by the Swiss National Science Foundation under grant number P500PT 206946. Leonidas Guibas is supported by a Vannevar Bush Faculty Fellowship. This work is partially supported by projects SMARTGAZEII CPP2021-008760 and MoHuCo PID2020-120049RB-I00., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

No Bells, Just Whistles: Sports Field Registration by Leveraging Geometric Properties

  • Gutiérrez-Pérez, Marc
  • Agudo Martínez, Antonio
Broadcast sports field registration is traditionally addressed as a homography estimation task, mapping the visible image area to a planar field model, predominantly focusing on the main camera shot. Addressing the shortcomings of previous approaches, we propose a novel calibration pipeline enabling camera calibration using a 3D soccer field model and extending the process to assess the multiple-view nature of broadcast videos. Our approach begins with a keypoint generation pipeline derived from SoccerNet dataset annotations, leveraging the geometric properties of the court. Subsequently, we execute classical camera calibration through DLT algorithm in a minimalist fashion, without further refinement. Through extensive experimentation on real-world soccer broadcast datasets such as SoccerNet-Calibration, WorldCup 2014 and TS-WorldCup, our method demonstrates superior performance in both multiple- and single-view 3D camera calibration while maintaining competitive results in homography estimation compared to state-of-the-art techniques., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Photovoltaic Power Forecasting Using Sky Images and Sun Motion

  • Berresheim, Arne
  • Agudo Martínez, Antonio
Solar energy adoption is moving at a rapid pace. The variability in solar energy production causes grid stability issues and hinders mass adoption. To solve these issues, more accurate photovoltaic power forecasting systems are needed. In intra-hour forecasting, the most challenging issue is high output fluctuations due to cloud motion, which can occlude the sun. Using ground-based sky images, this paper proposes two convolutional neural network models for intra-hour nowcasting and forecasting that incorporate physical information on sun motion and cloud coverage by means of the sun area mean pixel intensity. Particularly, our models exploit that information instead of relying exclusively on photovoltaic output history data as it is standard in state of the art. Taking advantage of sun position and cloud coverage information, we were able to reduce the overall root mean squared error for the nowcasting task, making the model more accurate especially during cloudy days, and obtaining competitive results on forecasting. Moreover, our models are more robust against artifacts such as occlusion and noisy observations., This work has been supported by the project MoHuCo PID2020- 120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

PIRO: Permutation-invariant relational network for multi-person 3D pose estimation

  • Ugrinovic, Nicolás
  • Ruiz Ovejero, Adrià
  • Agudo Martínez, Antonio
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Recovering multi-person 3D poses from a single RGB image is an ill-conditioned problem due to the inherent 2D-3D depth ambiguity, inter-person occlusions, and body truncation. To tackle these issues, recent works have shown promising results by simultaneously reasoning for different individuals. However, in most cases this is done by only considering pairwise inter-person interactions or between pairs of body parts, thus hindering a holistic scene representation able to capture long-range interactions. Some approaches that jointly process all people in the scene require defining one of the individuals as a reference and a pre-defined person ordering or limiting the number of individuals thus being sensitive to these choice. In this paper, we overcome both these limitations, and we propose an approach for multi-person 3D pose estimation that captures longrange interactions independently of the input order. We build a residual-like permutation-invariant network that successfully refines potentially corrupted initial 3D poses estimated by off-the-shelf detectors. The residual function is learned via a Set Attention (Lee et al., 2019) mechanism. Despite of our model being relatively straightforward, a thorough evaluation demonstrates that our approach is able to boost the performance of the initially estimated 3D poses by large margins, achieving state-of-the-art results on two standardized benchmarks, This work is supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

PoseEmbroider: Towards a 3D, Visual, Semantic-Aware Human Pose Representation

  • Delmas, Ginger
  • Weinzaepfel, Philippe
  • Moreno-Noguer, Francesc
  • Rogez, Grégory
Aligning multiple modalities in a latent space, such as images and texts, has shown to produce powerful semantic visual representations, fueling tasks like image captioning, text-to-image generation, or image grounding. In the context of human-centric vision, albeit CLIP-like representations encode most standard human poses relatively well (such as standing or sitting), they lack sufficient acuteness to discern detailed or uncommon ones. Actually, while 3D human poses have been often associated with images (e.g. to perform pose estimation or pose-conditioned image generation), or more recently with text (e.g. for text-to-pose generation), they have seldom been paired with both. In this work, we combine 3D poses, person’s pictures and textual pose descriptions to produce an enhanced 3D-, visual- and semantic-aware human pose representation. We introduce a new transformer-based model, trained in a retrieval fashion, which can take as input any combination of the aforementioned modalities. When composing modalities, it outperforms a standard multi-modal alignment retrieval model, making it possible to sort out partial information (e.g. image with the lower body occluded). We showcase the potential of such an embroidered pose representation for (1) SMPL regression from image with optional text cue; and (2) on the task of fine-grained instruction generation, which consists in generating a text that describes how to move from one 3D pose to another (as a fitness coach). Unlike prior works, our model can take any kind of input (image and/or pose) without retraining., This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00, and by NAVER LABS Europe under technology transfer contract ‘Text4Pose’., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Uncalibrated and Unsupervised Photometric Stereo with Piecewise Regularizer

  • Casanova, Alejandro
  • Agudo Martínez, Antonio
Photometric stereo is a technique for recovering a rigid object's 3D shape, reflectance properties, lighting conditions, and specular highlights from multiple images captured under varying lighting conditions. Variational, uncalibrated, and unsupervised formulations have recently provided detailed and robust solutions to the problem, reducing the need for prior knowledge about shape geometry or lighting conditions. However, uncalibrated methods, especially when applied to real-world data, may be susceptible to noise and depth errors near boundaries or self-occlusions, stemming from missing or noisy data, surface orientation ambiguity, and calibration issues. In this paper, we introduce a novel piecewise depth regularizer to mitigate these errors, enhancing stability and improving robustness against initialization errors. We demonstrate the effectiveness of our approach through evaluations on both synthetic and real-world data, showcasing its promise in enhancing the accuracy and reliability of photometric stereo for practical applications., This work has been supported by the project MoHuCo PID2020- 120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Back to MLP: a simple baseline for human motion prediction

  • Guo, Wen
  • Du, Yuming
  • Shen, Xi
  • Lepetit, Vincent
  • Alameda-Pineda, Xavier
  • Moreno-Noguer, Francesc
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. State-of-the-art approaches provide good results, however, they rely on deep learning architectures of arbitrary complexity, such as Recurrent Neural Networks (RNN), Transformers or Graph Convolutional Networks (GCN), typically requiring multiple training stages and more than 2 million parameters. In this paper, we show that, after combining with a series of standard practices, such as applying Discrete Cosine Transform (DCT), predicting residual displacement of joints and optimizing velocity as an auxiliary loss, a light-weight network based on multi-layer perceptrons (MLPs) with only 0.14 million parameters can surpass the state-of-the-art performance. An exhaustive evaluation on the Human3.6M, AMASS, and 3DPW datasets shows that our method, named siMLpe, consistently outperforms all other approaches. We hope that our simple method could serve as a strong baseline for the community and allow re-thinking of the human motion prediction problem. The code is publicly available at https://github.com/dulucas/siMLPe., This research was supported by ANR-3IA MIAI (ANR19-P3IA-0003), ANR-JCJC ML3RI (ANR-19-CE33-0008-01), H2020 SPRING (funded by EC under GA #871245), by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and by an Amazon Research Award. This project has received funding from the CHISTERA IPALM project., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

InstantGeoAvatar: effective geometry and appearance modeling of animatable avatars from monocular video

  • Budría Fernández, Álvaro Francesc
  • Lopez Rodriguez, Adrián
  • Lorente Corominas, Òscar
  • Moreno-Noguer, Francesc
Lecture Notes in Computer Science (LNCS, volume 15477) (c) 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd., We present InstantGeoAvatar, a method for efficient and effective learning from monocular video of detailed 3D geometry and appearance of animatable implicit human avatars. Our key observation is that the optimization of a hash grid encoding to represent a signed distance function (SDF) of the human subject is fraught with instabilities and bad local minima. We thus propose a principled geometry-aware SDF regularization scheme that seamlessly fits into the volume rendering pipeline and adds negligible computational overhead. Our regularization scheme significantly outperforms previous approaches for training SDFs on hash grids. We obtain competitive results in geometry reconstruction and novel view synthesis in as little as five minutes of training time, a significant reduction from the several hours required by previous work. InstantGeoAvatar represents a significant leap forward towards achieving interactive reconstruction of virtual avatars.
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Photovoltaic power forecasting using sky images and sun motion

  • Berresheim, Arne
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Solar energy adoption is moving at a rapid pace. The variability in solar energy production causes grid stability issues and hinders mass adoption. To solve these issues, more accurate photovoltaic power forecasting systems are needed. In intra-hour forecasting, the most challenging issue is high output fluctuations due to cloud motion, which can occlude the sun. Using ground-based sky images, this paper proposes two convolutional neural network models for intra-hour nowcasting and forecasting that incorporate physical information on sun motion and cloud coverage by means of the sun area mean pixel intensity. Particularly, our models exploit that information instead of relying exclusively on photovoltaic output history data as it is standard in state of the art. Taking advantage of sun position and cloud coverage information, we were able to reduce the overall root mean squared error for the nowcasting task, making the model more accurate especially during cloudy days, and obtaining competitive results on forecasting. Moreover, our models are more robust against artifacts such as occlusion and noisy observations., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Uncalibrated and unsupervised photometric stereo with piecewise regularizer

  • Casanova Martin, Alejandro
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
Photometric stereo is a technique for recovering a rigid object's 3D shape, reflectance properties, lighting conditions, and specular highlights from multiple images captured under varying lighting conditions. Variational, uncalibrated, and unsupervised formulations have recently provided detailed and robust solutions to the problem, reducing the need for prior knowledge about shape geometry or lighting conditions. However, uncalibrated methods, especially when applied to real-world data, may be susceptible to noise and depth errors near boundaries or self-occlusions, stemming from missing or noisy data, surface orientation ambiguity, and calibration issues. In this paper, we introduce a novel piecewise depth regularizer to mitigate these errors, enhancing stability and improving robustness against initialization errors. We demonstrate the effectiveness of our approach through evaluations on both synthetic and real-world data, showcasing its promise in enhancing the accuracy and reliability of photometric stereo for practical applications., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

LayerNet: high-resolution semantic 3D reconstruction of clothed people

  • Corona Puyane, Enric
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Pons-Moll, Gerard
  • Moreno-Noguer, Francesc
In this paper we introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry; and LayerNet, a deep network that given a single image of a person simultaneously performs detailed 3D reconstruction of body and clothes. In contrast to existing learning-based approaches that require training specific models for each type of garment, SMPLicit can represent in a unified manner different garment topologies (eg from sleeveless tops to hoodies and open jackets), while controlling other properties like garment size or tightness/looseness. LayerNet follows a coarse-to-fine multi-stage strategy by first predicting smooth cloth geometries from SMPLicit, which are then refined by an image-guided displacement network that gracefully fits the body recovering high-frequency details and wrinkles. LayerNet achieves competitive accuracy in the task of 3D reconstruction against current `garment-agnostic' state of the art for images of people in up-right positions and controlled environments, and consistently surpasses these methods on challenging body poses and uncontrolled settings. Furthermore, the semantically rich outcome of our approach is suitable for performing Virtual Try-on tasks directly on 3D, a task which, so far, has only been addressed in the 2D domain., This work is supported in part by the Spanish government with the project MoHuCo PID2020-120049RBI00, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual Humans) and German Federal Ministry of Education and Research (BMBF): Tubingen AI ¨Center, FKZ: 01IS18039A. Gerard Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The project was made possible by funding from the Carl Zeiss Foundation. We thank NVIDIA for donating GPUs, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Single-view 3d body and cloth reconstruction under complex poses

  • Ugrinovic Kehdy, Nicolas
  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Moreno-Noguer, Francesc
Recent advances in 3D human shape reconstruction from single images have shown impressive results, leveraging on deep networks that model the so-called implicit function to learn the occupancy status of arbitrarily dense 3D points in space. However, while current algorithms based on this paradigm, like PiFuHD (Saito et al., 2020), are able to estimate accurate geometry of the human shape and clothes, they require high-resolution input images and are not able to capture complex body poses. Most training and evaluation is performed on 1k-resolution images of humans standing in front of the camera under neutral body poses. In this paper, we leverage publicly available data to extend existing implicit function-based models to deal with images of humans that can have arbitrary poses and self-occluded limbs. We argue that the representation power of the implicit function is not sufficient to simultaneously model details of the geometry and of the body pose. We, therefore, propose a coarse- to-fine approach in which we first learn an implicit function that maps the input image to a 3D body shape with a low level of detail, but which correctly fits the underlying human pose, despite its complexity. We then learn a displacement map, conditioned on the smoothed surface and on the input image, which encodes the high-frequency details of the clothes and body. In the experimental section, we show that this coarse-to-fine strategy represents a very good trade-off between shape detail and pose correctness, comparing favorably to the most recent state-of-the-art approaches. Our code will be made publicly available., This work is supported by the Spanish government with the projects MoHuCo PID2020-120049RB-I00 and María de Maeztu Seal of Excellence MDM-2016- 0656., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

MultiPhys: multi-person physics-aware 3D motion estimation

  • Ugrinovic Kehdy, Nicolas
  • Pan, Boxiao
  • Pavlakos, Georgios
  • Paschalidou, Despoina
  • Shen, Bokui
  • Sánchez Riera, Jordi|||0000-0002-4803-5742
  • Moreno-Noguer, Francesc
  • Guibas, Leonidas
We introduce MultiPhys, a method designed for recovering multi-person motion from monocular videos. Our focus lies in capturing coherent spatial placement between pairs of individuals across varying degrees of engagement. MultiPhys, being physically aware, exhibits robustness to jittering and occlusions, and effectively eliminates penetration issues between the two individuals. We devise a pipeline in which the motion estimated by a kinematic-based method is fed into a physics simulator in an autoregressive manner. We introduce distinct components that enable our model to harness the simulator's properties without compromising the accuracy of the kinematic estimates. This results in final motion estimates that are both kinematically coherent and physically compliant. Extensive evaluations on three challenging datasets characterized by substantial inter-person interaction show that our method significantly reduces errors associated with penetration and foot skating, while performing competitively with the state-of-the-art on motion accuracy and smoothness.Results and code can be found in our project page., Despoina Paschalidou is supported by the Swiss National Science Foundation under grant number P500PT 206946. Leonidas Guibas is supported by a Vannevar Bush Faculty Fellowship. This work is partially supported by projects SMARTGAZEII CPP2021-008760 and MoHuCo PID2020-120049RB-I00., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Learned vertex descent: a new direction for 3D human model fitting

  • Corona Puyane, Enric
  • Pons-Moll, Gerard
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Moreno-Noguer, Francesc
We propose a novel optimization-based paradigm for 3D human model fitting on images and scans. In contrast to existing approaches that directly regress the parameters of a low-dimensional statistical body model (e.g. SMPL) from input images, we train an ensemble of per vertex neural fields network. The network predicts, in a distributed manner, the vertex descent direction towards the ground truth, based on neural features extracted at the current vertex projection. At inference, we employ this network, dubbed LVD, within a gradient-descent optimization pipeline until its convergence, which typically occurs in a fraction of a second even when initializing all vertices into a single point. An exhaustive evaluation demonstrates that our approach is able to capture the underlying body of clothed people with very different body shapes, achieving a significant improvement compared to state-of-the-art. LVD is also applicable to 3D model fitting of humans and hands, for which we show a significant improvement to the SOTA with a much simpler and faster method., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Detail-aware uncalibrated photometric stereo

  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.”, Photometric stereo is the problem of jointly inferring the 3D reconstruction, reflectance, lighting and specularities of an object from a set of visual signals. Recently, some variational, uncalibrated, unsupervised and unified formulations have provided robust solutions to the problem while reducing the prior knowledge about the shape geometry or the lighting conditions. Unfortunately, these approaches cannot still produce solutions with an ample variety of details in the shape. That is mainly due to the non-convex and non-linear nature of the problem which requires the best initialization as possible. In this context, we propose a fully interpretable formulation that combines a physically-aware image formation model under perspective projection with a minimal detail-aware initialization and that it can handle general lighting. As a result, our formulation can consider multiple scenarios composed of unknown complex geometries and lighting patterns. Experimental results on challenging synthetic and real datasets show the effectiveness of our approach to capture more fine details, outperforming state-of-the-art techniques in terms of 3D reconstruction., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Robust multimodal and multi-object tracking for autonomous driving applications

  • Pérez Quintana, Marc
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work, we present a method for Multi-Object Tracking (MOT) that uses unsynchronized multimodal detections from a configurable set of sensors such as cameras, radars and lidars. All the information is processed from the sensors with modality-specific detectors and then combined in the MOT module that incorporates a Kalman filter and tracklet management logic. To be able to deploy our system in real-world driving applications, we handle localization errors, misclassifications and partial bounding-box detections from the object detector in the MOT module. We show promising results and compare them with respect to competing approaches in two challenging real-world scenarios, including a traffic jam chauffeur as well as a traffic monitoring application on highways., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033 and by the Government of Catalonia under 2020 DI 105., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Footbots: a transformer-based architecture for motion prediction in soccer

  • Capellera Font, Guillem
  • Ferraz Colomina, Luis
  • Rubio Sola, Jose Antonio|||0000-0003-1625-1472
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Moreno-Noguer, Francesc
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Motion prediction in soccer involves capturing complex dynamics from player and ball interactions. We present FootBots, an encoder-decoder transformer-based architecture addressing motion prediction and conditioned motion prediction through equivariance properties. FootBots captures temporal and social dynamics using set attention blocks and multi-attention block decoder. Our evaluation utilizes two datasets: a real soccer dataset and a tailored synthetic one. Insights from the synthetic dataset highlight the effectiveness of FootBots’ social attention mechanism and the significance of conditioned motion prediction. Empirical results on real soccer data demonstrate that FootBots outperforms baselines in motion prediction and excels in conditioned tasks, such as predicting the players based on the ball position, predicting the offensive (defensive) team based on the ball and the defensive (offensive) team, and predicting the ball position based on all players. Our evaluation connects quantitative and qualitative findings. https://youtu.be/9kaEkfzG3L8
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Uncalibrated, unified and unsupervised specular-aware photometric stereo

  • Estévez Graña, Pablo
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
In this paper we present a variational approach to simultaneously recover the 3D reconstruction, reflectance, lighting and specularities of an object, all of them, from a set of RGB images. The approach works in an uncalibrated, unified and unsupervised manner, without assuming any prior knowledge of the shape geometry or training data to constrain the solution and under general lighting. To this end, the approach exploits a physically-aware image formation model that in combination with a perspective projection one and under spherical harmonics lighting gives a fully interpretable algorithm. Integrability is implicitly ensured as the shape is coded by a depth map rather than normal vectors. As a consequence, a wide variety of illumination conditions and complex geometries can be acquired. Our claims have been experimentally validated on challenging synthetic and real datasets, obtaining a good trade-off between accuracy and computational budget in comparison with competing approaches., This work has been partially supported by the Spanish Ministry of Science and Innovation under project MoHuCo PID2020-120049RB. The authors wish to thank B. Haefner for fruitful discussions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Spline human motion recovery

  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Simultaneous camera pose, 4D reconstruction of an object and deformation clustering from incomplete 2D point tracks in a video is a challenging problem. To solve it, in this work we introduce a union of piecewise subspaces to encode the 4D shape, where two modalities based on B-splines and Catmull-Rom curves are considered. We demonstrate that formulating the problem in terms of B-spline or Catmull-Rom functions, allows for a better physical interpretation of the resulting priors while C1 and C2 continuities are automatically imposed without needing any additional constraint. An optimization framework is proposed to sort out the problem in a unified, accurate, unsupervised and efficient manner. We extensively validate our claims on a wide range of human motions, including articulated and continuous deformations as well as those cases with noisy and missing measurements where our approach provides competing joint solutions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Generating attribution maps with disentangled masked backpropagation

  • Ruiz Ovejero, Adrià
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Moreno-Noguer, Francesc
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Attribution map visualization has arisen as one of the most effective techniques to understand the underlying inference process of Convolutional Neural Networks. In this task, the goal is to compute an score for each image pixel related to its contribution to the network output. In this paper, we introduce Disentangled Masked Backpropagation (DMBP), a novel gradient-based method that leverages on the piecewise linear nature of ReLU networks to decompose the model function into different linear mappings. This decomposition aims to disentangle the attribution maps into positive, negative and nuisance factors by learning a set of variables masking the contribution of each filter during back-propagation. A thorough evaluation over standard architectures (ResNet50 and VGG16) and benchmark datasets (PASCAL VOC and ImageNet) demonstrates that DMBP generates more visually interpretable attribution maps than previous approaches. Additionally, we quantitatively show that the maps produced by our method are more consistent with the true contribution of each pixel to the final network output., This work has been partially funded by the Spanish government with the project MoHuCo PID2020-120049RB-I00. Adria Ruiz acknowledges financial support from MICINN (Spain) through the program Juan de la Cierva., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

PoseFix: correcting 3D human poses with natural language

  • Delmas, Ginger
  • Weinzaepfel, Philippe
  • Moreno-Noguer, Francesc
  • Rogez, Grégory
Automatically producing instructions to modify one’s posture could open the door to endless applications, such as personalized coaching and in-home physical therapy. Tackling the reverse problem (i.e., refining a 3D pose based on some natural language feedback) could help for assisted 3D character animation or robot teaching, for instance. Although a few recent works explore the connections between natural language and 3D human pose, none focus on describing 3D body pose differences. In this paper, we tackle the problem of correcting 3D human poses with natural language. To this end, we introduce the PoseFix dataset, which consists of several thousand paired 3D poses and their corresponding text feedback, that describe how the source pose needs to be modified to obtain the target pose. We demonstrate the potential of this dataset on two tasks: (1) text-based pose editing, that aims at generating corrected 3D body poses given a query pose and a text modifier; and (2) correctional text generation, where instructions are generated based on the differences between two body poses., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

NeRFLight: Fast and Light Neural Radiance Fields using a shared feature grid

  • Rivas-Manzaneque, Fernando
  • Sierra Acosta, Jorge|||0000-0002-3740-2893
  • Peñate Sánchez, Adrián
  • Moreno-Noguer, Francesc
  • Ribeiro Seijas, Angela
While original Neural Radiance Fields (NeRF) have shown impressive results in modeling the appearance of a scene with compact MLP architectures, they are not able to achieve real-time rendering. This has been recently addressed by either baking the outputs of NeRF into a data structure or arranging trainable parameters in an explicit feature grid. These strategies, however, significantly increase the memory footprint of the model which prevents their deployment on bandwidth-constrained applications. In this paper, we extend the grid-based approach to achieve real-time view synthesis at more than 150 FPS using a lightweight model. Our main contribution is a novel architecture in which the density field of NeRF-based representations is split into N regions and the density is modeled using N different decoders which reuse the same feature grid. This results in a smaller grid where each feature is located in more than one spatial position, forcing them to learn a compact representation that is valid for different parts of the scene. We further reduce the size of the final model by disposing of the features symmetrically on each region, which favors feature pruning after training while also allowing smooth gradient transitions between neighboring voxels. An exhaustive evaluation demonstrates that our method achieves real-time performance and quality metrics on a pair with state-of-the-art with an improvement of more than 2× in the FPS/MB ratio., This work is partially supported by the European Union under the project FlexiGroBots (DOI: 10.3030/101017111) and by the Spanish government under projects MoHuCo (PID2020-120049RB-I00) and SmartWEED-DARWEEM (PID2020-113229RB-C43). Adrian Penate-Sanchez is supported by a Beatriz Galindo grant., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis

  • Ordoñez Apraez, Daniel Felipe
  • Martín Muñoz, Mario|||0000-0002-4125-6630
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Moreno-Noguer, Francesc
We present a comprehensive study on discrete morphological symmetries of dynamical systems, which are commonly observed in biological and artificial locomoting systems, such as legged, swimming, and flying animals/robots/virtual characters. These symmetries arise from the presence of one or more planes/axis of symmetry in the system’s morphology, resulting in harmonious duplication and distribution of body parts. Significantly, we characterize how morphological symmetries extend to symmetries in the system’s dynamics, optimal control policies, and in all proprioceptive and exteroceptive measurements related to the system’s dynamics evolution. In the context of data-driven methods, symmetry represents an inductive bias that justifies the use of data augmentation or symmetric function approximators. To tackle this, we present a theoretical and practical framework for identifying the system’s morphological symmetry group G and characterizing the symmetries in proprioceptive and exteroceptive data measurements. We then exploit these symmetries using data augmentation and G-equivariant neural networks. Our experiments on both synthetic and real-world applications provide empirical evidence of the advantageous outcomes resulting from the exploitation of these symmetries, including improved sample efficiency, enhanced generalization, and reduction of trainable parameters., This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net Chistera project IPALM PCI2019-103386., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Morphological symmetries in robot learning

  • Ordoñez Apraez, Daniel Felipe
  • Martín Muñoz, Mario|||0000-0002-4125-6630
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Moreno-Noguer, Francesc
This work studies the impact of morphological symmetries in learning applications in robotics. Morphological symmetries are a predominant feature in both biological and robotic systems, arising from the presence of planes/axis of symmetry in the system's morphology. This results in harmonious duplication and distribution of body parts (e.g., humans' sagittal/left-right symmetry). Morphological symmetries become a significant learning prior as they extend to symmetries in the system's dynamics, optimal control policies, and in all proprioceptive and exteroceptive measurements, related to the system's dynamics evolution \cite{ordonez2023discrete}. Exploiting these symmetries in learning applications offers several advantageous outcomes, such as the use of data augmentation to mitigate the cost and challenges of data collection, or the use of equivariant/invariant function approximation models (e.g., neural networks) to improve sample efficiency and generalization, while reducing the number of trainable parameters. Lastly, we provide a video presentation and an open access repository reproducing our experiments and allowing for rapid prototyping in robot learning applications exploiting morphological symmetries., This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net Chistera project IPALM PCI2019-103386., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Generating attribution maps with disentangled masked backpropagation

  • Ruiz Ovejero, Adrià
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Attribution map visualization has arisen as one of the most effective techniques to understand the underlying inference process of Convolutional Neural Networks. In this task, the goal is to compute an score for each image pixel related to its contribution to the network output. In this paper, we introduce Disentangled Masked Backpropagation (DMBP), a novel gradient-based method that leverages on the piecewise linear nature of ReLU networks to decompose the model function into different linear mappings. This decomposition aims to disentangle the attribution maps into positive, negative and nuisance factors by learning a set of variables masking the contribution of each filter during back-propagation. A thorough evaluation over standard architectures (ResNet50 and VGG16) and benchmark datasets (PASCAL VOC and ImageNet) demonstrates that DMBP generates more visually interpretable attribution maps than previous approaches. Additionally, we quantitatively show that the maps produced by our method are more consistent with the true contribution of each pixel to the final network output., This work has been partially funded by the Spanish government with the project MoHuCo PID2020-120049RB-I00. Adria Ruiz acknowledges financial support from MICINN (Spain) through the program Juan de la Cierva., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Single-view 3d body and cloth reconstruction under complex poses

  • Ugrinovic Kehdy, Nicolas
  • Pumarola Peris, Albert
  • Sanfeliu Cortés, Alberto
  • Moreno-Noguer, Francesc
Recent advances in 3D human shape reconstruction from single images have shown impressive results, leveraging on deep networks that model the so-called implicit function to learn the occupancy status of arbitrarily dense 3D points in space. However, while current algorithms based on this paradigm, like PiFuHD (Saito et al., 2020), are able to estimate accurate geometry of the human shape and clothes, they require high-resolution input images and are not able to capture complex body poses. Most training and evaluation is performed on 1k-resolution images of humans standing in front of the camera under neutral body poses. In this paper, we leverage publicly available data to extend existing implicit function-based models to deal with images of humans that can have arbitrary poses and self-occluded limbs. We argue that the representation power of the implicit function is not sufficient to simultaneously model details of the geometry and of the body pose. We, therefore, propose a coarse- to-fine approach in which we first learn an implicit function that maps the input image to a 3D body shape with a low level of detail, but which correctly fits the underlying human pose, despite its complexity. We then learn a displacement map, conditioned on the smoothed surface and on the input image, which encodes the high-frequency details of the clothes and body. In the experimental section, we show that this coarse-to-fine strategy represents a very good trade-off between shape detail and pose correctness, comparing favorably to the most recent state-of-the-art approaches. Our code will be made publicly available., This work is supported by the Spanish government with the projects MoHuCo PID2020-120049RB-I00 and María de Maeztu Seal of Excellence MDM-2016- 0656., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis

  • Ordoñez Apraez, Daniel Felipe
  • Martín Muñoz, Mario
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
We present a comprehensive study on discrete morphological symmetries of dynamical systems, which are commonly observed in biological and artificial locomoting systems, such as legged, swimming, and flying animals/robots/virtual characters. These symmetries arise from the presence of one or more planes/axis of symmetry in the system’s morphology, resulting in harmonious duplication and distribution of body parts. Significantly, we characterize how morphological symmetries extend to symmetries in the system’s dynamics, optimal control policies, and in all proprioceptive and exteroceptive measurements related to the system’s dynamics evolution. In the context of data-driven methods, symmetry represents an inductive bias that justifies the use of data augmentation or symmetric function approximators. To tackle this, we present a theoretical and practical framework for identifying the system’s morphological symmetry group G and characterizing the symmetries in proprioceptive and exteroceptive data measurements. We then exploit these symmetries using data augmentation and G-equivariant neural networks. Our experiments on both synthetic and real-world applications provide empirical evidence of the advantageous outcomes resulting from the exploitation of these symmetries, including improved sample efficiency, enhanced generalization, and reduction of trainable parameters., This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net Chistera project IPALM PCI2019-103386., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

COmparative study of feature localization methods for endoscopy image matching

  • Urdapilleta Martín, Ana
  • Agudo Martínez, Antonio
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.”, The purpose of this work is to determine which is the best general method of feature localization for image matching in endoscopy images. To accomplish this, we conduct an exhaustive analysis of ten well-known feature detectors, descriptors, and learned algorithms, such as SIFT, FAST, SURF, ORB, BRIEF, BRISK, FREAK, HARRIS, DFM, and LoFTR. The analysis is performed across six challenging medical datasets, including cardiorespiratory endoscopy, human laparoscopy, bronchoscopy, gastroscopy, rabbit laparoscopy, and pig laparoscopy. This framework is highly diverse, containing a variety of textures, camera motions, tissue deformations, and visual barriers. To determine which technique is the best on average, we perform a qualitative analysis of the inliers and a quantitative analysis using the number of keypoints, number of matches, number of inliers, computational cost, sparsity of the inliers, recall and 1-precision. To complete the study, we considered the sequential and template modes, as they are highly used in computer vision. Furthermore, we examine how those features may be exploited in the reconstruction of 3D shapes from visual cues., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

InstantGeoAvatar: effective geometry and appearance modeling of animatable avatars from monocular video

  • Budría Fernández, Álvaro Francesc
  • Lopez Rodriguez, Adrián
  • Lorente Corominas, Òscar
  • Moreno-Noguer, Francesc
Lecture Notes in Computer Science (LNCS, volume 15477) (c) 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd., We present InstantGeoAvatar, a method for efficient and effective learning from monocular video of detailed 3D geometry and appearance of animatable implicit human avatars. Our key observation is that the optimization of a hash grid encoding to represent a signed distance function (SDF) of the human subject is fraught with instabilities and bad local minima. We thus propose a principled geometry-aware SDF regularization scheme that seamlessly fits into the volume rendering pipeline and adds negligible computational overhead. Our regularization scheme significantly outperforms previous approaches for training SDFs on hash grids. We obtain competitive results in geometry reconstruction and novel view synthesis in as little as five minutes of training time, a significant reduction from the several hours required by previous work. InstantGeoAvatar represents a significant leap forward towards achieving interactive reconstruction of virtual avatars., Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Spline human motion recovery

  • Agudo Martínez, Antonio
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Simultaneous camera pose, 4D reconstruction of an object and deformation clustering from incomplete 2D point tracks in a video is a challenging problem. To solve it, in this work we introduce a union of piecewise subspaces to encode the 4D shape, where two modalities based on B-splines and Catmull-Rom curves are considered. We demonstrate that formulating the problem in terms of B-spline or Catmull-Rom functions, allows for a better physical interpretation of the resulting priors while C1 and C2 continuities are automatically imposed without needing any additional constraint. An optimization framework is proposed to sort out the problem in a unified, accurate, unsupervised and efficient manner. We extensively validate our claims on a wide range of human motions, including articulated and continuous deformations as well as those cases with noisy and missing measurements where our approach provides competing joint solutions., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Morphological symmetries in robot learning

  • Ordoñez Apraez, Daniel Felipe
  • Martín Muñoz, Mario
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
This work studies the impact of morphological symmetries in learning applications in robotics. Morphological symmetries are a predominant feature in both biological and robotic systems, arising from the presence of planes/axis of symmetry in the system's morphology. This results in harmonious duplication and distribution of body parts (e.g., humans' sagittal/left-right symmetry). Morphological symmetries become a significant learning prior as they extend to symmetries in the system's dynamics, optimal control policies, and in all proprioceptive and exteroceptive measurements, related to the system's dynamics evolution \cite{ordonez2023discrete}. Exploiting these symmetries in learning applications offers several advantageous outcomes, such as the use of data augmentation to mitigate the cost and challenges of data collection, or the use of equivariant/invariant function approximation models (e.g., neural networks) to improve sample efficiency and generalization, while reducing the number of trainable parameters. Lastly, we provide a video presentation and an open access repository reproducing our experiments and allowing for rapid prototyping in robot learning applications exploiting morphological symmetries., This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net Chistera project IPALM PCI2019-103386., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

NeRFLight: Fast and Light Neural Radiance Fields using a shared feature grid

  • Rivas-Manzaneque, Fernando
  • Sierra Acosta, Jorge
  • Peñate Sánchez, Adrián
  • Moreno-Noguer, Francesc
  • Ribeiro Seijas, Angela
While original Neural Radiance Fields (NeRF) have shown impressive results in modeling the appearance of a scene with compact MLP architectures, they are not able to achieve real-time rendering. This has been recently addressed by either baking the outputs of NeRF into a data structure or arranging trainable parameters in an explicit feature grid. These strategies, however, significantly increase the memory footprint of the model which prevents their deployment on bandwidth-constrained applications. In this paper, we extend the grid-based approach to achieve real-time view synthesis at more than 150 FPS using a lightweight model. Our main contribution is a novel architecture in which the density field of NeRF-based representations is split into N regions and the density is modeled using N different decoders which reuse the same feature grid. This results in a smaller grid where each feature is located in more than one spatial position, forcing them to learn a compact representation that is valid for different parts of the scene. We further reduce the size of the final model by disposing of the features symmetrically on each region, which favors feature pruning after training while also allowing smooth gradient transitions between neighboring voxels. An exhaustive evaluation demonstrates that our method achieves real-time performance and quality metrics on a pair with state-of-the-art with an improvement of more than 2× in the FPS/MB ratio., This work is partially supported by the European Union under the project FlexiGroBots (DOI: 10.3030/101017111) and by the Spanish government under projects MoHuCo (PID2020-120049RB-I00) and SmartWEED-DARWEEM (PID2020-113229RB-C43). Adrian Penate-Sanchez is supported by a Beatriz Galindo grant., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Uncalibrated, unified and unsupervised specular-aware photometric stereo

  • Estévez Graña, Pablo
  • Agudo Martínez, Antonio
In this paper we present a variational approach to simultaneously recover the 3D reconstruction, reflectance, lighting and specularities of an object, all of them, from a set of RGB images. The approach works in an uncalibrated, unified and unsupervised manner, without assuming any prior knowledge of the shape geometry or training data to constrain the solution and under general lighting. To this end, the approach exploits a physically-aware image formation model that in combination with a perspective projection one and under spherical harmonics lighting gives a fully interpretable algorithm. Integrability is implicitly ensured as the shape is coded by a depth map rather than normal vectors. As a consequence, a wide variety of illumination conditions and complex geometries can be acquired. Our claims have been experimentally validated on challenging synthetic and real datasets, obtaining a good trade-off between accuracy and computational budget in comparison with competing approaches., This work has been partially supported by the Spanish Ministry of Science and Innovation under project MoHuCo PID2020-120049RB. The authors wish to thank B. Haefner for fruitful discussions., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

LayerNet: high-resolution semantic 3D reconstruction of clothed people

  • Corona Puyane, Enric
  • Alenyà Ribas, Guillem
  • Pons-Moll, Gerard
  • Moreno-Noguer, Francesc
In this paper we introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry; and LayerNet, a deep network that given a single image of a person simultaneously performs detailed 3D reconstruction of body and clothes. In contrast to existing learning-based approaches that require training specific models for each type of garment, SMPLicit can represent in a unified manner different garment topologies (eg from sleeveless tops to hoodies and open jackets), while controlling other properties like garment size or tightness/looseness. LayerNet follows a coarse-to-fine multi-stage strategy by first predicting smooth cloth geometries from SMPLicit, which are then refined by an image-guided displacement network that gracefully fits the body recovering high-frequency details and wrinkles. LayerNet achieves competitive accuracy in the task of 3D reconstruction against current `garment-agnostic' state of the art for images of people in up-right positions and controlled environments, and consistently surpasses these methods on challenging body poses and uncontrolled settings. Furthermore, the semantically rich outcome of our approach is suitable for performing Virtual Try-on tasks directly on 3D, a task which, so far, has only been addressed in the 2D domain., This work is supported in part by the Spanish government with the project MoHuCo PID2020-120049RBI00, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual Humans) and German Federal Ministry of Education and Research (BMBF): Tubingen AI ¨Center, FKZ: 01IS18039A. Gerard Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The project was made possible by funding from the Carl Zeiss Foundation. We thank NVIDIA for donating GPUs, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

PoseFix: correcting 3D human poses with natural language

  • Delmas, Ginger
  • Weinzaepfel, Philippe
  • Moreno-Noguer, Francesc
  • Rogez, Grégory
Automatically producing instructions to modify one’s posture could open the door to endless applications, such as personalized coaching and in-home physical therapy. Tackling the reverse problem (i.e., refining a 3D pose based on some natural language feedback) could help for assisted 3D character animation or robot teaching, for instance. Although a few recent works explore the connections between natural language and 3D human pose, none focus on describing 3D body pose differences. In this paper, we tackle the problem of correcting 3D human poses with natural language. To this end, we introduce the PoseFix dataset, which consists of several thousand paired 3D poses and their corresponding text feedback, that describe how the source pose needs to be modified to obtain the target pose. We demonstrate the potential of this dataset on two tasks: (1) text-based pose editing, that aims at generating corrected 3D body poses given a query pose and a text modifier; and (2) correctional text generation, where instructions are generated based on the differences between two body poses., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Photovoltaic power forecasting using sky images and sun motion

  • Berresheim, Arne
  • Agudo Martínez, Antonio
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Solar energy adoption is moving at a rapid pace. The variability in solar energy production causes grid stability issues and hinders mass adoption. To solve these issues, more accurate photovoltaic power forecasting systems are needed. In intra-hour forecasting, the most challenging issue is high output fluctuations due to cloud motion, which can occlude the sun. Using ground-based sky images, this paper proposes two convolutional neural network models for intra-hour nowcasting and forecasting that incorporate physical information on sun motion and cloud coverage by means of the sun area mean pixel intensity. Particularly, our models exploit that information instead of relying exclusively on photovoltaic output history data as it is standard in state of the art. Taking advantage of sun position and cloud coverage information, we were able to reduce the overall root mean squared error for the nowcasting task, making the model more accurate especially during cloudy days, and obtaining competitive results on forecasting. Moreover, our models are more robust against artifacts such as occlusion and noisy observations., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

MultiPhys: multi-person physics-aware 3D motion estimation

  • Ugrinovic Kehdy, Nicolas
  • Pan, Boxiao
  • Pavlakos, Georgios
  • Paschalidou, Despoina
  • Shen, Bokui
  • Sánchez Riera, Jordi
  • Moreno-Noguer, Francesc
  • Guibas, Leonidas
We introduce MultiPhys, a method designed for recovering multi-person motion from monocular videos. Our focus lies in capturing coherent spatial placement between pairs of individuals across varying degrees of engagement. MultiPhys, being physically aware, exhibits robustness to jittering and occlusions, and effectively eliminates penetration issues between the two individuals. We devise a pipeline in which the motion estimated by a kinematic-based method is fed into a physics simulator in an autoregressive manner. We introduce distinct components that enable our model to harness the simulator's properties without compromising the accuracy of the kinematic estimates. This results in final motion estimates that are both kinematically coherent and physically compliant. Extensive evaluations on three challenging datasets characterized by substantial inter-person interaction show that our method significantly reduces errors associated with penetration and foot skating, while performing competitively with the state-of-the-art on motion accuracy and smoothness.Results and code can be found in our project page., Despoina Paschalidou is supported by the Swiss National Science Foundation under grant number P500PT 206946. Leonidas Guibas is supported by a Vannevar Bush Faculty Fellowship. This work is partially supported by projects SMARTGAZEII CPP2021-008760 and MoHuCo PID2020-120049RB-I00., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Learned vertex descent: a new direction for 3D human model fitting

  • Corona Puyane, Enric
  • Pons-Moll, Gerard
  • Alenyà Ribas, Guillem
  • Moreno-Noguer, Francesc
We propose a novel optimization-based paradigm for 3D human model fitting on images and scans. In contrast to existing approaches that directly regress the parameters of a low-dimensional statistical body model (e.g. SMPL) from input images, we train an ensemble of per vertex neural fields network. The network predicts, in a distributed manner, the vertex descent direction towards the ground truth, based on neural features extracted at the current vertex projection. At inference, we employ this network, dubbed LVD, within a gradient-descent optimization pipeline until its convergence, which typically occurs in a fraction of a second even when initializing all vertices into a single point. An exhaustive evaluation demonstrates that our approach is able to capture the underlying body of clothed people with very different body shapes, achieving a significant improvement compared to state-of-the-art. LVD is also applicable to 3D model fitting of humans and hands, for which we show a significant improvement to the SOTA with a much simpler and faster method., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Robust multimodal and multi-object tracking for autonomous driving applications

  • Pérez Quintana, Marc
  • Agudo Martínez, Antonio
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work, we present a method for Multi-Object Tracking (MOT) that uses unsynchronized multimodal detections from a configurable set of sensors such as cameras, radars and lidars. All the information is processed from the sensors with modality-specific detectors and then combined in the MOT module that incorporates a Kalman filter and tracklet management logic. To be able to deploy our system in real-world driving applications, we handle localization errors, misclassifications and partial bounding-box detections from the object detector in the MOT module. We show promising results and compare them with respect to competing approaches in two challenging real-world scenarios, including a traffic jam chauffeur as well as a traffic monitoring application on highways., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033 and by the Government of Catalonia under 2020 DI 105., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Detail-aware uncalibrated photometric stereo

  • Agudo Martínez, Antonio
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.”, Photometric stereo is the problem of jointly inferring the 3D reconstruction, reflectance, lighting and specularities of an object from a set of visual signals. Recently, some variational, uncalibrated, unsupervised and unified formulations have provided robust solutions to the problem while reducing the prior knowledge about the shape geometry or the lighting conditions. Unfortunately, these approaches cannot still produce solutions with an ample variety of details in the shape. That is mainly due to the non-convex and non-linear nature of the problem which requires the best initialization as possible. In this context, we propose a fully interpretable formulation that combines a physically-aware image formation model under perspective projection with a minimal detail-aware initialization and that it can handle general lighting. As a result, our formulation can consider multiple scenarios composed of unknown complex geometries and lighting patterns. Experimental results on challenging synthetic and real datasets show the effectiveness of our approach to capture more fine details, outperforming state-of-the-art techniques in terms of 3D reconstruction., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Uncalibrated and unsupervised photometric stereo with piecewise regularizer

  • Casanova Martin, Alejandro
  • Agudo Martínez, Antonio
Photometric stereo is a technique for recovering a rigid object's 3D shape, reflectance properties, lighting conditions, and specular highlights from multiple images captured under varying lighting conditions. Variational, uncalibrated, and unsupervised formulations have recently provided detailed and robust solutions to the problem, reducing the need for prior knowledge about shape geometry or lighting conditions. However, uncalibrated methods, especially when applied to real-world data, may be susceptible to noise and depth errors near boundaries or self-occlusions, stemming from missing or noisy data, surface orientation ambiguity, and calibration issues. In this paper, we introduce a novel piecewise depth regularizer to mitigate these errors, enhancing stability and improving robustness against initialization errors. We demonstrate the effectiveness of our approach through evaluations on both synthetic and real-world data, showcasing its promise in enhancing the accuracy and reliability of photometric stereo for practical applications., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Back to MLP: a simple baseline for human motion prediction

  • Guo, Wen
  • Du, Yuming
  • Shen, Xi
  • Lepetit, Vincent
  • Alameda-Pineda, Xavier
  • Moreno-Noguer, Francesc
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. State-of-the-art approaches provide good results, however, they rely on deep learning architectures of arbitrary complexity, such as Recurrent Neural Networks (RNN), Transformers or Graph Convolutional Networks (GCN), typically requiring multiple training stages and more than 2 million parameters. In this paper, we show that, after combining with a series of standard practices, such as applying Discrete Cosine Transform (DCT), predicting residual displacement of joints and optimizing velocity as an auxiliary loss, a light-weight network based on multi-layer perceptrons (MLPs) with only 0.14 million parameters can surpass the state-of-the-art performance. An exhaustive evaluation on the Human3.6M, AMASS, and 3DPW datasets shows that our method, named siMLpe, consistently outperforms all other approaches. We hope that our simple method could serve as a strong baseline for the community and allow re-thinking of the human motion prediction problem. The code is publicly available at https://github.com/dulucas/siMLPe., This research was supported by ANR-3IA MIAI (ANR19-P3IA-0003), ANR-JCJC ML3RI (ANR-19-CE33-0008-01), H2020 SPRING (funded by EC under GA #871245), by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and by an Amazon Research Award. This project has received funding from the CHISTERA IPALM project., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Footbots: a transformer-based architecture for motion prediction in soccer

  • Capellera Font, Guillem
  • Ferraz Colomina, Luis
  • Rubio Sola, Jose Antonio
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Motion prediction in soccer involves capturing complex dynamics from player and ball interactions. We present FootBots, an encoder-decoder transformer-based architecture addressing motion prediction and conditioned motion prediction through equivariance properties. FootBots captures temporal and social dynamics using set attention blocks and multi-attention block decoder. Our evaluation utilizes two datasets: a real soccer dataset and a tailored synthetic one. Insights from the synthetic dataset highlight the effectiveness of FootBots’ social attention mechanism and the significance of conditioned motion prediction. Empirical results on real soccer data demonstrate that FootBots outperforms baselines in motion prediction and excels in conditioned tasks, such as predicting the players based on the ball position, predicting the offensive (defensive) team based on the ball and the defensive (offensive) team, and predicting the ball position based on all players. Our evaluation connects quantitative and qualitative findings. https://youtu.be/9kaEkfzG3L8, Postprint (author's final draft)