Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 21
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Conditional-Flow NeRF: Accurate 3D modelling with reliable uncertainty quantification

  • Shen, Jianxiong
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
  • Ruiz Ovejero, Adrià
Trabajo presentado en la 17th European Conference on Computer Vision, celebrada en Tel Aviv (Israel), los días 23 y 27 de octubre de 2022, A critical limitation of current methods based on Neural Radiance Fields (NeRF) is that they are unable to quantify the uncertainty associated with the learned appearance and geometry of the scene. This information is paramount in real applications such as medical diagnosis or autonomous driving where, to reduce potentially catastrophic failures, the confidence on the model outputs must be included into the decision-making process. In this context, we introduce Conditional-Flow NeRF (CF-NeRF), a novel probabilistic framework to incorporate uncertainty quantification into NeRF-based approaches. For this purpose, our method learns a distribution over all possible radiance fields modelling which is used to quantify the uncertainty associated with the modelled scene. In contrast to previous approaches enforcing strong constraints over the radiance field distribution, CF-NeRF learns it in a flexible and fully data-driven manner by coupling Latent Variable Modelling and Conditional Normalizing Flows. This strategy allows to obtain reliable uncertainty estimation while preserving model expressivity. Compared to previous state-of-the-art methods proposed for uncertainty quantification in NeRF, our experiments show that the proposed method achieves significantly lower prediction errors and more reliable uncertainty values for synthetic novel view and depth-map estimation., This work is supported partly by the Chinese Scholarship Council (CSC) under grant (201906120031), by the Spanish government under project MoHuCo PID2020-120049RB-I00 and the Chistera project IPALM PCI2019-103386. We also thank Nvidia for hardware donation under the GPU grant program.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

An adaptable approach to learn realistic legged locomotion without examples

  • Ordoñez Apraez, Daniel Felipe
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
  • Martín, Mario
Trabajo presentado en el International Conference on Robotics and Automation (ICRA), celebrado en Philadelphia, PA (Estados Unidos), del 23 al 27 de mayo de 2022, Learning controllers that reproduce legged locomotion in nature has been a long-time goal in robotics and computer graphics. While yielding promising results, recent approaches are not yet flexible enough to be applicable to legged systems of different morphologies. This is partly because they often rely on precise motion capture references or elaborate learning environments that ensure the naturality of the emergent locomotion gaits but prevent generalization. This work proposes a generic approach for ensuring realism in locomotion by guiding the learning process with the spring-loaded inverted pendulum model as a reference. Leveraging on the exploration capacities of Reinforcement Learning (RL), we learn a control policy that fills in the information gap between the template model and full-body dynamics required to maintain stable and periodic locomotion. The proposed approach can be applied to robots of different sizes and morphologies and adapted to any RL technique and control architecture. We present experimental results showing that even in a model-free setup and with a simple reactive control architecture, the learned policies can generate realistic and energy-efficient locomotion gaits for a bipedal and a quadrupedal robot. And most importantly, this is achieved without using motion capture, strong constraints in the dynamics or kinematics of the robot, nor prescribing limb coordination. We provide supplemental videos for qualitative analysis of the naturality of the learned gaits., This work’s experiments were run at the Barcelona Supercomputing Center in collaboration with the HPAI group. This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net Chistera project IPALM PCI2019-103386.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Safari from visual signals: Recovering volumetric 3D shapes

  • Agudo Martínez, Antonio
Trabajo presentado en la International Conference on Acoustics, Speech and Signal Processing (ICASSP), celebrada en Singapur, del 23 al 27 de mayo de 2022, In this paper we propose a convex approach for recovering a detailed 3D volumetric geometry of several objects from visual signals. To this end, we first present a minimal detailed surface energy that is optimized together with a volume constraint by considering some geometrical priors, and without requiring neither additional training data nor templates in order to constrain the solution. Our problem can be efficiently solved by means of a gradient descent, and be applied for single RGB images or monocular videos even with very small rigid motions. Temporal-aware solutions and driven by point correspondences are incorporated without assuming any 2D tracking data over time. Thanks to this formulation, both rigid and non-rigid objects can be considered. We have extensively validated our approach in a wide variety of scenarios in the wild, recovering challenging type of shapes that have not been previously attempted without assuming any training data., This work has been partially supported by the Spanish Ministry of Science and Innovation under project MoHuCo PID2020-120049RB.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Enhancing egocentric 3D pose estimation with third person views

  • Dhamanaskar, Ameya
  • Dimiccoli, Mariella
  • Corona, Enric
  • Pumarola, Albert
  • Moreno-Noguer, Francesc
We propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. The main technical contribution consists of leveraging high-level features linking first- and third-views in a joint embedding space. To learn such embedding space we introduce First2Third-Pose, a new paired synchronized dataset of nearly 2000 videos depicting human activities captured from both first- and third-view perspectives. We explicitly consider spatial- and motion-domain features, combined using a semi-Siamese architecture trained in a self-supervised fashion. Experimental results demonstrate that the joint multi-view embedded space learned with our dataset is useful to extract discriminatory features from arbitrary single-view egocentric videos, with no need to perform any sort of domain adaptation or knowledge of camera parameters. An extensive evaluation demonstrates that we achieve significant improvement in egocentric 3D body pose estimation performance on two unconstrained datasets, over three supervised state-of-the-art approaches. The collected dataset and pre-trained model are available for research purposes., This work has been partially supported by projects PID2020-120049RB-I00 and PID2019-110977GA-I00 funded by MCIN/ AEI /10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”, as well as by grant RYC-2017-22563 funded by MCIN/ AEI /10.13039/501100011033 and by “ESF Investing in your future”, and network RED2018-102511-T funded by MCIN/ AEI., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

NeRFLight: Fast and Light Neural Radiance Fields using a Shared Feature Grid

  • Rivas-Manzaneque, Fernando
  • Sierra-Acosta, Jorge
  • Penate-Sanchez, Adrian
  • Moreno-Noguer, Francesc
  • Ribeiro Seijas, Ángela
Trabajo presentado en la Conference on Computer Vision and Pattern Recognition (CVPR), celebrada en Vancuver (Canadá), del 17 al 24 de junio de 2023, While original Neural Radiance Fields (NeRF) have shown impressive results in modeling the appearance of a scene with compact MLP architectures, they are not able to achieve real-time rendering. This has been recently addressed by either baking the outputs of NeRF into a data structure or arranging trainable parameters in an explicit feature grid. These strategies, however, significantly increase the memory footprint of the model which prevents their deployment on bandwidth-constrained applications. In this paper, we extend the grid-based approach to achieve real-time view synthesis at more than 150 FPS using a lightweight model. Our main contribution is a novel architecture in which the density field of NeRF-based representations is split into N regions and the density is modeled using N different decoders which reuse the same feature grid. This results in a smaller grid where each feature is located in more than one spatial position, forcing them to learn a compact representation that is valid for different parts of the scene. We further reduce the size of the final model by disposing of the features symmetrically on each region, which favors feature pruning after training while also allowing smooth gradient transitions between neighboring voxels. An exhaustive evaluation demonstrates that our method achieves real-time performance and quality metrics on a pair with state-of-the-art with an improvement of more than 2× in the FPS/MB ratio., This work is partially supported by the European Union under the project FlexiGroBots (DOI: 10.3030/101017111) and by the Spanish government under projects MoHuCo (PID2020-120049RB-I00) and SmartWEED-DARWEEM (PID2020-113229RB-C43). Adrian Penate-Sanchez is supported by a Beatriz Galindo grant. The authors would like to thank Marcos Barrios for his support in the graphic design of the figures and videos included in the paper, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Detail-Aware Uncalibrated Photometric Stereo

  • Agudo Martínez, Antonio
Trabajo presentado en la International Conference on Acoustics, Speech and Signal Processing (ICASSP), celebrada en Rodas (Grecia), del 4 al 10 de junio de 2023, Photometric stereo is the problem of jointly inferring the 3D reconstruction, reflectance, lighting and specularities of an object from a set of visual signals. Recently, some variational, uncalibrated, unsupervised and unified formulations have provided robust solutions to the problem while reducing the prior knowledge about the shape geometry or the lighting conditions. Unfortunately, these approaches cannot still produce solutions with an ample variety of details in the shape. That is mainly due to the non-convex and non-linear nature of the problem which requires the best initialization as possible. In this context, we propose a fully interpretable formulation that combines a physically-aware image formation model under perspective projection with a minimal detail-aware initialization and that it can handle general lighting. As a result, our formulation can consider multiple scenarios composed of unknown complex geometries and lighting patterns. Experimental results on challenging synthetic and real datasets show the effectiveness of our approach to capture more fine details, outperforming state-of-the-art techniques in terms of 3D reconstruction., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Robust Wind Turbine Blade Segmentation from RGB Images in the Wild

  • Pérez-Gonzalo, Raül
  • Espersen, Andreas
  • Agudo Martínez, Antonio
Trabajo presentado en la IEEE International Conference on Image Processing (ICIP), celebradad en Kuala Lumpur (Malasia), del 8 al 11 de octubre de 2023, With the relentless growth of the wind industry, there is an imperious need to design automatic data-driven solutions for wind turbine maintenance. As structural health monitoring mainly relies on visual inspections, the first stage in any automatic solution is to identify the blade region on the image. Thus, we propose a novel segmentation algorithm that strengthens the U-Net results by a tailored loss, which pools the focal loss with a contiguity regularization term. To attain top performing results, a set of additional steps are proposed to ensure a reliable, generic, robust and efficient algorithm. First, we leverage our prior knowledge on the images by filling the holes enclosed by temporarily-classified blade pixels and by the image boundaries. Subsequently, the mislead classified pixels are successfully amended by training an on-the-fly random forest. Our algorithm demonstrates its effectiveness reaching a non-trivial 97.39% of accuracy., This work has been supported by the Innovation Fund Denmark under 2021 ID1044-0044A and by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Back to MLP: A Simple Baseline for Human Motion Prediction

  • Guo, Wen
  • Du, Yuming
  • Shen, Xi
  • Lepetit, Vincent
  • Alameda-Pineda, Xavier
  • Moreno-Noguer, Francesc
Trabajo presentado en la IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), celebrada en Waikoloa, HI (Estados Unidos), del 2 al 7 de enero de 2023, This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. State-of-the-art approaches provide good results, however, they rely on deep learning architectures of arbitrary complexity, such as Recurrent Neural Networks (RNN), Transformers or Graph Convolutional Networks (GCN), typically requiring multiple training stages and more than 2 million parameters. In this paper, we show that, after combining with a series of standard practices, such as applying Discrete Cosine Transform (DCT), predicting residual displacement of joints and optimizing velocity as an auxiliary loss, a light-weight network based on multi-layer perceptrons (MLPs) with only 0.14 million parameters can surpass the state-of-the-art performance. An exhaustive evaluation on the Human3.6M, AMASS, and 3DPW datasets shows that our method, named siMLpe, consistently outperforms all other approaches. We hope that our simple method could serve as a strong baseline for the community and allow re-thinking of the human motion prediction problem. The code is publicly available at https://github.com/dulucas/siMLPe., This research was supported by ANR-3IA MIAI (ANR- 19-P3IA-0003), ANR-JCJC ML3RI (ANR-19-CE33-0008- 01), H2020 SPRING (funded by EC under GA #871245), by the Spanish government with the project MoHuCo PID2020-120049RB-I00 and by an Amazon Research Award. This project has received funding from the CHISTERA IPALM project., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

LayerNet: High-Resolution Semantic 3D Reconstruction of Clothed People

  • Corona, Enric
  • Alenyà, Guillem
  • Pons-Moll, Gerard
  • Moreno-Noguer, Francesc
In this article, we introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry; and LayerNet, a deep network that given a single image of a person simultaneously performs detailed 3D reconstruction of body and clothes. In contrast to existing learning-based approaches that require training specific models for each type of garment, SMPLicit can represent in a unified manner different garment topologies (e.g. from sleeveless tops to hoodies and open jackets), while controlling other properties like garment size or tightness/looseness. LayerNet follows a coarse-to-fine multi-stage strategy by first predicting smooth cloth geometries from SMPLicit, which are then refined by an image-guided displacement network that gracefully fits the body recovering high-frequency details and wrinkles. LayerNet achieves competitive accuracy in the task of 3D reconstruction against current 'garment-agnostic' state of the art for images of people in up-right positions and controlled environments, and consistently surpasses these methods on challenging body poses and uncontrolled settings. Furthermore, the semantically rich outcome of our approach is suitable for performing Virtual Try-on tasks directly on 3D, a task which, so far, has only been addressed in the 2D domain., This work is supported in part by the Spanish government with the project MoHuCo PID2020-120049RBI00, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual Humans) and German Federal Ministry of Education and Research (BMBF): Tubingen AI ¨ Center, FKZ: 01IS18039A. Gerard Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The project was made possible by funding from the Carl Zeiss Foundation. We thank NVIDIA for donating GPUs., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Generalized Nested Latent Variable Models For Lossy Coding Applied To Wind Turbine Scenarios

  • Pérez-Gonzalo, Raül
  • Espersen, Andreas
  • Agudo Martínez, Antonio
Rate-distortion optimization through neural networks has accomplished competitive results in compression efficiency and image quality. This learning-based approach seeks to minimize the compromise between compression rate and reconstructed image quality by automatically extracting and retaining crucial information, while discarding less critical details. A successful technique consists in introducing a deep hyperprior that operates within a 2-level nested latent variable model, enhancing compression by capturing complex data dependencies. This paper extends this concept by designing a generalized L-level nested generative model with a Markov chain structure. We demonstrate as L increases that a trainable prior is detrimental and explore a common dimensionality along the distinct latent variables to boost compression performance. As this structured framework can represent autoregressive coders, we outperform the hyperprior model and achieve state-of-the-art performance while reducing substantially the computational cost. Our experimental evaluation is performed on wind turbine scenarios to study its application on visual inspections., This work has been supported by the Innovation Fund Denmark under 2021 ID1044-0044A and by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

InstantAvatar: Efficient 3D Head Reconstruction via Surface Rendering

  • Canela, Antonio
  • Caselles, Pol
  • Malik, Ibrar
  • Ramon, Eduard
  • Martínez García, Jaime
  • Sánchez-Riera, Jordi
  • Triginer, Gil
  • Moreno-Noguer, Francesc
Recent advances in full-head reconstruction have been obtained by optimizing a neural field through differentiable surface or volume rendering to represent a single scene. While these techniques achieve an unprecedented accuracy, they take several minutes, or even hours, due to the expensive optimization process required. In this work, we introduce InstantAvatar, a method that recovers full-head avatars from few images (down to just one) in a few seconds on commodity hardware. In order to speed up the reconstruction process, we propose a system that combines, for the first time, a voxel-grid neural field representation with a surface renderer. Notably, a naive combination of these two techniques leads to unstable optimizations that do not converge to valid solutions. In order to overcome this limitation, we present a novel statistical model that learns a prior distribution over 3D head signed distance functions using a voxel-grid based architecture. The use of this prior model, in combination with other design choices, results into a system that achieves 3D head reconstructions with comparable accuracy as the state-of-the-art with a 100 × speed-up., This work has been partially supported by: MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033; DeeLight PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033; and the industrial doctorate 2021 DI 00102 funded by the Government of Catalonia., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

InstantGeoAvatar: Effective Geometry and Appearance Modeling of Animatable Avatars from Monocular Video

  • Budria, Alvaro
  • Lopez-Rodriguez, Adrian
  • Lorente, Òscar
  • Moreno-Noguer, Francesc
We present InstantGeoAvatar, a method for efficient and effective learning from monocular video of detailed 3D geometry and appearance of animatable implicit human avatars. Our key observation is that the optimization of a hash grid encoding to represent a signed distance function (SDF) of the human subject is fraught with instabilities and bad local minima. We thus propose a principled geometry-aware SDF regularization scheme that seamlessly fits into the volume rendering pipeline and adds negligible computational overhead. Our regularization scheme significantly outperforms previous approaches for training SDFs on hash grids. We obtain competitive results in geometry reconstruction and novel view synthesis in as little as five minutes of training time, a significant reduction from the several hours required by previous work. InstantGeoAvatar represents a significant leap forward towards achieving interactive reconstruction of virtual avatars., This work has been supported by the project MOHUCO PID2020-120049RB-I00 funded by MCIU/AEI/10.13039/501100011033, and by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

MultiPhys: Multi-Person Physics-Aware 3D Motion Estimation

  • Ugrinovic, Nicolas
  • Pan, Boxiao
  • Pavlakos, Georgios
  • Paschalidou, Despoina
  • Shen, Bokui
  • Sánchez-Riera, Jordi
  • Moreno-Noguer, Francesc
  • Guibas, Leonidas
We introduce MultiPhys, a method designed for recovering multi-person motion from monocular videos. Our focus lies in capturing coherent spatial placement between pairs of individuals across varying degrees of engagement. MultiPhys, being physically aware, exhibits robustness to jittering and occlusions, and effectively eliminates penetration issues between the two individuals. We devise a pipeline in which the motion estimated by a kinematic-based method is fed into a physics simulator in an autoregressive manner. We introduce distinct components that enable our model to harness the simulator's properties without compromising the accuracy of the kinematic estimates. This results in final motion estimates that are both kinematically coherent and physically compliant. Extensive evaluations on three challenging datasets characterized by substantial inter-person interaction show that our method significantly reduces errors associated with penetration and foot skating, while performing competitively with the state-of-the-art on motion accuracy and smoothness. Results and code can be found in our project page., Despoina Paschalidou is supported by the Swiss National Science Foundation under grant number P500PT 206946. Leonidas Guibas is supported by a Vannevar Bush Faculty Fellowship. This work is partially supported by projects SMARTGAZEII CPP2021-008760 and MoHuCo PID2020-120049RB-I00., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

No Bells, Just Whistles: Sports Field Registration by Leveraging Geometric Properties

  • Gutiérrez-Pérez, Marc
  • Agudo Martínez, Antonio
Broadcast sports field registration is traditionally addressed as a homography estimation task, mapping the visible image area to a planar field model, predominantly focusing on the main camera shot. Addressing the shortcomings of previous approaches, we propose a novel calibration pipeline enabling camera calibration using a 3D soccer field model and extending the process to assess the multiple-view nature of broadcast videos. Our approach begins with a keypoint generation pipeline derived from SoccerNet dataset annotations, leveraging the geometric properties of the court. Subsequently, we execute classical camera calibration through DLT algorithm in a minimalist fashion, without further refinement. Through extensive experimentation on real-world soccer broadcast datasets such as SoccerNet-Calibration, WorldCup 2014 and TS-WorldCup, our method demonstrates superior performance in both multiple- and single-view 3D camera calibration while maintaining competitive results in homography estimation compared to state-of-the-art techniques., This work has been supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Photovoltaic Power Forecasting Using Sky Images and Sun Motion

  • Berresheim, Arne
  • Agudo Martínez, Antonio
Solar energy adoption is moving at a rapid pace. The variability in solar energy production causes grid stability issues and hinders mass adoption. To solve these issues, more accurate photovoltaic power forecasting systems are needed. In intra-hour forecasting, the most challenging issue is high output fluctuations due to cloud motion, which can occlude the sun. Using ground-based sky images, this paper proposes two convolutional neural network models for intra-hour nowcasting and forecasting that incorporate physical information on sun motion and cloud coverage by means of the sun area mean pixel intensity. Particularly, our models exploit that information instead of relying exclusively on photovoltaic output history data as it is standard in state of the art. Taking advantage of sun position and cloud coverage information, we were able to reduce the overall root mean squared error for the nowcasting task, making the model more accurate especially during cloudy days, and obtaining competitive results on forecasting. Moreover, our models are more robust against artifacts such as occlusion and noisy observations., This work has been supported by the project MoHuCo PID2020- 120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

PIRO: Permutation-invariant relational network for multi-person 3D pose estimation

  • Ugrinovic, Nicolás
  • Ruiz Ovejero, Adrià
  • Agudo Martínez, Antonio
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Recovering multi-person 3D poses from a single RGB image is an ill-conditioned problem due to the inherent 2D-3D depth ambiguity, inter-person occlusions, and body truncation. To tackle these issues, recent works have shown promising results by simultaneously reasoning for different individuals. However, in most cases this is done by only considering pairwise inter-person interactions or between pairs of body parts, thus hindering a holistic scene representation able to capture long-range interactions. Some approaches that jointly process all people in the scene require defining one of the individuals as a reference and a pre-defined person ordering or limiting the number of individuals thus being sensitive to these choice. In this paper, we overcome both these limitations, and we propose an approach for multi-person 3D pose estimation that captures longrange interactions independently of the input order. We build a residual-like permutation-invariant network that successfully refines potentially corrupted initial 3D poses estimated by off-the-shelf detectors. The residual function is learned via a Set Attention (Lee et al., 2019) mechanism. Despite of our model being relatively straightforward, a thorough evaluation demonstrates that our approach is able to boost the performance of the initially estimated 3D poses by large margins, achieving state-of-the-art results on two standardized benchmarks, This work is supported by the project MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

PoseEmbroider: Towards a 3D, Visual, Semantic-Aware Human Pose Representation

  • Delmas, Ginger
  • Weinzaepfel, Philippe
  • Moreno-Noguer, Francesc
  • Rogez, Grégory
Aligning multiple modalities in a latent space, such as images and texts, has shown to produce powerful semantic visual representations, fueling tasks like image captioning, text-to-image generation, or image grounding. In the context of human-centric vision, albeit CLIP-like representations encode most standard human poses relatively well (such as standing or sitting), they lack sufficient acuteness to discern detailed or uncommon ones. Actually, while 3D human poses have been often associated with images (e.g. to perform pose estimation or pose-conditioned image generation), or more recently with text (e.g. for text-to-pose generation), they have seldom been paired with both. In this work, we combine 3D poses, person’s pictures and textual pose descriptions to produce an enhanced 3D-, visual- and semantic-aware human pose representation. We introduce a new transformer-based model, trained in a retrieval fashion, which can take as input any combination of the aforementioned modalities. When composing modalities, it outperforms a standard multi-modal alignment retrieval model, making it possible to sort out partial information (e.g. image with the lower body occluded). We showcase the potential of such an embroidered pose representation for (1) SMPL regression from image with optional text cue; and (2) on the task of fine-grained instruction generation, which consists in generating a text that describes how to move from one 3D pose to another (as a fitness coach). Unlike prior works, our model can take any kind of input (image and/or pose) without retraining., This work is supported by the Spanish government with the project MoHuCo PID2020-120049RB-I00, and by NAVER LABS Europe under technology transfer contract ‘Text4Pose’., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Uncalibrated and Unsupervised Photometric Stereo with Piecewise Regularizer

  • Casanova, Alejandro
  • Agudo Martínez, Antonio
Photometric stereo is a technique for recovering a rigid object's 3D shape, reflectance properties, lighting conditions, and specular highlights from multiple images captured under varying lighting conditions. Variational, uncalibrated, and unsupervised formulations have recently provided detailed and robust solutions to the problem, reducing the need for prior knowledge about shape geometry or lighting conditions. However, uncalibrated methods, especially when applied to real-world data, may be susceptible to noise and depth errors near boundaries or self-occlusions, stemming from missing or noisy data, surface orientation ambiguity, and calibration issues. In this paper, we introduce a novel piecewise depth regularizer to mitigate these errors, enhancing stability and improving robustness against initialization errors. We demonstrate the effectiveness of our approach through evaluations on both synthetic and real-world data, showcasing its promise in enhancing the accuracy and reliability of photometric stereo for practical applications., This work has been supported by the project MoHuCo PID2020- 120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033, Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

LayerNet: high-resolution semantic 3D reconstruction of clothed people

  • Corona Puyane, Enric
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Pons-Moll, Gerard
  • Moreno-Noguer, Francesc
In this paper we introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry; and LayerNet, a deep network that given a single image of a person simultaneously performs detailed 3D reconstruction of body and clothes. In contrast to existing learning-based approaches that require training specific models for each type of garment, SMPLicit can represent in a unified manner different garment topologies (eg from sleeveless tops to hoodies and open jackets), while controlling other properties like garment size or tightness/looseness. LayerNet follows a coarse-to-fine multi-stage strategy by first predicting smooth cloth geometries from SMPLicit, which are then refined by an image-guided displacement network that gracefully fits the body recovering high-frequency details and wrinkles. LayerNet achieves competitive accuracy in the task of 3D reconstruction against current `garment-agnostic' state of the art for images of people in up-right positions and controlled environments, and consistently surpasses these methods on challenging body poses and uncontrolled settings. Furthermore, the semantically rich outcome of our approach is suitable for performing Virtual Try-on tasks directly on 3D, a task which, so far, has only been addressed in the 2D domain., This work is supported in part by the Spanish government with the project MoHuCo PID2020-120049RBI00, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual Humans) and German Federal Ministry of Education and Research (BMBF): Tubingen AI ¨Center, FKZ: 01IS18039A. Gerard Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The project was made possible by funding from the Carl Zeiss Foundation. We thank NVIDIA for donating GPUs, Peer Reviewed
Zaguán. Repositorio Digital de la Universidad de Zaragoza
Publicación Artículo científico (JournalArticle)

Matching and Recovering 3D People From Multiple Views

  • Perez-Yus, A.
  • Agudo, A.
This paper introduces an approach to simultaneously match and recover 3D people from multiple calibrated cameras. To this end, we present an affinity measure between 2D detections across different views that enforces an uncertainty geometric consistency. This similarity is then exploited by a novel multi-view matching algorithm to cluster the detections, being robust against partial observations as well as bad detections and without assuming any prior about the number of people in the scene. After that, the multi-view correspondences are used in order to efficiently infer the 3D pose of each body by means of a 3D pictorial structure model in combination with physico-geometric constraints. Our algorithm is thoroughly evaluated on challenging scenarios where several human bodies are performing different activities which involve complex motions, producing large occlusions in some views and noisy observations. We outperform state-of-the-art results in terms of matching and 3D reconstruction.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

LayerNet: high-resolution semantic 3D reconstruction of clothed people

  • Corona Puyane, Enric
  • Alenyà Ribas, Guillem
  • Pons-Moll, Gerard
  • Moreno-Noguer, Francesc
In this paper we introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry; and LayerNet, a deep network that given a single image of a person simultaneously performs detailed 3D reconstruction of body and clothes. In contrast to existing learning-based approaches that require training specific models for each type of garment, SMPLicit can represent in a unified manner different garment topologies (eg from sleeveless tops to hoodies and open jackets), while controlling other properties like garment size or tightness/looseness. LayerNet follows a coarse-to-fine multi-stage strategy by first predicting smooth cloth geometries from SMPLicit, which are then refined by an image-guided displacement network that gracefully fits the body recovering high-frequency details and wrinkles. LayerNet achieves competitive accuracy in the task of 3D reconstruction against current `garment-agnostic' state of the art for images of people in up-right positions and controlled environments, and consistently surpasses these methods on challenging body poses and uncontrolled settings. Furthermore, the semantically rich outcome of our approach is suitable for performing Virtual Try-on tasks directly on 3D, a task which, so far, has only been addressed in the 2D domain., This work is supported in part by the Spanish government with the project MoHuCo PID2020-120049RBI00, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual Humans) and German Federal Ministry of Education and Research (BMBF): Tubingen AI ¨Center, FKZ: 01IS18039A. Gerard Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The project was made possible by funding from the Carl Zeiss Foundation. We thank NVIDIA for donating GPUs, Peer Reviewed, Postprint (author's final draft)