Pasar al contenido principal

GENERACION DE AVATARES REALISTAS EN ENTORNOS NO CONTROLADOS

PID2023-151184OB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2023
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS (CSIC)

Resultados relacionados

Resultados totales (Incluyendo duplicados): 25
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

InstantGeoAvatar: Effective Geometry and Appearance Modeling of Animatable Avatars from Monocular Video

  • Budria, Alvaro
  • Lopez-Rodriguez, Adrian
  • Lorente, Òscar
  • Moreno-Noguer, Francesc
We present InstantGeoAvatar, a method for efficient and effective learning from monocular video of detailed 3D geometry and appearance of animatable implicit human avatars. Our key observation is that the optimization of a hash grid encoding to represent a signed distance function (SDF) of the human subject is fraught with instabilities and bad local minima. We thus propose a principled geometry-aware SDF regularization scheme that seamlessly fits into the volume rendering pipeline and adds negligible computational overhead. Our regularization scheme significantly outperforms previous approaches for training SDFs on hash grids. We obtain competitive results in geometry reconstruction and novel view synthesis in as little as five minutes of training time, a significant reduction from the several hours required by previous work. InstantGeoAvatar represents a significant leap forward towards achieving interactive reconstruction of virtual avatars., This work has been supported by the project MOHUCO PID2020-120049RB-I00 funded by MCIU/AEI/10.13039/501100011033, and by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-agent Sports

  • Capellera, Guillem
  • Ferraz, Luis
  • Rubio Romano, Antonio
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Understanding trajectories in multi-agent scenarios requires addressing various tasks, including predicting future movements, imputing missing observations, inferring the status of unseen agents, and classifying different global states. Traditional data-driven approaches often handle these tasks separately with specialized models. We introduce TranSPORTmer, a unified transformer-based framework capable of addressing all these tasks, showcasing its application to the intricate dynamics of multi-agent sports scenarios like soccer and basketball. Using Set Attention Blocks, TranSPORTmer effectively captures temporal dynamics and social interactions in an equivariant manner. The model’s tasks are guided by an input mask that conceals missing or yet-to-be-predicted observations. Additionally, we introduce a CLS extra agent to classify states along soccer trajectories, including passes, possessions, uncontrolled states, and out-of-play intervals, contributing to an enhancement in modeling trajectories. Evaluations on soccer and basketball datasets show that TranSPORTmer outperforms state-of-the-art task-specific models in player forecasting, player forecasting-imputation, ball inference, and ball imputation. https://youtu.be/8VtSRm8oGoE., This work has been supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2023 DI 00058., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Morphological Symmetries in Robotics

  • Ordoñez Apraez, Daniel Felipe
  • Turrisi, Giulio
  • Kostic, Vladimir
  • Martín, Mario
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
  • Pontil, Massimiliano
  • Semini, Claudio
  • Mastalli, Carlos
18 pages, 11 figures, We present a comprehensive framework for studying and leveraging morphological symmetries in robotic systems. These are intrinsic properties of the robot's morphology, frequently observed in animal biology and robotics, which stem from the replication of kinematic structures and the symmetrical distribution of mass. We illustrate how these symmetries extend to the robot's state space and both proprioceptive and exteroceptive sensor measurements, resulting in the equivariance of the robot's equations of motion and optimal control policies. Thus, we recognize morphological symmetries as a relevant and previously unexplored physics-informed geometric prior, with significant implications for both data-driven and analytical methods used in modeling, control, estimation and design in robotics. For data-driven methods, we demonstrate that morphological symmetries can enhance the sample efficiency and generalization of machine learning models through data augmentation, or by applying equivariant/invariant constraints on the model's architecture. In the context of analytical methods, we employ abstract harmonic analysis to decompose the robot's dynamics into a superposition of lower-dimensional, independent dynamics. We substantiate our claims with both synthetic and real-world experiments conducted on bipedal and quadrupedal robots. Lastly, we introduce the repository MorphoSymm to facilitate the practical use of the theory and applications outlined in this work., The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: GRAVATAR project PID2023-151184OB-I00 funded by the Spanish Ministry of Science, Innovation, and Universities MCIU, the State Research Agency AEI/10.13039/501100011033 and by the European Regional Development Fund ERDF; PNRR-MUR Project PE000013-CUP-J53C22003010006. Future Artificial Intelligence Research (FAIR), funded by the European Union, NextGenerationEU., Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

InstantGeoAvatar: effective geometry and appearance modeling of animatable avatars from monocular video

  • Budría Fernández, Álvaro Francesc
  • Lopez Rodriguez, Adrián
  • Lorente Corominas, Òscar
  • Moreno-Noguer, Francesc
Lecture Notes in Computer Science (LNCS, volume 15477) (c) 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd., We present InstantGeoAvatar, a method for efficient and effective learning from monocular video of detailed 3D geometry and appearance of animatable implicit human avatars. Our key observation is that the optimization of a hash grid encoding to represent a signed distance function (SDF) of the human subject is fraught with instabilities and bad local minima. We thus propose a principled geometry-aware SDF regularization scheme that seamlessly fits into the volume rendering pipeline and adds negligible computational overhead. Our regularization scheme significantly outperforms previous approaches for training SDFs on hash grids. We obtain competitive results in geometry reconstruction and novel view synthesis in as little as five minutes of training time, a significant reduction from the several hours required by previous work. InstantGeoAvatar represents a significant leap forward towards achieving interactive reconstruction of virtual avatars.
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Unified uncertainty-aware diffusion for multi-agent trajectory modeling

  • Capellera Font, Guillem
  • Rubio Romano, Antonio
  • Ferraz, Luis
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
Multi-agent trajectory modeling has primarily focused on forecasting future states, often overlooking broader tasks like trajectory completion, which are crucial for real-world applications such as correcting tracking data. Existing methods also generally predict agents’ states without offering any state-wise measure of uncertainty. Moreover, popular multi-modal sampling methods lack any error probability estimates for each generated scene under the same prior observations, making it difficult to rank the predictions during inference time. We introduce U2Diff, a unified diffusion model designed to handle trajectory completion while providing state-wise uncertainty estimates jointly. This uncertainty estimation is achieved by augmenting the simple denoising loss with the negative log-likelihood of the predicted noise and propagating latent space uncertainty to the real state space. Additionally, we incorporate a Rank Neural Network in post-processing to enable error probability estimation for each generated mode, demonstrating a strong correlation with the error relative to ground truth. Our method outperforms the state-of-the-art solutions in trajectory completion and forecasting across four challenging sports datasets (NBA, Basketball-U, Football-U, Soccer-U), highlighting the effectiveness of uncertainty and error probability estimation., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Implicit shape and appearance priors for few-shot full head reconstruction

  • Caselles Rico, Pol|||0009-0002-4943-0557
  • Ramon Maldonado, Eduard
  • García Giráldez, Jaime
  • Triginer, Gil
  • Moreno-Noguer, Francesc
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Recent advancements in learning techniques that employ coordinate-based neural representations have yielded remarkable results in multi-view 3D reconstruction tasks. However, these approaches often require a substantial number of input views (typically several tens) and computationally intensive optimization procedures to achieve their effectiveness. In this paper, we address these limitations specifically for the problem of few-shot full 3D head reconstruction. We accomplish this by incorporating a probabilistic shape and appearance prior into coordinate-based representations, enabling faster convergence and improved generalization when working with only a few input images (even as low as a single image). During testing, we leverage this prior to guiding the fitting process of a signed distance function using a differentiable renderer. By incorporating the statistical prior alongside parallelizable ray tracing and dynamic caching strategies, we achieve an efficient and accurate approach to few-shot full 3D head reconstruction. Moreover, we extend the H3DS dataset, which now comprises 60 high-resolution 3D full-head scans and their corresponding posed images and masks, which we use for evaluation purposes. By leveraging this dataset, we demonstrate the remarkable capabilities of our approach in achieving state-of-the-art results in geometry reconstruction while being an order of magnitude faster than previous approaches., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

GLVD: guided learned vertex descent

  • Caselles Rico, Pol|||0009-0002-4943-0557
  • Moreno-Noguer, Francesc
Existing 3D face modeling methods usually depend on 3D Morphable Models, which inherently constrain the representation capacity to fixed shape priors. Optimization-based approaches offer high-quality reconstructions but tend to be computationally expensive. In this work, we introduce GLVD, a hybrid method for 3D face reconstruction from few-shot images that extends Learned Vertex Descent (LVD) [11] by integrating per-vertex neural field optimization with global structural guidance from dynamically predicted 3D keypoints. By incorporating relative spatial encoding, GLVD iteratively refines mesh vertices without requiring dense 3D supervision. This enables expressive and adaptable geometry reconstruction while maintaining computational efficiency. GLVD achieves state-of-the-art performance in single-view settings and remains highly competitive in multi-view scenarios, all while substantially reducing inference time., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Temporally accurate events detection through ball possessor recognition in soccer

  • Peral Sánchez, Marc|||0000-0001-6521-6476
  • Capellera Font, Guillem
  • Rubio Sola, Jose Antonio|||0000-0003-1625-1472
  • Ferraz Colomina, Luis
  • Moreno-Noguer, Francesc
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
Recognizing specific actions in soccer games has become an increasingly important research topic. One key area focuses on accurately identifying when passes and receptions occur, as these are frequent actions in games and critical for analysts reviewing match strategies. However, most current methods do not pinpoint when these actions happen precisely enough and often fail to show which player is making the move. Our new method uses video footage to detect passes and receptions and identifies which player is involved in each action by following possession of the ball at each moment. We create video clips, or tubes, for every player on the field, determine who has the ball, and use this information to recognize when these key actions take place. Our results show that our system is better than the latest models in spotting passes and can identify most events with an accuracy down to 0.6 seconds., This work has been supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2020 DI 00106.
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Multi-modal soccer scene analysis with masked pre-training

  • Peral Sánchez, Marc|||0000-0001-6521-6476
  • Capellera Font, Guillem
  • Ferraz, Luis
  • Rubio Romano, Antonio
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work we propose a multi-modal architecture for analyzing soccer scenes from tactical camera footage, with a focus on three core tasks: ball trajectory inference, ball state classification, and ball possessor identification. To this end, our solution integrates three distinct input modalities (player trajectories, player types and image crops of individual players) into a unified framework that processes spatial and temporal dynamics using a cascade of sociotemporal transformer blocks. Unlike prior methods, which rely heavily on accurate ball tracking or handcrafted heuristics, our approach infers the ball trajectory without direct access to its past or future positions, and robustly identifies the ball state and ball possessor under noisy or occluded conditions from real top league matches. We also introduce CropDrop, a modality-specific masking pre-training strategy that prevents over-reliance on image features and encourages the model to rely on cross-modal patterns during pre-training. We show the effectiveness of our approach on a large-scale dataset providing substantial improvements over state-of-the-art baselines in all tasks. Our results highlight the benefits of combining structured and visual cues in a transformer-based architecture, and the importance of realistic masking strategies in multi-modal learning., This work has been partially sup- ported by the project GRAVATAR PID2023-151184OB- I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2020 DI 00106., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Capítulo o parte de un libro (BookPart)

TranSPORTmer: a holistic approach to trajectory understanding in multi-agent sports scenarios

  • Capellera Font, Guillem
  • Ferraz Colomina, Luis
  • Rubio Romano, Antonio
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Moreno-Noguer, Francesc
Understanding trajectories in multi-agent scenarios requires addressing diverse tasks, including predicting future movements, imputing missing observations, inferring the status of unseen agents, and classifying different states. Traditional data-driven approaches often handle these tasks independently, utilizing specific models tailored for individual purposes. In this paper, we introduce TranSPORTmer, a unified framework capable of simultaneously addressing all these tasks at once, showcasing its application to the intricate dynamics of multi-agent soccer and basketball scenarios. TranSPORTmer builds upon a Transformer architecture and employs Set Attention Blocks sequentially to model trajectories in an equivariant manner, effectively capturing temporal dynamics and social interactions among agents. The model’s task is guided by an input mask, concealing missing or yet-to-be-predicted observations. Additionally, we introduce a CLS extra agent to classify states along soccer trajectories, including passes, possessions, out-of-play intervals, and uncontrolled states, contributing to an enhancement in modeling trajectories. To validate TranSPORTmer’s efficacy, we conducted a thorough evaluation on a soccer and two basketball datasets. Across various metrics, we demonstrate that our general-purpose model consistently outperforms task-specific state-of-the-art architectures in player forecasting, unified player forecasting-imputation, ball inference, and ball imputation., This work has been supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2023 DI 00058
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Better together: uncalibrated photometric stereo with shading and specularities

  • Adrián Matos, Jesús Miguel
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This work introduces a novel variational approach to uncalibrated photometric stereo for robustly recovering 3D surface shape, reflectance, and lighting from images under unknown illumination. Unlike classical methods that require calibration, this framework formulates the task as an unsupervised joint optimization problem for both diffuse and specular reflectance. The proposed method incorporates a specialized depth regularizer that acts separately on interior regions and object boundaries, substantially improving depth and albedo estimation. Optimization is performed via a lagged block coordinate descent scheme, alternately updating surface depth, normals, lighting, and albedo components. The effectiveness of this approach is validated on both synthetic and real-world datasets, where it demonstrates strong quantitative improvements over baselines, especially near object boundaries and in regions with specular highlights. The results show that a physically-motivated variational framework can faithfully recover geometry and reflectance from uncalibrated multi-illumination images, advancing the state of the art., Work supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Recovering and classifying upper limb impairment trajectories after stroke

  • Méndez López, Martín
  • Fornés Bisquerra, Alicia
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Upper limb impairment is a loss of motor function after a stroke, leading to difficulties in performing daily tasks. With a low remission rate six months after stroke, monitoring during this critical period is essential. Telemonitoring with inertial sensors has become a common approach that requires the identification and recognition of specific movements. However, data quality in clinical databases is a challenge, making data augmentation necessary. In this work, we propose a novel method that can learn a trajectory subspace from partial 3D signals in an unsupervised manner. Our method is simple yet effective, producing novel human-feasible motions compatible with the estimated trajectory subspace. To this end, three approaches are introduced from global to local models that can capture a wide variety of human motions. Our method outperforms the results in the state of the art in both control and patient subjects., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Class prototypical loss for enhanced feature separation in 3D object detection

  • Pérez Quintana, Marc
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Dubbelman, Gijs
  • Jancura, Pavol
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We present a novel loss to increase the class separation of learned features for 3D object detection from lidar point clouds. To correctly classify objects, learned object-level feature distributions of each class need to be distinct. Therefore, we hypothesize that if we make the feature distributions of the classes more separated, then the overall performance of the object detector will improve. To this end, we calculate class prototypes as the mean and covariance of the feature vectors extracted from the annotated objects of each class. Then, we exploit these prototypes with a novel class prototypical loss, defined as the Mahalanobis distance from the feature vector of annotated objects to the corresponding class prototype. This auxiliary loss is then integrated with other object detection losses to improve the object-level feature separation between classes and the overall performance of the detector. We show results applying this loss to the NuScenes dataset where we get improvements of +3.85% and +1.76% mAP for 1 and 10 frames, respectively, compared to the baseline Centerpoint detector, while keeping the same inference computational cost., This work has been supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2020 DI 105
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

SoccerNet-v3D: leveraging sports broadcast replays for 3D scene understanding

  • Gutiérrez Pérez, Marc
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Sports video analysis is a key domain in computer vision, enabling detailed spatial understanding through multi-view correspondences. In this work, we introduce SoccerNetv-3D and ISSIA-3D, two enhanced and scalable datasets designed for 3D scene understanding in soccer broadcast analysis. These datasets extend SoccerNet-v3 and ISSIA by incorporating field-line-based camera calibration and multi-view synchronization, enabling 3D object localization through triangulation. We propose a monocular 3D ball localization task built upon the triangulation of ground-truth (2 D) ball annotations, along with several calibration and reprojection metrics to assess annotation quality on demand. Additionally, we present a single-image 3D ball localization method as a baseline, leveraging camera calibration and ball size priors to estimate the ball's position from a monocular viewpoint. To further refine 2D annotations, we introduce a bounding box optimization technique that ensures alignment with the 3D scene representation. Our proposed datasets establish new benchmarks for 3D soccer scene understanding, enhancing both spatial and temporal analysis in sports analytics. Finally, we provide code to facilitate access to our annotations and the generation pipelines for the datasets 11https://github.com/mguti97/SoccerNet-v3D., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

SoccerNet-v3D: leveraging sports broadcast replays for 3D scene understanding

  • Gutiérrez Pérez, Marc
  • Agudo Martínez, Antonio
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Sports video analysis is a key domain in computer vision, enabling detailed spatial understanding through multi-view correspondences. In this work, we introduce SoccerNetv-3D and ISSIA-3D, two enhanced and scalable datasets designed for 3D scene understanding in soccer broadcast analysis. These datasets extend SoccerNet-v3 and ISSIA by incorporating field-line-based camera calibration and multi-view synchronization, enabling 3D object localization through triangulation. We propose a monocular 3D ball localization task built upon the triangulation of ground-truth (2 D) ball annotations, along with several calibration and reprojection metrics to assess annotation quality on demand. Additionally, we present a single-image 3D ball localization method as a baseline, leveraging camera calibration and ball size priors to estimate the ball's position from a monocular viewpoint. To further refine 2D annotations, we introduce a bounding box optimization technique that ensures alignment with the 3D scene representation. Our proposed datasets establish new benchmarks for 3D soccer scene understanding, enhancing both spatial and temporal analysis in sports analytics. Finally, we provide code to facilitate access to our annotations and the generation pipelines for the datasets 11https://github.com/mguti97/SoccerNet-v3D., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Temporally accurate events detection through ball possessor recognition in soccer

  • Peral Sánchez, Marc
  • Capellera Font, Guillem
  • Rubio Sola, Jose Antonio
  • Ferraz Colomina, Luis
  • Moreno-Noguer, Francesc
  • Agudo Martínez, Antonio
Recognizing specific actions in soccer games has become an increasingly important research topic. One key area focuses on accurately identifying when passes and receptions occur, as these are frequent actions in games and critical for analysts reviewing match strategies. However, most current methods do not pinpoint when these actions happen precisely enough and often fail to show which player is making the move. Our new method uses video footage to detect passes and receptions and identifies which player is involved in each action by following possession of the ball at each moment. We create video clips, or tubes, for every player on the field, determine who has the ball, and use this information to recognize when these key actions take place. Our results show that our system is better than the latest models in spotting passes and can identify most events with an accuracy down to 0.6 seconds., This work has been supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2020 DI 00106., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Class prototypical loss for enhanced feature separation in 3D object detection

  • Pérez Quintana, Marc
  • Agudo Martínez, Antonio
  • Dubbelman, Gijs
  • Jancura, Pavol
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We present a novel loss to increase the class separation of learned features for 3D object detection from lidar point clouds. To correctly classify objects, learned object-level feature distributions of each class need to be distinct. Therefore, we hypothesize that if we make the feature distributions of the classes more separated, then the overall performance of the object detector will improve. To this end, we calculate class prototypes as the mean and covariance of the feature vectors extracted from the annotated objects of each class. Then, we exploit these prototypes with a novel class prototypical loss, defined as the Mahalanobis distance from the feature vector of annotated objects to the corresponding class prototype. This auxiliary loss is then integrated with other object detection losses to improve the object-level feature separation between classes and the overall performance of the detector. We show results applying this loss to the NuScenes dataset where we get improvements of +3.85% and +1.76% mAP for 1 and 10 frames, respectively, compared to the baseline Centerpoint detector, while keeping the same inference computational cost., This work has been supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2020 DI 105, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Multi-modal soccer scene analysis with masked pre-training

  • Peral Sánchez, Marc
  • Capellera Font, Guillem
  • Ferraz, Luis
  • Rubio Romano, Antonio
  • Agudo Martínez, Antonio
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work we propose a multi-modal architecture for analyzing soccer scenes from tactical camera footage, with a focus on three core tasks: ball trajectory inference, ball state classification, and ball possessor identification. To this end, our solution integrates three distinct input modalities (player trajectories, player types and image crops of individual players) into a unified framework that processes spatial and temporal dynamics using a cascade of sociotemporal transformer blocks. Unlike prior methods, which rely heavily on accurate ball tracking or handcrafted heuristics, our approach infers the ball trajectory without direct access to its past or future positions, and robustly identifies the ball state and ball possessor under noisy or occluded conditions from real top league matches. We also introduce CropDrop, a modality-specific masking pre-training strategy that prevents over-reliance on image features and encourages the model to rely on cross-modal patterns during pre-training. We show the effectiveness of our approach on a large-scale dataset providing substantial improvements over state-of-the-art baselines in all tasks. Our results highlight the benefits of combining structured and visual cues in a transformer-based architecture, and the importance of realistic masking strategies in multi-modal learning., This work has been partially sup- ported by the project GRAVATAR PID2023-151184OB- I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2020 DI 00106., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

InstantGeoAvatar: effective geometry and appearance modeling of animatable avatars from monocular video

  • Budría Fernández, Álvaro Francesc
  • Lopez Rodriguez, Adrián
  • Lorente Corominas, Òscar
  • Moreno-Noguer, Francesc
Lecture Notes in Computer Science (LNCS, volume 15477) (c) 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd., We present InstantGeoAvatar, a method for efficient and effective learning from monocular video of detailed 3D geometry and appearance of animatable implicit human avatars. Our key observation is that the optimization of a hash grid encoding to represent a signed distance function (SDF) of the human subject is fraught with instabilities and bad local minima. We thus propose a principled geometry-aware SDF regularization scheme that seamlessly fits into the volume rendering pipeline and adds negligible computational overhead. Our regularization scheme significantly outperforms previous approaches for training SDFs on hash grids. We obtain competitive results in geometry reconstruction and novel view synthesis in as little as five minutes of training time, a significant reduction from the several hours required by previous work. InstantGeoAvatar represents a significant leap forward towards achieving interactive reconstruction of virtual avatars., Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación

TranSPORTmer: a holistic approach to trajectory understanding in multi-agent sports scenarios

  • Capellera Font, Guillem
  • Ferraz Colomina, Luis
  • Rubio Romano, Antonio
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Understanding trajectories in multi-agent scenarios requires addressing diverse tasks, including predicting future movements, imputing missing observations, inferring the status of unseen agents, and classifying different states. Traditional data-driven approaches often handle these tasks independently, utilizing specific models tailored for individual purposes. In this paper, we introduce TranSPORTmer, a unified framework capable of simultaneously addressing all these tasks at once, showcasing its application to the intricate dynamics of multi-agent soccer and basketball scenarios. TranSPORTmer builds upon a Transformer architecture and employs Set Attention Blocks sequentially to model trajectories in an equivariant manner, effectively capturing temporal dynamics and social interactions among agents. The model’s task is guided by an input mask, concealing missing or yet-to-be-predicted observations. Additionally, we introduce a CLS extra agent to classify states along soccer trajectories, including passes, possessions, out-of-play intervals, and uncontrolled states, contributing to an enhancement in modeling trajectories. To validate TranSPORTmer’s efficacy, we conducted a thorough evaluation on a soccer and two basketball datasets. Across various metrics, we demonstrate that our general-purpose model consistently outperforms task-specific state-of-the-art architectures in player forecasting, unified player forecasting-imputation, ball inference, and ball imputation., This work has been supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE and by the Government of Catalonia under 2023 DI 00058, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Implicit shape and appearance priors for few-shot full head reconstruction

  • Caselles Rico, Pol
  • Ramon Maldonado, Eduard
  • García Giráldez, Jaime
  • Triginer, Gil
  • Moreno-Noguer, Francesc
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Recent advancements in learning techniques that employ coordinate-based neural representations have yielded remarkable results in multi-view 3D reconstruction tasks. However, these approaches often require a substantial number of input views (typically several tens) and computationally intensive optimization procedures to achieve their effectiveness. In this paper, we address these limitations specifically for the problem of few-shot full 3D head reconstruction. We accomplish this by incorporating a probabilistic shape and appearance prior into coordinate-based representations, enabling faster convergence and improved generalization when working with only a few input images (even as low as a single image). During testing, we leverage this prior to guiding the fitting process of a signed distance function using a differentiable renderer. By incorporating the statistical prior alongside parallelizable ray tracing and dynamic caching strategies, we achieve an efficient and accurate approach to few-shot full 3D head reconstruction. Moreover, we extend the H3DS dataset, which now comprises 60 high-resolution 3D full-head scans and their corresponding posed images and masks, which we use for evaluation purposes. By leveraging this dataset, we demonstrate the remarkable capabilities of our approach in achieving state-of-the-art results in geometry reconstruction while being an order of magnitude faster than previous approaches., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

GLVD: guided learned vertex descent

  • Caselles Rico, Pol
  • Moreno-Noguer, Francesc
Existing 3D face modeling methods usually depend on 3D Morphable Models, which inherently constrain the representation capacity to fixed shape priors. Optimization-based approaches offer high-quality reconstructions but tend to be computationally expensive. In this work, we introduce GLVD, a hybrid method for 3D face reconstruction from few-shot images that extends Learned Vertex Descent (LVD) [11] by integrating per-vertex neural field optimization with global structural guidance from dynamically predicted 3D keypoints. By incorporating relative spatial encoding, GLVD iteratively refines mesh vertices without requiring dense 3D supervision. This enables expressive and adaptable geometry reconstruction while maintaining computational efficiency. GLVD achieves state-of-the-art performance in single-view settings and remains highly competitive in multi-view scenarios, all while substantially reducing inference time., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Recovering and classifying upper limb impairment trajectories after stroke

  • Méndez López, Martín
  • Fornés Bisquerra, Alicia
  • Agudo Martínez, Antonio
© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Upper limb impairment is a loss of motor function after a stroke, leading to difficulties in performing daily tasks. With a low remission rate six months after stroke, monitoring during this critical period is essential. Telemonitoring with inertial sensors has become a common approach that requires the identification and recognition of specific movements. However, data quality in clinical databases is a challenge, making data augmentation necessary. In this work, we propose a novel method that can learn a trajectory subspace from partial 3D signals in an unsupervised manner. Our method is simple yet effective, producing novel human-feasible motions compatible with the estimated trajectory subspace. To this end, three approaches are introduced from global to local models that can capture a wide variety of human motions. Our method outperforms the results in the state of the art in both control and patient subjects., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Unified uncertainty-aware diffusion for multi-agent trajectory modeling

  • Capellera Font, Guillem
  • Rubio Romano, Antonio
  • Ferraz, Luis
  • Agudo Martínez, Antonio
Multi-agent trajectory modeling has primarily focused on forecasting future states, often overlooking broader tasks like trajectory completion, which are crucial for real-world applications such as correcting tracking data. Existing methods also generally predict agents’ states without offering any state-wise measure of uncertainty. Moreover, popular multi-modal sampling methods lack any error probability estimates for each generated scene under the same prior observations, making it difficult to rank the predictions during inference time. We introduce U2Diff, a unified diffusion model designed to handle trajectory completion while providing state-wise uncertainty estimates jointly. This uncertainty estimation is achieved by augmenting the simple denoising loss with the negative log-likelihood of the predicted noise and propagating latent space uncertainty to the real state space. Additionally, we incorporate a Rank Neural Network in post-processing to enable error probability estimation for each generated mode, demonstrating a strong correlation with the error relative to ground truth. Our method outperforms the state-of-the-art solutions in trajectory completion and forecasting across four challenging sports datasets (NBA, Basketball-U, Football-U, Soccer-U), highlighting the effectiveness of uncertainty and error probability estimation., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Better together: uncalibrated photometric stereo with shading and specularities

  • Adrián Matos, Jesús Miguel
  • Agudo Martínez, Antonio
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This work introduces a novel variational approach to uncalibrated photometric stereo for robustly recovering 3D surface shape, reflectance, and lighting from images under unknown illumination. Unlike classical methods that require calibration, this framework formulates the task as an unsupervised joint optimization problem for both diffuse and specular reflectance. The proposed method incorporates a specialized depth regularizer that acts separately on interior regions and object boundaries, substantially improving depth and albedo estimation. Optimization is performed via a lagged block coordinate descent scheme, alternately updating surface depth, normals, lighting, and albedo components. The effectiveness of this approach is validated on both synthetic and real-world datasets, where it demonstrates strong quantitative improvements over baselines, especially near object boundaries and in regions with specular highlights. The results show that a physically-motivated variational framework can faithfully recover geometry and reflectance from uncalibrated multi-illumination images, advancing the state of the art., Work supported by the project GRAVATAR PID2023-151184OB-I00 funded by MCIU/AEI/10.13039/501100011033 and by ERDF, UE., Peer Reviewed, Postprint (author's final draft)