Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 10
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Rooms: a WebGPU-Based VR platform for intuitive 3D sculpting and animation

  • Marquerie Cordoba, Juan Sebastian
  • Rodríguez, Alejandro
  • García, Pablo L.
  • Marques, Ricardo
  • Blat, Josep
Data de publicació electrònica: 21-07-2026, We propose Rooms, an open-source XR content creation platform designed to democratize 3D modeling and animation by leveraging the intuitiveness of VR motion controls and the high-performance capabilities of the modern WebGPU API. Built on top of wgpuEngine, Rooms provides real-time, cross-platform immersive sculpting and animation tools directly accessible through XR-enabled web browsers and desktop systems. The system implements a Signed Distance Field (SDF) sculpting pipeline that enables smooth Constructive Solid Geometry (CSG) operations for clay-like modeling. This paper details the architecture, the XR-first user interaction workflow, and the technical implementation, which includes a GPU-driven, sparse SDF baking pipeline. User studies confirm that the immersive, motion-based interaction model successfully lowers the entry barrier for non-expert users. Rooms effectively paves the way for future web-based XR creation environments by guiding the development of accessible and intuitive tools that harness the full power of modern GPU APIs on the web., This project has received partial financial support from the MAX-R project (Mixed Augmented and eXtended Reality media pipeline, H2020-Digital, Industry and Space, Grant Agreement No.101070072) and from the Wi-XR project (MCIN/FEDER, PID2021-123995NB-I00). Ricardo Marques acknowledges the support of CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. We thank the MAX-R partners, and especially UPF-WiNet and FABW for mutual support in user studies.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Bayesian DivideMix++ for enhanced learning with noisy labels

  • Nagarajan, Bhalaji
  • Marques, Ricardo
  • Aguilar, Eduardo J.
  • Radeva, Petia
Leveraging inexpensive and human intervention-based annotating methodologies, such as crowdsourcing and web crawling, often leads to datasets with noisy labels. Noisy labels can have a detrimental impact on the performance and generalization of deep neural networks. Robust models that are able to handle and mitigate the effect of these noisy labels are thus essential. In this work, we explore the open challenges of neural network memorization and uncertainty in creating robust learning algorithms with noisy labels. To overcome them, we propose a novel framework called “Bayesian DivideMix++” with two critical components: (i) DivideMix++, to enhance the robustness against memorization and (ii) Monte-Carlo MixMatch, which focuses on improving the effectiveness towards label uncertainty. DivideMix++ improves the pipeline by integrating the warm-up and augmentation pipeline with self-supervised pre-training and dedicated different data augmentations for loss analysis and backpropagation. Monte-Carlo MixMatch leverages uncertainty measurements to mitigate the influence of uncertain samples by reducing their weight in the data augmentation MixMatch step. We validate our proposed pipeline using four datasets encompassing various synthetic and real-world noise settings. We demonstrate the effectiveness and merits of our proposed pipeline using extensive experiments. Bayesian DivideMix++ outperforms the state-of-the-art models by considerable differences in all experiments. Our findings underscore the potential of leveraging these modifications to enhance the performance and generalization of deep neural networks in practical scenarios., This work was partially funded by the Horizon EU project MUSAE (No. 01070421), 2021-SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220-VET-000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), DeepFoodVol (AEI-MICINN, PDC2022-133642-I00), IDEATE (AEI-MICINN, PID2022-141566NB-I00), A-BMC (AEI-MICINN, CNS2022-135480), CERCA Programme/Generalitat de Catalunya, and Agencia Nacional de Investigación y Desarrollo de Chile (ANID) (Grant No. FONDECYT INICIACIÓN 11230262). Ricardo Marques acknowledges the support of the Serra Húnter Programme. B. Nagarajan acknowledges the support of FPI Becas, MICINN, Spain . The authors thankfully acknowledge the computer resources at FinisTerrae III and the technical support provided by the Galician Supercomputing Center (CESGA) (RES-IM-2023-2-0025).
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

FoodMem: near real-time and precise food video segmentation

  • AlMughrabi, Ahmad
  • Galán, Adrián
  • Marques, Ricardo
  • Radeva, Petia
Food segmentation, including in videos, is vital for addressing real-world health, agriculture, and food biotechnology issues. Current limitations lead to inaccurate nutritional analysis, inefficient crop management, and suboptimal food processing, impacting food security and public health. Improving segmentation techniques can enhance dietary assessments, agricultural productivity, and the food production process. This study introduces the development of a robust framework for high-quality, near-real-time segmentation and tracking of food items in videos, using minimal hardware resources. We present FoodMem, a novel framework designed to segment food items from video sequences of 360-degree unbounded scenes. FoodMem can consistently generate masks of food portions in a video sequence, overcoming the limitations of existing semantic segmentation models, such as flickering and prohibitive inference speeds in video processing contexts. To address these issues, FoodMem leverages a two-phase solution: a transformer segmentation phase to create initial segmentation masks and a memory-based tracking phase to monitor food masks in complex scenes. Our framework outperforms current state-of-the-art food segmentation models, yielding superior performance across various conditions, such as camera angles, lighting, reflections, scene complexity, and food diversity.2This results in reduced segmentation noise, elimination of artifacts, and completion of missing segments. We also introduce a new annotated food dataset encompassing challenging scenarios absent in previous benchmarks. Extensive experiments conducted on MetaFood3D, Nutrition5k, and Vegetables & Fruits datasets demonstrate that FoodMem enhances the state-of-the-art by 2.5% mean average precision in food video segmentation and is faster on average. The source code is available at: 3., This work was partially funded by the EU project MUSAE (No. 01070421), 2021- SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220- VET000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), and Grants PID2022141566NB-I00 (IDEATE), PDC2022-133642-I00 (DeepFoodVol), and CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. A. AlMughrabi acknowledges the support of FPI Becas, MICINN, Spain.
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Decoding class dynamics in learning with noisy labels

  • Tatjer, Albert
  • Nagarajan, Bhalaji
  • Marques, Ricardo
  • Radeva, Petia
The creation of large-scale datasets annotated by humans inevitably introduces noisy labels, leading to reduced generalization in deep-learning models. Sample selection-based learning with noisy labels is a recent approach that exhibits promising upbeat performance improvements. The selection of clean samples amongst the noisy samples is an important criterion in the learning process of these models. In this work, we delve deeper into the clean-noise split decision and highlight the aspect that effective demarcation of samples would lead to better performance. We identify the Global Noise Conundrum in the existing models, where the distribution of samples is treated globally. We propose a per-class-based local distribution of samples and demonstrate the effectiveness of this approach in having a better clean-noise split. We validate our proposal on several benchmarks — both real and synthetic, and show substantial improvements over different state-of-the-art algorithms. We further propose a new metric, classiness to extend our analysis and highlight the effectiveness of the proposed method. Source code and instructions to reproduce this paper are available at https://github.com/aldakata/CCLM/, This work was partially funded by the EU project MUSAE (No. 01070421), 2021-SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220-VET-000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), CERCA Programme/Generalitat de Catalunya, and Grants PID2022-141566NB-I00 (IDEATE), PDC2022-133642-I00 (DeepFoodVol), and CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. R. Marques acknowledges the support of the Serra Húnter Programme. B. Nagarajan acknowledges the support of FPI Becas, MICINN, Spain .
Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

Fast and accurate gaussian process modelling of real-world materials

  • Colom, Arnau
  • Bouville, Christian
  • Pettre, Julien
  • Bouatouch, Kadi
  • Marques, Ricardo
Our goal in this article is to propose a fast and easy to implement BRDF modeling method that provides both accurate and compact representations for all types of BRDF, i.e., isotropic or anisotropic. To achieve this objective, we use a Bayesian regression method with a Gaussian process prior which allows obtaining compact BRDF representations in a purely analytical way. For this purpose, we use a generalzed distance covariance kernel which is much better suited to BRDF features than the usual Gaussian kernel. To speed up the processing, we adapt this method to the specificities of BRDFs through an appropriate input data structure and distribution of observations so as to drastically reduce the problem dimensionality through an efficient factorization method. In this way, all calculations at both fitting and rendering steps are reduced to basic matrix products and the computation of a BRDF representation with our modeling method takes only a few seconds. Furthermore, rather than using a systematic approach as in state-of-the-art methods, the size and complexity of the BRDF representation can be adapted to the application requirements as regards the fitting accuracy and rendering constraints. Besides, our BRDF representation can be easily converted to spherical harmonics expansions, which allows easier integration in usual rendering algorithms. We also propose importance sampling methods derived from our BRDF modeling method that leads to fast and easy implementations. Experimental applications of our method to various types of isotropic and anisotropic BRDFs show that state-of-the-art methods can be outperformed in most cases by using a small set of observations for the regression., This work was partially funded by the Research Consolidation grant CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. It has also been partially founded by PID2022141566NB-I00 (IDEATE)..
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Bayesian DivideMix++ for enhanced learning with noisy labels

  • Nagarajan, Bhalaji
  • Marques, Ricardo
  • Aguilar, Eduardo J.
  • Radeva, Petia
Leveraging inexpensive and human intervention-based annotating methodologies, such as crowdsourcing and web crawling, often leads to datasets with noisy labels. Noisy labels can have a detrimental impact on the performance and generalization of deep neural networks. Robust models that are able to handle and mitigate the effect of these noisy labels are thus essential. In this work, we explore the open challenges of neural network memorization and uncertainty in creating robust learning algorithms with noisy labels. To overcome them, we propose a novel framework called “Bayesian DivideMix++” with two critical components: (i) DivideMix++, to enhance the robustness against memorization and (ii) Monte-Carlo MixMatch, which focuses on improving the effectiveness towards label uncertainty. DivideMix++ improves the pipeline by integrating the warm-up and augmentation pipeline with self-supervised pre-training and dedicated different data augmentations for loss analysis and backpropagation. Monte-Carlo MixMatch leverages uncertainty measurements to mitigate the influence of uncertain samples by reducing their weight in the data augmentation MixMatch step. We validate our proposed pipeline using four datasets encompassing various synthetic and real-world noise settings. We demonstrate the effectiveness and merits of our proposed pipeline using extensive experiments. Bayesian DivideMix++ outperforms the state-of-the-art models by considerable differences in all experiments. Our findings underscore the potential of leveraging these modifications to enhance the performance and generalization of deep neural networks in practical scenarios., This work was partially funded by the Horizon EU project MUSAE (No. 01070421), 2021-SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220-VET-000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), DeepFoodVol (AEI-MICINN, PDC2022-133642-I00), IDEATE (AEI-MICINN, PID2022-141566NB-I00), A-BMC (AEI-MICINN, CNS2022-135480), CERCA Programme/Generalitat de Catalunya, and Agencia Nacional de Investigación y Desarrollo de Chile (ANID) (Grant No. FONDECYT INICIACIÓN 11230262). Ricardo Marques acknowledges the support of the Serra Húnter Programme. B. Nagarajan acknowledges the support of FPI Becas, MICINN, Spain . The authors thankfully acknowledge the computer resources at FinisTerrae III and the technical support provided by the Galician Supercomputing Center (CESGA) (RES-IM-2023-2-0025).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Decoding class dynamics in learning with noisy labels

  • Tatjer, Albert
  • Nagarajan, Bhalaji
  • Marques, Ricardo
  • Radeva, Petia
The creation of large-scale datasets annotated by humans inevitably introduces noisy labels, leading to reduced generalization in deep-learning models. Sample selection-based learning with noisy labels is a recent approach that exhibits promising upbeat performance improvements. The selection of clean samples amongst the noisy samples is an important criterion in the learning process of these models. In this work, we delve deeper into the clean-noise split decision and highlight the aspect that effective demarcation of samples would lead to better performance. We identify the Global Noise Conundrum in the existing models, where the distribution of samples is treated globally. We propose a per-class-based local distribution of samples and demonstrate the effectiveness of this approach in having a better clean-noise split. We validate our proposal on several benchmarks — both real and synthetic, and show substantial improvements over different state-of-the-art algorithms. We further propose a new metric, classiness to extend our analysis and highlight the effectiveness of the proposed method. Source code and instructions to reproduce this paper are available at https://github.com/aldakata/CCLM/, This work was partially funded by the EU project MUSAE (No. 01070421), 2021-SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220-VET-000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), CERCA Programme/Generalitat de Catalunya, and Grants PID2022-141566NB-I00 (IDEATE), PDC2022-133642-I00 (DeepFoodVol), and CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. R. Marques acknowledges the support of the Serra Húnter Programme. B. Nagarajan acknowledges the support of FPI Becas, MICINN, Spain .
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Fast and accurate gaussian process modelling of real-world materials

  • Colom, Arnau
  • Bouville, Christian
  • Pettre, Julien
  • Bouatouch, Kadi
  • Marques, Ricardo
Our goal in this article is to propose a fast and easy to implement BRDF modeling method that provides both accurate and compact representations for all types of BRDF, i.e., isotropic or anisotropic. To achieve this objective, we use a Bayesian regression method with a Gaussian process prior which allows obtaining compact BRDF representations in a purely analytical way. For this purpose, we use a generalzed distance covariance kernel which is much better suited to BRDF features than the usual Gaussian kernel. To speed up the processing, we adapt this method to the specificities of BRDFs through an appropriate input data structure and distribution of observations so as to drastically reduce the problem dimensionality through an efficient factorization method. In this way, all calculations at both fitting and rendering steps are reduced to basic matrix products and the computation of a BRDF representation with our modeling method takes only a few seconds. Furthermore, rather than using a systematic approach as in state-of-the-art methods, the size and complexity of the BRDF representation can be adapted to the application requirements as regards the fitting accuracy and rendering constraints. Besides, our BRDF representation can be easily converted to spherical harmonics expansions, which allows easier integration in usual rendering algorithms. We also propose importance sampling methods derived from our BRDF modeling method that leads to fast and easy implementations. Experimental applications of our method to various types of isotropic and anisotropic BRDFs show that state-of-the-art methods can be outperformed in most cases by using a small set of observations for the regression., This work was partially funded by the Research Consolidation grant CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. It has also been partially founded by PID2022141566NB-I00 (IDEATE)..
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

FoodMem: near real-time and precise food video segmentation

  • AlMughrabi, Ahmad
  • Galán, Adrián
  • Marques, Ricardo
  • Radeva, Petia
Food segmentation, including in videos, is vital for addressing real-world health, agriculture, and food biotechnology issues. Current limitations lead to inaccurate nutritional analysis, inefficient crop management, and suboptimal food processing, impacting food security and public health. Improving segmentation techniques can enhance dietary assessments, agricultural productivity, and the food production process. This study introduces the development of a robust framework for high-quality, near-real-time segmentation and tracking of food items in videos, using minimal hardware resources. We present FoodMem, a novel framework designed to segment food items from video sequences of 360-degree unbounded scenes. FoodMem can consistently generate masks of food portions in a video sequence, overcoming the limitations of existing semantic segmentation models, such as flickering and prohibitive inference speeds in video processing contexts. To address these issues, FoodMem leverages a two-phase solution: a transformer segmentation phase to create initial segmentation masks and a memory-based tracking phase to monitor food masks in complex scenes. Our framework outperforms current state-of-the-art food segmentation models, yielding superior performance across various conditions, such as camera angles, lighting, reflections, scene complexity, and food diversity.2This results in reduced segmentation noise, elimination of artifacts, and completion of missing segments. We also introduce a new annotated food dataset encompassing challenging scenarios absent in previous benchmarks. Extensive experiments conducted on MetaFood3D, Nutrition5k, and Vegetables & Fruits datasets demonstrate that FoodMem enhances the state-of-the-art by 2.5% mean average precision in food video segmentation and is faster on average. The source code is available at: 3., This work was partially funded by the EU project MUSAE (No. 01070421), 2021- SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220- VET000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), and Grants PID2022141566NB-I00 (IDEATE), PDC2022-133642-I00 (DeepFoodVol), and CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. A. AlMughrabi acknowledges the support of FPI Becas, MICINN, Spain.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Rooms: a WebGPU-Based VR platform for intuitive 3D sculpting and animation

  • Marquerie Cordoba, Juan Sebastian
  • Rodríguez, Alejandro
  • García, Pablo L.
  • Marques, Ricardo
  • Blat, Josep
Data de publicació electrònica: 21-07-2026, We propose Rooms, an open-source XR content creation platform designed to democratize 3D modeling and animation by leveraging the intuitiveness of VR motion controls and the high-performance capabilities of the modern WebGPU API. Built on top of wgpuEngine, Rooms provides real-time, cross-platform immersive sculpting and animation tools directly accessible through XR-enabled web browsers and desktop systems. The system implements a Signed Distance Field (SDF) sculpting pipeline that enables smooth Constructive Solid Geometry (CSG) operations for clay-like modeling. This paper details the architecture, the XR-first user interaction workflow, and the technical implementation, which includes a GPU-driven, sparse SDF baking pipeline. User studies confirm that the immersive, motion-based interaction model successfully lowers the entry barrier for non-expert users. Rooms effectively paves the way for future web-based XR creation environments by guiding the development of accessible and intuitive tools that harness the full power of modern GPU APIs on the web., This project has received partial financial support from the MAX-R project (Mixed Augmented and eXtended Reality media pipeline, H2020-Digital, Industry and Space, Grant Agreement No.101070072) and from the Wi-XR project (MCIN/FEDER, PID2021-123995NB-I00). Ricardo Marques acknowledges the support of CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. We thank the MAX-R partners, and especially UPF-WiNet and FABW for mutual support in user studies.