Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 8
Encontrada(s) 1 página(s)

UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Stretch your reach: studying self-avatar and controller misalignment in virtual reality interaction

  • Pontón Martínez, José Luis|||0000-0001-6576-4528
  • Keshavarz, Reza
  • Beacco Porres, Alejandro|||0000-0001-8192-1431
  • Pelechano Gómez, Núria|||0000-0002-1437-245X
Immersive Virtual Reality typically requires a head-mounted display (HMD) to visualize the environment and hand-held controllers to interact with the virtual objects. Recently, many applications display full-body avatars to represent the user and animate the arms to follow the controllers. Embodiment is higher when the self-avatar movements align correctly with the user. However, having a full-body self-avatar following the user’s movements can be challenging due to the disparities between the virtual body and the user’s body. This can lead to misalignments in the hand position that can be noticeable when interacting with virtual objects. In this work, we propose five different interaction modes to allow the user to interact with virtual objects despite the self-avatar and controller misalignment and study their influence on embodiment, proprioception, preference, and task performance. We modify aspects such as whether the virtual controllers are rendered, whether controllers are rendered in their real physical location or attached to the user’s hand, and whether stretching the avatar arms to always reach the real controllers. We evaluate the interaction modes both quantitatively (performance metrics) and qualitatively (embodiment, proprioception, and user preference questionnaires). Our results show that the stretching arms solution, which provides body continuity and guarantees that the virtual hands or controllers are in the correct location, offers the best results in embodiment, user preference, proprioception, and performance. Also, rendering the controller does not have an effect on either embodiment or user preference., This work has received funding from the European Union’sHorizon 2020 research and innovation programme under HORIZON-CL42022-HUMAN-01 grant agreement No 101093159 (XR4ED), and from MCIN/AEI/10.13039/501100011033/FEDER "A way to make Europe", UE (PID2021-122136OB-C21). Jose Luis Ponton was also funded by the Spanish Ministry of Universities (FPU21/01927)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

SparsePoser: Real-time full-body motion reconstruction from sparse data

  • Pontón Martínez, José Luis|||0000-0001-6576-4528
  • Yun, Haoran
  • Aristidou, Andreas
  • Andújar Gran, Carlos Antonio|||0000-0002-8480-4713
  • Pelechano Gómez, Núria|||0000-0002-1437-245X
Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie SkŁodowska-Curie grant agreement No 860768 (CLIPE project), HORIZON-CL4-2022-HUMAN-01 grant agreement No 101093159 (XR4ED), and from MCIN/AEI/10.13039/501100011033/FEDER, UE (PID2021-122136OB-C21). This project has also received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No 739578 and the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy. Jose Luis Ponton was also funded by the Spanish Ministry of Universities (FPU21/01927)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

DragPoser: motion reconstruction from variable sparse tracking signals via latent space optimization

  • Pontón Martínez, José Luis|||0000-0001-6576-4528
  • Pujol Puig, Eduard|||0000-0001-6406-7499
  • Aristidou, Andreas
  • Andújar Gran, Carlos Antonio|||0000-0002-8480-4713
  • Pelechano Gómez, Núria|||0000-0002-1437-245X
High-quality motion reconstruction that follows the user's movements can be achieved by high-end mocap systems with many sensors. However, obtaining such animation quality with fewer input devices is gaining popularity as it brings mocap closer to the general public. The main challenges include the loss of end-effector accuracy in learning-based approaches, or the lack of naturalness and smoothness in IK-based solutions. In addition, such systems are often finely tuned to a specific number of trackers and are highly sensitive to missing data, e.g., in scenarios where a sensor is occluded or malfunctions. In response to these challenges, we introduce DragPoser, a novel deep-learning-based motion reconstruction system that accurately represents hard and dynamic constraints, attaining real-time high end-effectors position accuracy. This is achieved through a pose optimization process within a structured latent space. Our system requires only one-time training on a large human motion dataset, and then constraints can be dynamically defined as losses, while the pose is iteratively refined by computing the gradients of these losses within the latent space. To further enhance our approach, we incorporate a Temporal Predictor network, which employs a Transformer architecture to directly encode temporality within the latent space. This network ensures the pose optimization is confined to the manifold of valid poses and also leverages past pose data to predict temporally coherent poses. Results demonstrate that DragPoser surpasses both IK-based and the latest data-driven methods in achieving precise end-effector positioning, while it produces natural poses and temporally coherent motion. In addition, our system showcases robustness against on-the-fly constraint modifications, and exhibits adaptability to various input configurations and changes. The complete source code, trained model, animation databases, and supplementary material used in this paper can be found at https://upc-virvig.github.io/DragPoser, This work has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under HORIZON-CL4-2022-HUMAN-01 grant agreement No 101093159 (XR4ED) and grant agreement No 739578 (RISE), from MCIN/AEI/10.13039/501100011033/FEDER “A way to make Europe”, UE (PID2021-122136OB-C21), from the Departament of Research and Universities of the Government of Catalonia (2021 SGR 01035), and the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy. Jose Luis Ponton was also funded by the Spanish Ministry of Universities (FPU21/01927), and Eduard Pujol by Universitat Politècnica de Catalunya and Banco Santander (FPI-UPC predoctoral grant)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

XR4ED: an extended reality platform for education

  • Liarokapis, Fotis
  • Milata, Vaclav
  • Pontón Martínez, José Luis|||0000-0001-6576-4528
  • Pelechano Gómez, Núria|||0000-0002-1437-245X
  • Zacharatos, Haris
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. https://dx.doi.org/10.1109/MCG.2024.3406139, Recent developments in extended reality (XR) are already demonstrating the benefits of this technology in the educational sector. Unfortunately, educators may not be familiar with XR technology and may find it difficult to adopt this technology in their classrooms. This article presents the overall architecture and objectives of an EU-funded project dedicated to XR for education, called Extended Reality for Education (XR4ED). The goal of the project is to provide a platform, where educators will be able to build XR teaching experiences without the need to have programming or 3-D modeling expertise. The platform will provide the users with a marketplace to obtain, for example, 3-D models, avatars, and scenarios; graphical user interfaces to author new teaching environments; and communication channels to allow for collaborative virtual reality (VR). This article describes the platform and focuses on a key aspect of collaborative and social XR, which is the use of avatars. We show initial results on a) a marketplace which is used for populating educational content into XR environments, b) an intelligent augmented reality assistant that communicates between nonplayer characters and learners, and c) self-avatars providing nonverbal communication in collaborative VR., This work was supported in part by the European Union’s Horizon Europe research and innovation programme under Grant 101093159, in part by the European Union’s Horizon 2020 Research and Innovation Programme under Grant 739578, and in part by the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy under Grant 101061303., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

XR4ED: an extended reality platform for education

  • Liarokapis, Fotis
  • Milata, Vaclav
  • Pontón Martínez, José Luis
  • Pelechano Gómez, Núria
  • Zacharatos, Haris
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. https://dx.doi.org/10.1109/MCG.2024.3406139, Recent developments in extended reality (XR) are already demonstrating the benefits of this technology in the educational sector. Unfortunately, educators may not be familiar with XR technology and may find it difficult to adopt this technology in their classrooms. This article presents the overall architecture and objectives of an EU-funded project dedicated to XR for education, called Extended Reality for Education (XR4ED). The goal of the project is to provide a platform, where educators will be able to build XR teaching experiences without the need to have programming or 3-D modeling expertise. The platform will provide the users with a marketplace to obtain, for example, 3-D models, avatars, and scenarios; graphical user interfaces to author new teaching environments; and communication channels to allow for collaborative virtual reality (VR). This article describes the platform and focuses on a key aspect of collaborative and social XR, which is the use of avatars. We show initial results on a) a marketplace which is used for populating educational content into XR environments, b) an intelligent augmented reality assistant that communicates between nonplayer characters and learners, and c) self-avatars providing nonverbal communication in collaborative VR., This work was supported in part by the European Union’s Horizon Europe research and innovation programme under Grant 101093159, in part by the European Union’s Horizon 2020 Research and Innovation Programme under Grant 739578, and in part by the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy under Grant 101061303., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

SparsePoser: Real-time full-body motion reconstruction from sparse data

  • Pontón Martínez, José Luis
  • Yun, Haoran
  • Aristidou, Andreas
  • Andújar Gran, Carlos Antonio
  • Pelechano Gómez, Núria
Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie SkŁodowska-Curie grant agreement No 860768 (CLIPE project), HORIZON-CL4-2022-HUMAN-01 grant agreement No 101093159 (XR4ED), and from MCIN/AEI/10.13039/501100011033/FEDER, UE (PID2021-122136OB-C21). This project has also received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No 739578 and the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy. Jose Luis Ponton was also funded by the Spanish Ministry of Universities (FPU21/01927)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

DragPoser: motion reconstruction from variable sparse tracking signals via latent space optimization

  • Pontón Martínez, José Luis
  • Pujol Puig, Eduard
  • Aristidou, Andreas
  • Andújar Gran, Carlos Antonio
  • Pelechano Gómez, Núria
High-quality motion reconstruction that follows the user's movements can be achieved by high-end mocap systems with many sensors. However, obtaining such animation quality with fewer input devices is gaining popularity as it brings mocap closer to the general public. The main challenges include the loss of end-effector accuracy in learning-based approaches, or the lack of naturalness and smoothness in IK-based solutions. In addition, such systems are often finely tuned to a specific number of trackers and are highly sensitive to missing data, e.g., in scenarios where a sensor is occluded or malfunctions. In response to these challenges, we introduce DragPoser, a novel deep-learning-based motion reconstruction system that accurately represents hard and dynamic constraints, attaining real-time high end-effectors position accuracy. This is achieved through a pose optimization process within a structured latent space. Our system requires only one-time training on a large human motion dataset, and then constraints can be dynamically defined as losses, while the pose is iteratively refined by computing the gradients of these losses within the latent space. To further enhance our approach, we incorporate a Temporal Predictor network, which employs a Transformer architecture to directly encode temporality within the latent space. This network ensures the pose optimization is confined to the manifold of valid poses and also leverages past pose data to predict temporally coherent poses. Results demonstrate that DragPoser surpasses both IK-based and the latest data-driven methods in achieving precise end-effector positioning, while it produces natural poses and temporally coherent motion. In addition, our system showcases robustness against on-the-fly constraint modifications, and exhibits adaptability to various input configurations and changes. The complete source code, trained model, animation databases, and supplementary material used in this paper can be found at https://upc-virvig.github.io/DragPoser, This work has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under HORIZON-CL4-2022-HUMAN-01 grant agreement No 101093159 (XR4ED) and grant agreement No 739578 (RISE), from MCIN/AEI/10.13039/501100011033/FEDER “A way to make Europe”, UE (PID2021-122136OB-C21), from the Departament of Research and Universities of the Government of Catalonia (2021 SGR 01035), and the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy. Jose Luis Ponton was also funded by the Spanish Ministry of Universities (FPU21/01927), and Eduard Pujol by Universitat Politècnica de Catalunya and Banco Santander (FPI-UPC predoctoral grant)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Stretch your reach: studying self-avatar and controller misalignment in virtual reality interaction

  • Pontón Martínez, José Luis
  • Keshavarz, Reza
  • Beacco Porres, Alejandro
  • Pelechano Gómez, Núria
Immersive Virtual Reality typically requires a head-mounted display (HMD) to visualize the environment and hand-held controllers to interact with the virtual objects. Recently, many applications display full-body avatars to represent the user and animate the arms to follow the controllers. Embodiment is higher when the self-avatar movements align correctly with the user. However, having a full-body self-avatar following the user’s movements can be challenging due to the disparities between the virtual body and the user’s body. This can lead to misalignments in the hand position that can be noticeable when interacting with virtual objects. In this work, we propose five different interaction modes to allow the user to interact with virtual objects despite the self-avatar and controller misalignment and study their influence on embodiment, proprioception, preference, and task performance. We modify aspects such as whether the virtual controllers are rendered, whether controllers are rendered in their real physical location or attached to the user’s hand, and whether stretching the avatar arms to always reach the real controllers. We evaluate the interaction modes both quantitatively (performance metrics) and qualitatively (embodiment, proprioception, and user preference questionnaires). Our results show that the stretching arms solution, which provides body continuity and guarantees that the virtual hands or controllers are in the correct location, offers the best results in embodiment, user preference, proprioception, and performance. Also, rendering the controller does not have an effect on either embodiment or user preference., This work has received funding from the European Union’sHorizon 2020 research and innovation programme under HORIZON-CL42022-HUMAN-01 grant agreement No 101093159 (XR4ED), and from MCIN/AEI/10.13039/501100011033/FEDER "A way to make Europe", UE (PID2021-122136OB-C21). Jose Luis Ponton was also funded by the Spanish Ministry of Universities (FPU21/01927)., Peer Reviewed, Postprint (author's final draft)