Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 28
Encontrada(s) 1 página(s)

UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Perception-intention-action cycle as a human acceptable way for improving human-robot collaborative tasks

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Rodríguez Linares, Nicolás Adrián|||0000-0003-4834-223X
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© ACM 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in HRI '23: Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, https://doi.org/10.1145/3568294.3580149., In Human-Robot Collaboration (HRC) tasks, the classical Perception-Action cycle can not fully explain the collaborative behaviour of the human-robot pair until it is extended to Perception-Intention-Action (PIA) cycle, giving to the human's intention a key role at the same level of the robot's perception and not as a subblock of this. Although part of the human's intention can be perceived or inferred by the other agent, this is prone to misunderstandings so the true intention has to be explicitly informed in some cases to fulfill the task. Here, we explore both types of intention and we combine them with the robot's perception through the concept of Situation Awareness (SA). We validate the PIA cycle and its acceptance by the user with a preliminary experiment in an object transportation task showing that its usage can increase trust in the robot., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Body gestures recognition for social human-robot interaction

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
In this paper, a solution for human gesture classification is proposed. The solution uses a Deep Learning model and is meant to be useful for non-verbal communication between humans and robots. The research focuses on the creation of the “temPoral bOdy geSTUre REcognition model” (POSTURE) that can recognise continuous gestures performed in real-life situations. The suggested model takes into account spatial and temporal components so as to achieve the recognition of more natural and intuitive gestures. In a first step, a framework extracts from all the images the corresponding landmarks for each of the body joints. Next, some data filtering techniques are applied with the aim of avoiding problems related with the data. And finally, different neural network configurations and approaches are tested to find the optimal performance. The validation of the model is accomplished throughout an extensive set of simulations and real-life experiments., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Inference vs. explicitness. Do we really need the perfect predictor? The human-robot collaborative object transportation case

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
When robots interact with humans, limitations in their internal models arise due to the uncertainty and even randomness of human behavior. This has led to attempts to predict human future actions and infer their intent. However, some authors argue for combining inference engines with communication systems that explicitly elicit human intention. This work builds on our Perception-Intention-Action (PIA) cycle, a framework that considers human intention at the same level as perception of the environment. The PIA cycle is used in a collaborative task to compare the effect on different human- robot interaction aspects of using a force predictor that infers human implicit intention versus a communication system that explicitly elicits human intention. A study with 18 volunteers shows that allowing humans to directly express themselves can achieve the same improvement as an intention predictor., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and by JST Moonshot R & D Grant Number JPMJMS2011. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Improving human-robot interaction effectiveness in human-robot collaborative object transportation using force prediction

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work, we analyse the use of a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We check that this force prediction can improve multiple parameters associated with effective Human-Robot Interaction (HRI) such as percep- tion of the robot’s contribution to the task, comfort or trust in the robot in a physical Human Robot Interaction (pHRI). We present a Deep Learning model that allows to predict the force that a human will exert in the next 1 s using as inputs the force previously exerted by the human, the robot’s velocity and environment information obtained from the robot’s LiDAR. Its success rate is up to 92.3% in testset and up to 89.1% in real experiments. We demonstrate that this force prediction, in addition to being able to be used directly to detect changes in the human’s intention, can be processed to obtain an estimate of the human’s desired trajectory. We have validated this approach with a user study involving 18 volunteers., Work supported under the European project CANOPIES (H2020 ICT-2020-2-101016906) and the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033) and by JST Moonshot R&D Grant Number JPMJMS2011.The first author acknowledges Spanish FPU grant with ref.FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Real-life experiment metrics for evaluating human-robot collaborative navigation tasks

  • Repiso Polo, Ely|||0000-0001-6540-8347
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., As robots move from laboratories and industries to the real world, they must develop new abilities to collaborate with humans in various aspects, including human-robot collaborative navigation (HRCN) tasks. Then, it is required to develop general methodologies to evaluate these robots’ behaviors. These methodologies should incorporate objective and subjective measurements. Objective measurements for evaluating a robot’s behavior while navigating with others can be accomplished using social distances in conjunction with task characteristics, people-robot relationships, and physical space. Additionally, the objective evaluation of the task must consider human behavior, which is influenced by changes and the structure of their environment. Subjective evaluations of robot’s behaviors can be conducted using surveys that address various aspects of robot usability. This includes people’s perceptions of their interaction during their collaborative task with the robot, focusing on aspects such as sociability, comfort, and task-intelligence. Moreover, the communicative interaction between the agents (people and robots) involved in the collaborative task should also be evaluated. Therefore, this paper presents a comprehensive methodology for objectively and subjectively evaluating HRCN tasks., This work has been supported by the Artificial Intelligence for Human- Robot Interaction (AI4HRI) project ANR-20-IADJ-0006. Also, this Work has been supported under the EU project grant CANOPIES (H2020- ICT-2020-2-101016906) and by the JST Moonshot R D Grant Number JPMJMS2011. Ely Repiso was also supported during her PhD by Spanish Ministry of Science and Innovation under a FPI-grant (BES-2014-067713) and during her Post Doc by the project AI4HRI, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Attention deep learning based model for predicting the 3D Human Body Pose using the Robot Human Handover Phases

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This work proposes a human motion prediction model for handover operations. We use in this work, the different phases of the handover operation to improve the human motion predictions. Our attention deep learning based model takes into account the position of the robot’s End Effector and the phase in the handover operation to predict future human poses. Our model outputs a distribution of possible positions rather than one deterministic position, a key feature in order to allow robots to collaborate with humans. The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained with an Intel RealSense D435i camera attached inside the robot’s head. The results shown a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Classification of humans social relations within urban areas

  • Castro Arcusa, Oscar
  • Repiso Polo, Ely|||0000-0001-6540-8347
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
This paper presents the design of deep learning architectures which allow to classify the social relationship existing between two people who are walking in a side-by-side formation into four possible categories --colleagues, couple, family or friendship. The models are developed using Neural Networks or Recurrent Neural Networks to achieve the classification and are trained and evaluated using a database obtained from humans walking together in an urban environment. The best achieved model accomplishes a good accuracy in the classification problem and its results enhance the outcomes from a previous study [1]. In addition, we have developed several models to classify the social interactions in two categories --“intimate" and "acquaintances", where the best model achieves a very good performance, and for a real robot this classification is enough to be able to customize its behavior to its users. Furthermore, the proposed models show their future potential to improve its efficiency and to be implemented in a real robot., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Human motion trajectory prediction using the social force model for real-time and low computational cost applications

  • Gil Viyuela, Óscar|||0000-0001-8812-3168
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG, Human motion trajectory prediction is a very important functionality for human-robot collaboration, specifically in accompanying, guiding, or approaching tasks, but also in social robotics, self-driving vehicles, or security systems. In this paper, a novel trajectory prediction model, Social Force Generative Adversarial Network (SoFGAN), is proposed. SoFGAN uses a Generative Adversarial Network (GAN) and Social Force Model (SFM) to generate different plausible people trajectories reducing collisions in a scene. Furthermore, a Conditional Variational Autoencoder (CVAE) module is added to emphasize the destination learning. We show that our method is more accurate in making predictions in UCY or BIWI datasets than most of the current state-of-the-art models and also reduces collisions in comparison to other approaches. Through real-life experiments, we demonstrate that the model can be used in real-time without GPU’s to perform good quality predictions with a low computational cost., Ó. Gil—Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and JST Moonshot R & D Grant Number JPMJMS2011-85., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Exploring transformers and visual transformers for force prediction in human-robot collaborative transportation tasks

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we analyze the possibilities offered by Deep Learning State-of-the-Art architectures such as Transformers and Visual Transformers in generating a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We outperform our previous predictor by achieving a success rate of 93.8% in testset and 90.9% in real experiments with 21 volunteers predicting in both cases the force that the human will exert during the next 1 s. A modification in the architecture allows us to obtain a second output from the model with a velocity prediction, which allows us to improve the capabilities of our predictor if it is used to estimate the trajectory that the humanrobot pair will follow. An ablation test is also performed to verify the relative contribution to performance of each input., Work supported under the European project CANOPIES (H2020ICT-2020-2-101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Human-robot harvesting plan negotiation: perception, grape mapping and shared planning

  • Dalmasso Blanch, Marc|||0000-0003-1734-7863
  • García Ruiz, Francisco
  • Ciarfuglia, Thomas A.
  • Saraceni, Leonardo
  • Motoi, Ionut M.
  • Gil Moya, Emilio|||0000-0002-3929-5649
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper we present an application of plan negotiation for distributing work in Human-Robot Teams (HRT), engaging in a shared plan generation process. This model has been experimented over data collected from a real-life table grape vineyard field in Corsira (Aprilia, Italy) on the context of the EU CANOPIES project. In this work we describe a complete system up to negotiation agreement, the perception system to detect the grapes position and quality, the grape map generation process and the proposed plan negotiation approach, and protocol used to interactively generate harvest plans. Our proposed approach tackles harvest plan generation as a two step negotiation process on negotiation domains of increasing embedded information. The shared planning process was evaluated using both a subjective user study and teammate quantitative models., This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016906 – Project CANOPIES, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Force and velocity prediction in human-robot collaborative transportation tasks through video retentive networks

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, In this article, we propose a generalization of a Deep Learning State-of-the-Art architecture such as Retentive Networks so that it can accept video sequences as input. With this generalization, we design a force/velocity predictor applied to the medium-distance Human-Robot collaborative object transportation task. We achieve better results than with our previous predictor by reaching success rates in testset of up to 93.7% in predicting the force to be exerted by the human and up to 96.5% in the velocity of the human-robot pair during the next 1 s, and up to 91.0% and 95.0% respectively in real experiments. This new architecture also manages to improve inference times by up to 32.8% with different graphics cards. Finally, an ablation test allows us to detect that one of the input variables used so far, such as the position of the task goal, could be discarded allowing this goal to be chosen dynamically by the human instead of being pre-set., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Context and intention for 3D human motion prediction: experimentation and user study in handover tasks

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work we present a novel attention deep learning model that uses context and human intention for 3D human body motion prediction in handover human-robot tasks. This model uses a multi-head attention architecture which incorporates as inputs the human motion, the robot end effector and the position of the obstacles. The outputs of the model are the predicted motion of the human body and the predicted human intention. We use this model to analyze a handover collaborative task with a robot where the robot is able to predict the future motion of the human and use this information in it’s planner. Several experiments are performed where human volunteers fill a standard poll to rate different features, taking into account when the robot uses the prediction versus when the robot doesn’t use the prediction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016- 0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI / 10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT2020-2-101016906)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Context attention: human motion prediction using context information and deep learning attention models

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
This work proposes a human motion prediction model for handover operations. The model uses a multi-headed attention architecture to process the human skeleton data together with contextual data from the operation. This contextual data consists on the position of the robot’s End Effector (REE). The model input is a sequence of 5 seconds skeleton position and it outputs the predicted 2.5 future seconds position. We provide results of the human upper body and the human right hand or Human End Effector (HEE). The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained using OpenPose with an Intel RealSense D435i camera set inside the robot’s head. The results show a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the ROCOTRANSP project (PID2019-106702RB-C21/AEI/10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Body gesture recognition to control a social mobile robot

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Romero Martín, Rut|||0000-0002-9257-3898
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
In this work, we propose a gesture-based language to allow humans to interact with robots using their body in a natural way. We have created a new gesture detection model using neural networks and a new dataset of humans making a collection of body gestures to train this architecture. Furthermore, we compare body gesture communication with other communication channels to demonstrate the importance of adding this knowledge to robots. The presented approach is validated in diverse simulations and real-life experiments with non-trained volunteers. This attains promising results and establishes that it is a valuable framework for social robotic applications, such as human robot collaboration or human-robot interaction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Real-life experiment metrics for evaluating human-robot collaborative navigation tasks

  • Repiso Polo, Ely
  • Garrell Zulueta, Anais
  • Sanfeliu Cortés, Alberto
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., As robots move from laboratories and industries to the real world, they must develop new abilities to collaborate with humans in various aspects, including human-robot collaborative navigation (HRCN) tasks. Then, it is required to develop general methodologies to evaluate these robots’ behaviors. These methodologies should incorporate objective and subjective measurements. Objective measurements for evaluating a robot’s behavior while navigating with others can be accomplished using social distances in conjunction with task characteristics, people-robot relationships, and physical space. Additionally, the objective evaluation of the task must consider human behavior, which is influenced by changes and the structure of their environment. Subjective evaluations of robot’s behaviors can be conducted using surveys that address various aspects of robot usability. This includes people’s perceptions of their interaction during their collaborative task with the robot, focusing on aspects such as sociability, comfort, and task-intelligence. Moreover, the communicative interaction between the agents (people and robots) involved in the collaborative task should also be evaluated. Therefore, this paper presents a comprehensive methodology for objectively and subjectively evaluating HRCN tasks., This work has been supported by the Artificial Intelligence for Human- Robot Interaction (AI4HRI) project ANR-20-IADJ-0006. Also, this Work has been supported under the EU project grant CANOPIES (H2020- ICT-2020-2-101016906) and by the JST Moonshot R D Grant Number JPMJMS2011. Ely Repiso was also supported during her PhD by Spanish Ministry of Science and Innovation under a FPI-grant (BES-2014-067713) and during her Post Doc by the project AI4HRI, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Body gesture recognition to control a social mobile robot

  • Laplaza Galindo, Javier
  • Romero Martín, Rut
  • Sanfeliu Cortés, Alberto
  • Garrell Zulueta, Anais
In this work, we propose a gesture-based language to allow humans to interact with robots using their body in a natural way. We have created a new gesture detection model using neural networks and a new dataset of humans making a collection of body gestures to train this architecture. Furthermore, we compare body gesture communication with other communication channels to demonstrate the importance of adding this knowledge to robots. The presented approach is validated in diverse simulations and real-life experiments with non-trained volunteers. This attains promising results and establishes that it is a valuable framework for social robotic applications, such as human robot collaboration or human-robot interaction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Body gestures recognition for social human-robot interaction

  • Laplaza Galindo, Javier
  • Sanfeliu Cortés, Alberto
  • Garrell Zulueta, Anais
In this paper, a solution for human gesture classification is proposed. The solution uses a Deep Learning model and is meant to be useful for non-verbal communication between humans and robots. The research focuses on the creation of the “temPoral bOdy geSTUre REcognition model” (POSTURE) that can recognise continuous gestures performed in real-life situations. The suggested model takes into account spatial and temporal components so as to achieve the recognition of more natural and intuitive gestures. In a first step, a framework extracts from all the images the corresponding landmarks for each of the body joints. Next, some data filtering techniques are applied with the aim of avoiding problems related with the data. And finally, different neural network configurations and approaches are tested to find the optimal performance. The validation of the model is accomplished throughout an extensive set of simulations and real-life experiments., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Exploring transformers and visual transformers for force prediction in human-robot collaborative transportation tasks

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we analyze the possibilities offered by Deep Learning State-of-the-Art architectures such as Transformers and Visual Transformers in generating a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We outperform our previous predictor by achieving a success rate of 93.8% in testset and 90.9% in real experiments with 21 volunteers predicting in both cases the force that the human will exert during the next 1 s. A modification in the architecture allows us to obtain a second output from the model with a velocity prediction, which allows us to improve the capabilities of our predictor if it is used to estimate the trajectory that the humanrobot pair will follow. An ablation test is also performed to verify the relative contribution to performance of each input., Work supported under the European project CANOPIES (H2020ICT-2020-2-101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Human-robot harvesting plan negotiation: perception, grape mapping and shared planning

  • Dalmasso Blanch, Marc
  • García Ruiz, Francisco
  • Ciarfuglia, Thomas A.
  • Saraceni, Leonardo
  • Motoi, Ionut M.
  • Gil Moya, Emilio
  • Sanfeliu Cortés, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper we present an application of plan negotiation for distributing work in Human-Robot Teams (HRT), engaging in a shared plan generation process. This model has been experimented over data collected from a real-life table grape vineyard field in Corsira (Aprilia, Italy) on the context of the EU CANOPIES project. In this work we describe a complete system up to negotiation agreement, the perception system to detect the grapes position and quality, the grape map generation process and the proposed plan negotiation approach, and protocol used to interactively generate harvest plans. Our proposed approach tackles harvest plan generation as a two step negotiation process on negotiation domains of increasing embedded information. The shared planning process was evaluated using both a subjective user study and teammate quantitative models., This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016906 – Project CANOPIES, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Context attention: human motion prediction using context information and deep learning attention models

  • Laplaza Galindo, Javier
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
This work proposes a human motion prediction model for handover operations. The model uses a multi-headed attention architecture to process the human skeleton data together with contextual data from the operation. This contextual data consists on the position of the robot’s End Effector (REE). The model input is a sequence of 5 seconds skeleton position and it outputs the predicted 2.5 future seconds position. We provide results of the human upper body and the human right hand or Human End Effector (HEE). The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained using OpenPose with an Intel RealSense D435i camera set inside the robot’s head. The results show a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the ROCOTRANSP project (PID2019-106702RB-C21/AEI/10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Context and intention for 3D human motion prediction: experimentation and user study in handover tasks

  • Laplaza Galindo, Javier
  • Garrell Zulueta, Anais
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work we present a novel attention deep learning model that uses context and human intention for 3D human body motion prediction in handover human-robot tasks. This model uses a multi-head attention architecture which incorporates as inputs the human motion, the robot end effector and the position of the obstacles. The outputs of the model are the predicted motion of the human body and the predicted human intention. We use this model to analyze a handover collaborative task with a robot where the robot is able to predict the future motion of the human and use this information in it’s planner. Several experiments are performed where human volunteers fill a standard poll to rate different features, taking into account when the robot uses the prediction versus when the robot doesn’t use the prediction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016- 0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI / 10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT2020-2-101016906)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Improving human-robot interaction effectiveness in human-robot collaborative object transportation using force prediction

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work, we analyse the use of a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We check that this force prediction can improve multiple parameters associated with effective Human-Robot Interaction (HRI) such as percep- tion of the robot’s contribution to the task, comfort or trust in the robot in a physical Human Robot Interaction (pHRI). We present a Deep Learning model that allows to predict the force that a human will exert in the next 1 s using as inputs the force previously exerted by the human, the robot’s velocity and environment information obtained from the robot’s LiDAR. Its success rate is up to 92.3% in testset and up to 89.1% in real experiments. We demonstrate that this force prediction, in addition to being able to be used directly to detect changes in the human’s intention, can be processed to obtain an estimate of the human’s desired trajectory. We have validated this approach with a user study involving 18 volunteers., Work supported under the European project CANOPIES (H2020 ICT-2020-2-101016906) and the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033) and by JST Moonshot R&D Grant Number JPMJMS2011.The first author acknowledges Spanish FPU grant with ref.FPU19/06582., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Inference vs. explicitness. Do we really need the perfect predictor? The human-robot collaborative object transportation case

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
When robots interact with humans, limitations in their internal models arise due to the uncertainty and even randomness of human behavior. This has led to attempts to predict human future actions and infer their intent. However, some authors argue for combining inference engines with communication systems that explicitly elicit human intention. This work builds on our Perception-Intention-Action (PIA) cycle, a framework that considers human intention at the same level as perception of the environment. The PIA cycle is used in a collaborative task to compare the effect on different human- robot interaction aspects of using a force predictor that infers human implicit intention versus a communication system that explicitly elicits human intention. A study with 18 volunteers shows that allowing humans to directly express themselves can achieve the same improvement as an intention predictor., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and by JST Moonshot R & D Grant Number JPMJMS2011. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Classification of humans social relations within urban areas

  • Castro Arcusa, Oscar
  • Repiso Polo, Ely
  • Garrell Zulueta, Anais
  • Sanfeliu Cortés, Alberto
This paper presents the design of deep learning architectures which allow to classify the social relationship existing between two people who are walking in a side-by-side formation into four possible categories --colleagues, couple, family or friendship. The models are developed using Neural Networks or Recurrent Neural Networks to achieve the classification and are trained and evaluated using a database obtained from humans walking together in an urban environment. The best achieved model accomplishes a good accuracy in the classification problem and its results enhance the outcomes from a previous study [1]. In addition, we have developed several models to classify the social interactions in two categories --“intimate" and "acquaintances", where the best model achieves a very good performance, and for a real robot this classification is enough to be able to customize its behavior to its users. Furthermore, the proposed models show their future potential to improve its efficiency and to be implemented in a real robot., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Force and velocity prediction in human-robot collaborative transportation tasks through video retentive networks

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, In this article, we propose a generalization of a Deep Learning State-of-the-Art architecture such as Retentive Networks so that it can accept video sequences as input. With this generalization, we design a force/velocity predictor applied to the medium-distance Human-Robot collaborative object transportation task. We achieve better results than with our previous predictor by reaching success rates in testset of up to 93.7% in predicting the force to be exerted by the human and up to 96.5% in the velocity of the human-robot pair during the next 1 s, and up to 91.0% and 95.0% respectively in real experiments. This new architecture also manages to improve inference times by up to 32.8% with different graphics cards. Finally, an ablation test allows us to detect that one of the input variables used so far, such as the position of the task goal, could be discarded allowing this goal to be chosen dynamically by the human instead of being pre-set., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Perception-intention-action cycle as a human acceptable way for improving human-robot collaborative tasks

  • Domínguez Vidal, José Enrique
  • Rodríguez Linares, Nicolás Adrián
  • Sanfeliu Cortés, Alberto
© ACM 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in HRI '23: Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, https://doi.org/10.1145/3568294.3580149., In Human-Robot Collaboration (HRC) tasks, the classical Perception-Action cycle can not fully explain the collaborative behaviour of the human-robot pair until it is extended to Perception-Intention-Action (PIA) cycle, giving to the human's intention a key role at the same level of the robot's perception and not as a subblock of this. Although part of the human's intention can be perceived or inferred by the other agent, this is prone to misunderstandings so the true intention has to be explicitly informed in some cases to fulfill the task. Here, we explore both types of intention and we combine them with the robot's perception through the concept of Situation Awareness (SA). We validate the PIA cycle and its acceptance by the user with a preliminary experiment in an object transportation task showing that its usage can increase trust in the robot., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Human motion trajectory prediction using the social force model for real-time and low computational cost applications

  • Gil Viyuela, Óscar
  • Sanfeliu Cortés, Alberto
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG, Human motion trajectory prediction is a very important functionality for human-robot collaboration, specifically in accompanying, guiding, or approaching tasks, but also in social robotics, self-driving vehicles, or security systems. In this paper, a novel trajectory prediction model, Social Force Generative Adversarial Network (SoFGAN), is proposed. SoFGAN uses a Generative Adversarial Network (GAN) and Social Force Model (SFM) to generate different plausible people trajectories reducing collisions in a scene. Furthermore, a Conditional Variational Autoencoder (CVAE) module is added to emphasize the destination learning. We show that our method is more accurate in making predictions in UCY or BIWI datasets than most of the current state-of-the-art models and also reduces collisions in comparison to other approaches. Through real-life experiments, we demonstrate that the model can be used in real-time without GPU’s to perform good quality predictions with a low computational cost., Ó. Gil—Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and JST Moonshot R & D Grant Number JPMJMS2011-85., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Attention deep learning based model for predicting the 3D Human Body Pose using the Robot Human Handover Phases

  • Laplaza Galindo, Javier
  • Pumarola Peris, Albert
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This work proposes a human motion prediction model for handover operations. We use in this work, the different phases of the handover operation to improve the human motion predictions. Our attention deep learning based model takes into account the position of the robot’s End Effector and the phase in the handover operation to predict future human poses. Our model outputs a distribution of possible positions rather than one deterministic position, a key feature in order to allow robots to collaborate with humans. The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained with an Intel RealSense D435i camera attached inside the robot’s head. The results shown a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906), Peer Reviewed, Postprint (author's final draft)