Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 73
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

Working with troubles and failures in conversation between humans and robots: workshop report

  • Förster, Frank
  • Wiltschko, Martina
  • Kapetanios, Epaminondas
This paper summarizes the structure and findings from the first Workshop on Troubles and Failures in Conversations between Humans and Robots. The workshop was organized to bring together a small, interdisciplinary group of researchers working on miscommunication from two complementary perspectives. One group of technology-oriented researchers was made up of roboticists, Human-Robot Interaction (HRI) researchers and dialogue system experts. The second group involved experts from conversation analysis, cognitive science, and linguistics. Uniting both groups of researchers is the belief that communication failures between humans and machines need to be taken seriously and that a systematic analysis of such failures may open fruitful avenues in research beyond current practices to improve such systems, including both speech-centric and multimodal interfaces. This workshop represents a starting point for this endeavour. The aim of the workshop was threefold: Firstly, to establish an interdisciplinary network of researchers that share a common interest in investigating communicative failures with a particular view towards robotic speech interfaces; secondly, to gain a partial overview of the 'failure landscape' as experienced by roboticists and HRI researchers; and thirdly, to determine the potential for creating a robotic benchmark scenario for testing future speech interfaces with respect to the identified failures. The present article summarizes both the 'failure landscape' surveyed during the workshop as well as the outcomes of the attempt to define a benchmark scenario., The workshop, the outcomes of which are described in this paper, was funded by the UK Engineering and Physical Science Research Council (EPSRC) Robotics & Autonomous Systems Network (UK-RAS) Pump Priming programme under the project title 'Charting the Limits and Developing Future Directions of Speech Interfaces for Robotics'. DG is supported under the EPSRC projects NLG for low-resource domains (EP/T024917/1) and CiViL (EP/T014598/1). Some of the authors are supported by the Engineering and Physical Sciences Research Council [grant numbers EP/V00784X/1, EP/X009343/1] including through the Trustworthy Autonomous Systems (TAS) Hub. One of the authors has been supported by the H2020 EU projects CANOPIES'A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems, Grant Agreement 101016906. DK is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy-EXC 2002/1 'Science of Intelligence'-project number 390523135.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Social robot navigation tasks: Combining machine learning techniques and social force model

  • Gil Viyuela, Óscar
  • Garrell, Anaís
  • Sanfeliu, Alberto
Social robot navigation in public spaces, buildings or private houses is a difficult problem that is not well solved due to environmental constraints (buildings, static objects etc.), pedestrians and other mobile vehicles. Moreover, robots have to move in a human-aware manner—that is, robots have to navigate in such a way that people feel safe and comfortable. In this work, we present two navigation tasks, social robot navigation and robot accompaniment, which combine machine learning techniques with the Social Force Model (SFM) allowing human-aware social navigation. The robots in both approaches use data from different sensors to capture the environment knowledge as well as information from pedestrian motion. The two navigation tasks make use of the SFM, which is a general framework in which human motion behaviors can be expressed through a set of functions depending on the pedestrians’ relative and absolute positions and velocities. Additionally, in both social navigation tasks, the robot’s motion behavior is learned using machine learning techniques: in the first case using supervised deep learning techniques and, in the second case, using Reinforcement Learning (RL). The machine learning techniques are combined with the SFM to create navigation models that behave in a social manner when the robot is navigating in an environment with pedestrians or accompanying a person. The validation of the systems was performed with a large set of simulations and real-life experiments with a new humanoid robot denominated IVO and with an aerial robot. The experiments show that the combination of SFM and machine learning can solve human-aware robot navigation in complex dynamic environments., This research was supported by the grant MDM-2016-0656 funded by MCIN/AEI/ 10.13039/501100011033, the grant ROCOTRANSP PID2019-106702RB-C21 funded by MCIN/AEI/ 10.13039/501100011033 and the grant CANOPIES H2020-ICT-2020-2-101016906 funded by the European Union.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Event-Based Line SLAM in Real-Time

  • Chamorro Hernández, William Oswaldo
  • Solà, Joan
  • Andrade-Cetto, Juan
Event-based cameras generate asynchronous streams of events, triggered proportionally to the logarithmic change of brightness in the scene. These cameras have very low latency and high dynamic range suitable to address challenging motion scenarios in robotics. In this work, we explore a new event-based line-SLAM approach following a parallel tracking and mapping philosophy. Our fast tracking algorithm, produces accurate camera pose estimates at a high rate by minimizing the event-line reprojection error with an error-state Kalman filter formulated entirely with Lie theory. The mapping thread leverages the natural edge highlighting strength of events to recover and optimize straight lines in human-made scenarios. The proper manipulation of matrix sparsity as well as the information sharing between tracking and mapping nodes allow us to achieve real-time performance on a standard multi-core CPU. This system was tested on several scenarios rich in straight edge objects, and compared against, ground truth and frame and event based state-of-the-art approaches., This work was supported by the EU H2020 projects TERRINET INFRAI-2017-1-730994 and CANOPIES ICT-2020-2-101016906, by project EBCON PID2020- 119244GB-I00 funded by MCIN/AEI/10.13039/501100011033, and by a scholarship from SENESCYT, Republic of Ecuador to WCH.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

ACHORD: Communication-Aware Multi-Robot Coordination with Intermittent Connectivity

  • Saboia, María
  • Clark, Lillian
  • Thangavelu, Vivek
  • Edlund, Jeffrey A.
  • Otsu, Kyohei
  • Correa, Gustavo J.
  • Varadharajan, Vivek Shankar
  • Santamaria-Navarro, Àngel
  • Touma, Thomas
  • Bouman, Amanda
  • Melikyan, Hovhannes
  • Pailevanian, Torkom
  • Kim, Sung-Kyun
  • Archanian, Avak
  • Stegun Vaquero, Tiago
  • Beltrame, Giovanni
  • Napp, Nils
  • Pessin, Gustavo
  • Agha-Mohammadi, Ali Akbar
Communication is an important capability for multi-robot exploration because (1) inter-robot communication (comms) improves coverage efficiency and (2) robot-to-base comms improves situational awareness. Exploring comms-restricted (e.g., subterranean) environments requires a multi-robot system to tolerate and anticipate intermittent connectivity, and to carefully consider comms requirements, otherwise mission-critical data may be lost. In this paper, we describe and analyze ACHORD (Autonomous & Collaborative High-Bandwidth Operations with Radio Droppables), a multi-layer networking solution which tightly co-designs the network architecture and high-level decision-making for improved comms. ACHORD provides bandwidth prioritization and timely and reliable data transfer despite intermittent connectivity. Furthermore, it exposes low-layer networking metrics to the application layer to enable robots to autonomously monitor, map, and extend the network via droppable radios, as well as restore connectivity to improve collaborative exploration. We evaluate our solution with respect to the comms performance in several challenging underground environments including the DARPA SubT Finals competition environment. Our findings support the use of data stratification and flow control to improve bandwidth-usage., This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration (80NM0018D0004). This work was supported in part by FUNDEP, by NASA Space Technology Research Fellowship Grant No. 80NSSC19K1189, and by NSF Grant No. #1846340. Further, it was partially supported by the Spanish Ministry of Science and Innovation under the project EB-CON (PID2020-119244GB-I00, MCIN/AEI/10.13039/501100011033) and by the European Commission under the project CANOPIES (H2020-ICT-2020- 2-101016906).
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Classification of humans social relations within urban areas

  • Castro Arcusa, Oscar
  • Repiso, Ely
  • Garrell, Anaís
  • Sanfeliu, Alberto
Trabajo presentado en el ROBOT2022: Fifth Iberian Robotics Conference, celebrada en Zaragoza (España), del 20 al 22 de noviembre de 2022, This paper presents the design of deep learning architectures which allow to classify the social relationship existing between two people who are walking in a side-by-side formation into four possible categories --colleagues, couple, family or friendship. The models are developed using Neural Networks or Recurrent Neural Networks to achieve the classification and are trained and evaluated using a database obtained from humans walking together in an urban environment. The best achieved model accomplishes a good accuracy in the classification problem and its results enhance the outcomes from a previous study [1]. In addition, we have developed several models to classify the social interactions in two categories --¿intimate" and "acquaintances", where the best model achieves a very good performance, and for a real robot this classification is enough to be able to customize its behavior to its users. Furthermore, the proposed models show their future potential to improve its efficiency and to be implemented in a real robot., This work has been supported by the Artificial Intelligence for Human–Robot Interaction (AI4HRI) project ANR-20-IADJ-0006. Also, this Work has been supported under the Spanish State Research Agency through the ROCOTRANSP project (PID2019-106702RB-C21/AEI/10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906).
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Context and Intention for 3D Human Motion Prediction: Experimentation and User study in Handover Tasks

  • Laplaza Galindo, Javier
  • Garrell, Anaís
  • Moreno-Noguer, Francesc
  • Sanfeliu, Alberto
Trabajo presentado en el 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), celebrado en Nápoles (Italia), del 29 de agosto al 2 de septiembre de 2022, In this work we present a novel attention deep learning model that uses context and human intention for 3D human body motion prediction in handover human-robot tasks. This model uses a multi-head attention architecture which incorporates as inputs the human motion, the robot end effector and the position of the obstacles. The outputs of the model are the predicted motion of the human body and the predicted human intention. We use this model to analyze a handover collaborative task with a robot where the robot is able to predict the future motion of the human and use this information in it¿s planner. Several experiments are performed where human volunteers fill a standard poll to rate different features, taking into account when the robot uses the prediction versus when the robot doesn¿t use the prediction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016- 0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI / 10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT- 2020-2-101016906)
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Weakly and semi-supervised detection, segmentation and tracking of table grapes with limited and noisy data

  • Ciarfuglia, Thomas A.
  • Motoi, Ionut M.
  • Saracen, Leonardo
  • Fawakherji, Mulham
  • Sanfeliu, Alberto
  • Nardi, Daniele
Detection, segmentation and tracking of fruits and vegetables are three fundamental tasks for precision agriculture, enabling robotic harvesting and yield estimation applications. However, modern algorithms are data hungry and it is not always possible to gather enough data to apply the best performing supervised approaches. Since data collection is an expensive and cumbersome task, the enabling technologies for using computer vision in agriculture are often out of reach for small businesses. Following previous work in this context (Ciarfuglia et al., 2022), where we proposed an initial weakly supervised solution to reduce the data needed to get state-of-the-art detection and segmentation in precision agriculture applications, here we improve that system and explore the problem of tracking fruits in orchards. We present the case of vineyards of table grapes in southern Lazio (Italy) since grapes are a difficult fruit to segment due to occlusion, colour and general illumination conditions. We consider the case in which there is some initial labelled data that could work as source data (e.g. wine grape data), but it is considerably different from the target data (e.g. table grape data). To improve detection and segmentation on the target data, we propose to train the segmentation algorithm with a weak bounding box label, while for tracking we leverage 3D Structure from Motion algorithms to generate new labels from already labelled samples. Finally, the two systems are combined in a full semi-supervised approach. Comparisons with state-of-the-art supervised solutions show how our methods are able to train new models that achieve high performances with few labelled images and with very simple labelling., This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016906 – Project CANOPIES, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Accurate detection and depth estimation of table grapes and peduncles for robot harvesting, combining monocular depth estimation and CNN methods

  • Coll-Ribes, Gabriel
  • Torres-Rodríguez, Iván J.
  • Grau Saldes, Antoni
  • Guerra, Edmundo
  • Sanfeliu, Alberto
Precision agriculture is a growing field in the agricultural industry and it holds great potential in fruit and vegetable harvesting. In this work, we present a robust accurate method for the detection and localization of the peduncle of table grapes, with direct implementation in automatic grape harvesting with robots. The bunch and peduncle detection methods presented in this work rely on a combination of instance segmentation and monocular depth estimation using Convolutional Neural Networks (CNN). Regarding depth estimation, we propose a combination of different depth techniques that allow precise localization of the peduncle using traditional stereo cameras, even with the particular complexity of grape peduncles. The methods proposed in this work have been tested on the WGISD (Embrapa Wine Grape Instance Segmentation) dataset, improving the results of state-of-the-art techniques. Furthermore, within the context of the EU project CANOPIES, the methods have also been tested on a dataset of 1,326 RGB-D images of table grapes, recorded at the Corsira Agricultural Cooperative Society (Aprilia, Italy), using a Realsense D435i camera located at the arm of a CANOPIES two-manipulator robot developed in the project. The detection results on the WGISD dataset show that the use of RGB-D information (mAP=0.949) leads to superior performance compared to the use of RGB data alone (mAP=0.891). This trend is also evident in the CANOPIES Grape Bunch and Peduncle dataset, where the mAP for RGB-D images (mAP=0.767) outperforms that of RGB data (mAP=0.725). Regarding depth estimation, our method achieves a mean squared error of 2.66 cm within a distance of 1 m in the CANOPIES dataset., Work supported under the European project CANOPIES with the grant (H2020- ICT-2020-2-101016906). The authors want to acknowledge the help and support of the Sapienza Universitá di Roma and the Universitá degli Studi Roma Tre researchers of the CANOPIES project in the data collection of the grapes images and ground truth data., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Body Gesture Recognition to Control a Social Mobile Robot

  • Laplaza Galindo, Javier
  • Romero, Ramón
  • Sanfeliu, Alberto
  • Garrell, Anaís
Trabajo presentado en el HRI ’23 - ACM/IEEE International Conference on Human-Robot Interaction, celebrado en Estocolmo (Suecia), en marzo de 2023, In this work, we propose a gesture-based language to allow humans to interact with robots using their body in a natural way. We have created a new gesture detection model using neural networks and a new dataset of humans making a collection of body gestures to train this architecture. Furthermore, we compare body gesture communication with other communication channels to demonstrate the importance of adding this knowledge to robots. The presented approach is validated in diverse simulations and real-life experiments with non-trained volunteers. This attains promising results and establishes that it is a valuable framework for social robotic applications, such as human robot collaboration or human-robot interaction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Real-Life Experiment Metrics for Evaluating Human-Robot Collaborative Navigation Tasks

  • Repiso, Ely
  • Garrell, Anaís
  • Sanfeliu, Alberto
Trabajo presentado en el 32nd IEEE International Conference on Robot and Human Interactive Communication, celebrado en Busan (Corea del Sur), del 28 al 31 de agosto de 2023, As robots move from laboratories and industries to the real world, they must develop new abilities to collaborate with humans in various aspects, including human-robot collaborative navigation (HRCN) tasks. Then, it is required to develop general methodologies to evaluate these robots' behaviors. These methodologies should incorporate objective and subjective measurements. Objective measurements for evaluating a robot's behavior while navigating with others can be accomplished using social distances in conjunction with task characteristics, people-robot relationships, and physical space. Additionally, the objective evaluation of the task must consider human behavior, which is influenced by changes and the structure of their environment. Subjective evaluations of robot's behaviors can be conducted using surveys that address various aspects of robot usability. This includes people's perceptions of their interaction during their collaborative task with the robot, focusing on aspects such as sociability, comfort, and task-intelligence. Moreover, the communicative interaction between the agents (people and robots) involved in the collaborative task should also be evaluated. Therefore, this paper presents a comprehensive methodology for objectively and subjectively evaluating HRCN tasks., This work has been supported by the Artificial Intelligence for HumanRobot Interaction (AI4HRI) project ANR-20-IADJ-0006. Also, this Work has been supported under the EU project grant CANOPIES (H2020- ICT-2020-2-101016906) and by the JST Moonshot R D Grant Number JPMJMS2011. Ely Repiso was also supported during her PhD by Spanish Ministry of Science and Innovation under a FPI-grant (BES-2014-067713) and during her Post Doc by the project AI4HRI, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Improving Human-Robot Interaction Effectiveness in Human-Robot Collaborative Object Transportation Using Force Prediction

  • Domínguez Vidal, José Enrique
  • Sanfeliu, Alberto
Trabajo presentado en la International Conference on Intelligent Robots and Systems (IROS), celebrada en Detroit, MI (Estados Unidos), del 1 al 5 de octubre de 2023, In this work, we analyse the use of a prediction of the human's force in a Human-Robot collaborative object transportation task at a middle distance. We check that this force prediction can improve multiple parameters associated with effective Human-Robot Interaction (HRI) such as perception of the robot's contribution to the task, comfort or trust in the robot in a physical Human Robot Interaction (pHRI). We present a Deep Learning model that allows to predict the force that a human will exert in the next 1 s using as inputs the force previously exerted by the human, the robot's velocity and environment information obtained from the robot's LiDAR. Its success rate is up to 92.3% in testset and up to 89.1 % in real experiments. We demonstrate that this force prediction, in addition to being able to be used directly to detect changes in the human's intention, can be processed to obtain an estimate of the human's desired trajectory. We have validated this approach with a user study involving 18 volunteers., Work supported under the European project CANOPIES (H2020- ICT-2020-2-101016906) and the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033) and by JST Moonshot R & D Grant Number JPMJMS2011. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

A Survey on Socially Aware Robot Navigation: Taxonomy and Future Challenges

  • Teja Singamaneni, Phani
  • Bachiller-Burgos, Pilar
  • Manso, Luis J.
  • Garrell, Anaís
  • Sanfeliu, Alberto
  • Spalanzani, Anne
  • Alami, Rachid
Socially aware robot navigation is gaining popularity with the increase in delivery and assistive robots. The research is further fueled by a need for socially aware navigation skills in autonomous vehicles to move safely and appropriately in spaces shared with humans. Although most of these are ground robots, drones are also entering the field. In this paper, we present a literature survey of the works on socially aware robot navigation in the past 10 years. We propose four different faceted taxonomies to navigate the literature and examine the field from four different perspectives. Through the taxonomic review, we discuss the current research directions and the extending scope of applications in various domains. Further, we put forward a list of current research opportunities and present a discussion on possible future challenges that are likely to emerge in the field., This work was supported by the Horizon Europe Framework Programme [Grant: euROBIN 101070596]; Agence Nationale de la Recherche [Grant: ANITI ANR-19-PI3A-0004]; European project CANOPIES [Grant: H2020-ICT-2020-2-101016906]; Spanish Government (MCIN/AEI) and European Union (“NextGenerationEU”/PRTR) [Grant: TED2021-131739-C22]; Spanish Government (MCIN) [Grant: PDC2022-133597-C41]; Spanish Government (MCIN) [Grant: PID2022-142039NA-I00]., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Human-Robot Collaborative Minimum Time Search through Sub-priors in Ant Colony Optimization

  • Gil Viyuela, Óscar
  • Sanfeliu, Alberto
Human-Robot Collaboration (HRC) has evolved into a highly promising issue owing to the latest breakthroughs in Artificial Intelligence (AI) and Human-Robot Interaction (HRI), among other reasons. This emerging growth increases the need to design multi-agent algorithms that can manage also human preferences. This paper presents an extension of the Ant Colony Optimization (ACO) meta-heuristic to solve the Minimum Time Search (MTS) task, in the case where humans and robots perform an object searching task together. The proposed model consists of two main blocks. The first one is a convolutional neural network (CNN) that provides the prior probabilities about where an object may be from a segmented image. The second one is the Sub-prior MTS-ACO algorithm (SP-MTS-ACO), which takes as inputs the prior probabilities and the particular search preferences of the agents in different sub-priors to generate search plans for all agents. The model has been tested in real experiments for the joint search of an object through a Vizanti web-based visualization in a tablet computer. The designed interface allows the communication between a human and our humanoid robot named IVO. The obtained results show an improvement in the search perception of the users without loss of efficiency., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and JST Moonshot R & D Grant Number JPMJMS2011-85., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Research data Set de datos (Dataset)

Data for publication. CANOPIES Grape bunch and peduncle dataset

  • Sanfeliu, Alberto
In September 2021, data of grape bunches and peduncles were collected in the vineyard of the Corsira Agricultural Cooperative Society (Aprilia, Italy) (Via Corsira (https://goo.gl/maps/UdqVTNQnJTGzRnY39), where table grapes are grown. A total of 45 sequences were recorded, with a total duration of 3,660 seconds (61 minutes) and containing a total of 103,775 images and 80,489 point clouds. The images were acquired on different days and at different times, from 9am to 5pm. They had different lighting conditions due to weather conditions, density of foliage, or time of day. The images were also taken at different distances from the bunches, ranging from 30cm to 2m. From the data recorded, ten sequences were selected in which the peduncles and bunches were fully or partially visible, discarding those taken from too far away to identify the peduncle. The CANOPIES Grape Bunch and Peduncle (GBPD) dataset contains a total of 810 RGB images and with their correspondent binary masks for grape bunches and peduncles. The images were recorded using an Intel RealSense Depth Camera D435. Both 640x480 and 1280x720 resolutions were used. The dataset was labelled using VGG Image Annotator (VIA) [1], a free software that can be used without any installation. The labelling work was divided among 8 people. Each of them was given strict guidelines on how to label the images in order to keep the labelling consistent. These data were used for accurate detection of table grapes and peduncles for harvesting which methods can be seen in the article [2]., European Commission CANOPIES – A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems 101016906, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Perception–Intention–Action Cycle in Human–Robot Collaborative Tasks: The Collaborative Lightweight Object Transportation Use-Case

  • Domínguez Vidal, José Enrique
  • Rodríguez Linares, Nicolás Adrián
  • Sanfeliu, Alberto
This study proposes to improve the reliability, robustness and human-like nature of Human–Robot Collaboration (HRC). For that, the classical Perception–Action cycle is extended to a Perception–Intention–Action (PIA) cycle, which includes an Intention stage at the same level as the Perception one, being in charge of obtaining both the implicit and the explicit intention of the human, opposing to classical approaches based on inferring everything from perception. This complete cycle is exposed theoretically including its use of the concept of Situation Awareness, which is shown as a key element for the correct understanding of the current situation and future action prediction. This enables the assignment of roles to the agents involved in a collaborative task and the building of collaborative plans. To visualize the cycle, a collaborative transportation task is used as a use-case. A force-based model is designed to combine the robot’s perception of its environment with the force exerted by the human and other factors in an illustrative way. Finally, a total of 58 volunteers participate in two rounds of experiments. In these, it is shown that the human agrees to explicitly state their intention without undue extra effort and that the human understands that this helps to minimize robot errors or misunderstandings. It is also shown that a system that correctly combines inference with explicit elicitation of the human’s intention is the best rated by the human on multiple parameters related to effective Human–Robot Interaction (HRI), such as perceived safety or trust in the robot., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020- ICT-2020-2-101016906), the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 MCIN/ AEI /10.13039/501100011033) and the JST Moonshot R & D Grant No. JPMJMS2011-85. The first author acknowledges Spanish FPU Grant with ref. FPU19/06582., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Anticipation and Proactivity. Unraveling Both Concepts in Human-Robot Interaction through a Handover Example

  • Domínguez Vidal, José Enrique
  • Sanfeliu, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Trabajo presentado en 33rd IEEE International Conference on Robot and Human Interactive Communication, celebrada en Pasadena, California, USA, del 26 al 30 de agosto de 2024, While robots have advanced in understanding their environments, collaborative tasks demand a deeper comprehension of human intentions to mitigate uncertainty. Anticipatory and proactive behaviours are pivotal in enhancing Human-Robot Interactions (HRI), yet literature often conflates these terms. This study elucidates the distinction between anticipation and proactivity, offering clear definitions and exemplifying their implications through a handover scenario. Through a user study with 24 volunteers performing a total of 72 experiments, we have found that humans are able to distinguish both behaviours and that there is a statistically significant increase in the anthropomorphism of the robot when it behaves proactively. Additionally, both anticipation and proactivity show statistically significant increases in multiple aspects of effective HRI (fluency, comfort, performance, etc.). However, no clear preference for either has been detected., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and by JST Moonshot R & D Grant Number JPMJMS2011. The first author acknowledges Spanish FPU grant with ref. FPU19/06582, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Exploring Preferences in Human-Robot Navigation Plan Proposal Representation

  • Dalmasso, Marc
  • Sanchez-Anguix, Víctor
  • Garrell, Anaís
  • Jiménez, Pablo
  • Sanfeliu, Alberto
In this paper, we present our focus on unraveling the intricacies of plan negotiation in human-robot collaborative navigation (HRCN) through a comprehensive exploration of human preferences over robot proposals in search tasks. Via online survey data, we explore the multidimensional landscape of diverse plan representations, negotiation contexts and negotiation domains. Our study seeks to identify the crucial factors that exert a significant influence over human perception, shedding light on the dynamic interplay between humans and robots and contributing valuable insights to advance the understanding of effective navigation plan negotiation strategies in human-robot teams (HRT)., This work was supported by JST Moonshot R & D Grant Number JPMJMS2011-85 and the European project CANOPIES with grant H2020-ICT-2020-2-101016906, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Exploring Transformers and Visual Transformers for Force Prediction in Human-Robot Collaborative Transportation Tasks

  • Domínguez Vidal, José Enrique
  • Sanfeliu, Alberto
In this paper, we analyze the possibilities offered by Deep Learning State-of-the-Art architectures such as Transformers and Visual Transformers in generating a prediction of the human's force in a Human-Robot collaborative object transportation task at a middle distance. We outperform our previous predictor by achieving a success rate of 93.8% in testset and 90.9% in real experiments with 21 volunteers predicting in both cases the force that the human will exert during the next 1 s. A modification in the architecture allows us to obtain a second output from the model with a velocity prediction, which allows us to improve the capabilities of our predictor if it is used to estimate the trajectory that the human-robot pair will follow. An ablation test is also performed to verify the relative contribution to performance of each input., Work supported under the European project CANOPIES (H2020- ICT-2020-2-101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Force and Velocity Prediction in Human-Robot Collaborative Transportation Tasks through Video Retentive Networks

  • Domínguez Vidal, José Enrique
  • Sanfeliu, Alberto
In this article, we propose a generalization of a Deep Learning State-of-the-Art architecture such as Retentive Networks so that it can accept video sequences as input. With this generalization, we design a force/velocity predictor applied to the medium-distance Human-Robot collaborative object transportation task. We achieve better results than with our previous predictor by reaching success rates in testset of up to 93.7% in predicting the force to be exerted by the human and up to 96.5% in the velocity of the human-robot pair during the next 1 s, and up to 91.0% and 95.0% respectively in real experiments. This new architecture also manages to improve inference times by up to 32.8% with different graphics cards. Finally, an ablation test allows us to detect that one of the input variables used so far, such as the position of the task goal, could be discarded allowing this goal to be chosen dynamically by the human instead of being pre-set., Work supported under the European project CANOPIES (H2020- ICT-2020-2-101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Perception-Driven Shared Control Architecture for Agricultural Robots Performing Harvesting Tasks

  • Palmieri, Jozsef
  • Di Lillo, Paolo
  • Sanfeliu, Alberto
  • Marino, Alessandro
This paper introduces a shared control framework designed specifically for agricultural mobile manipulators engaged in harvesting operations. The shared control strategy allows for achieving such operations by dynamically exchanging the control between the robotic system and a human operator depending on the uncertainty in the environment perception. For this purpose, the robot's behavior is dynamically adapted to switch between two control modes with a different level of autonomy of the robot. The level of autonomy is encoded in two different admittance behaviors which are included in a first-order Hierarchical Quadratic Programming (HQP) control framework, that allows the robot to simultaneously address other control objectives at the same time. Experimental results with a dual-arm mobile robot, developed as part of the EU-funded CANOPIES project, demonstrate the effectiveness of the proposed method in real conditions., This work was supported by H2020-ICT project CANOPIES-A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems (Grant Agreement N. 101016906)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

The human intention. A taxonomy attempt and its applications to robotics

  • Domínguez Vidal, José Enrique
  • Sanfeliu, Alberto
Original version submitted to the International Journal of Social Robotics. Final version available on the SORO website, Despite a surge in robotics research dedicated to inferring and understanding human intent, a universally accepted definition remains elusive since existing works often equate human intention with specific task-related goals. This article seeks to address this gap by examining the multifaceted nature of intention. Drawing on insights from psychology, it attempts to consolidate a definition of intention into a comprehensible framework for a broader audience. The article classifies different types of intention based on psychological and communication studies, offering guidance to researchers shifting from pure technical enhancements to a more human-centric perspective in robotics. It then demonstrates how various robotics studies can be aligned with these intention categories. Finally, through in-depth analyses of collaborative search and object transport use cases, the article underscores the significance of considering the diverse facets of human intention., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020-ICT-2020–2–101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011–85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

When the Inference Meets the Explicitness or Why Multimodality Can Make Us Forget About the Perfect Predictor

  • Domínguez Vidal, José Enrique
  • Sanfeliu, Alberto
Original version submitted to the International Journal of Social Robotics. Final version available on the SORO website, Although in the literature it is common to find predictors and inference systems that try to predict human intentions, the uncertainty of these models due to the randomness of human behavior has led some authors to start advocating the use of communication systems that explicitly elicit human intention. In this work, it is analyzed the use of four different communication systems with a human-robot collaborative object transportation task as experimental testbed: two intention predictors (one based on force prediction and another with an enhanced velocity prediction algorithm) and two explicit communication methods (a button interface and a voice-command recognition system). These systems were integrated into IVO, a custom mobile social robot equipped with force sensor to detect the force exchange between both agents and LiDAR to detect the environment. The collaborative task required transporting an object over a 5-7 meter distance with obstacles in the middle, demanding rapid decisions and precise physical coordination. 75 volunteers perform a total of 255 executions divided into three groups, testing inference systems in the first round, communication systems in the second, and the combined strategies in the third. The results show that, 1) once sufficient performance is achieved, the human no longer notices and positively assesses technical improvements; 2) the human prefers systems that are more natural to them even though they have higher failure rates; and 3) the preferred option is the right combination of both systems., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020- ICT-2020–2-101016906) and the JST Moonshot R&D Grant Number JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Perception-intention-action cycle as a human acceptable way for improving human-robot collaborative tasks

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Rodríguez Linares, Nicolás Adrián|||0000-0003-4834-223X
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© ACM 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in HRI '23: Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, https://doi.org/10.1145/3568294.3580149., In Human-Robot Collaboration (HRC) tasks, the classical Perception-Action cycle can not fully explain the collaborative behaviour of the human-robot pair until it is extended to Perception-Intention-Action (PIA) cycle, giving to the human's intention a key role at the same level of the robot's perception and not as a subblock of this. Although part of the human's intention can be perceived or inferred by the other agent, this is prone to misunderstandings so the true intention has to be explicitly informed in some cases to fulfill the task. Here, we explore both types of intention and we combine them with the robot's perception through the concept of Situation Awareness (SA). We validate the PIA cycle and its acceptance by the user with a preliminary experiment in an object transportation task showing that its usage can increase trust in the robot., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Weakly and semi-supervised detection, segmentation and tracking of table grapes with limited and noisy data

  • Ciarfuglia, Thomas A.
  • Motoi, Ionut M.
  • Saraceni, Leonardo
  • Fawakherji, Mulham
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Nardi, Daniele
Detection, segmentation and tracking of fruits and vegetables are three fundamental tasks for precision agriculture, enabling robotic harvesting and yield estimation applications. However, modern algorithms are data hungry and it is not always possible to gather enough data to apply the best performing supervised approaches. Since data collection is an expensive and cumbersome task, the enabling technologies for using computer vision in agriculture are often out of reach for small businesses. Following previous work in this context (Ciarfuglia et al., 2022), where we proposed an initial weakly supervised solution to reduce the data needed to get state-of-the-art detection and segmentation in precision agriculture applications, here we improve that system and explore the problem of tracking fruits in orchards. We present the case of vineyards of table grapes in southern Lazio (Italy) since grapes are a difficult fruit to segment due to occlusion, colour and general illumination conditions. We consider the case in which there is some initial labelled data that could work as source data (e.g. wine grape data), but it is considerably different from the target data (e.g. table grape data). To improve detection and segmentation on the target data, we propose to train the segmentation algorithm with a weak bounding box label, while for tracking we leverage 3D Structure from Motion algorithms to generate new labels from already labelled samples. Finally, the two systems are combined in a full semi-supervised approach. Comparisons with state-of-the-art supervised solutions show how our methods are able to train new models that achieve high performances with few labelled images and with very simple labelling., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Perception–intention–action cycle in human–robot collaborative tasks: the collaborative lightweight object transportation use-case

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Rodríguez Linares, Nicolás Adrián|||0000-0003-4834-223X
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
This study proposes to improve the reliability, robustness and human-like nature of Human–Robot Collaboration (HRC). For that, the classical Perception–Action cycle is extended to a Perception–Intention–Action (PIA) cycle, which includes an Intention stage at the same level as the Perception one, being in charge of obtaining both the implicit and the explicit intention of the human, opposing to classical approaches based on inferring everything from perception. This complete cycle is exposed theoretically including its use of the concept of Situation Awareness, which is shown as a key element for the correct understanding of the current situation and future action prediction. This enables the assignment of roles to the agents involved in a collaborative task and the building of collaborative plans. To visualize the cycle, a collaborative transportation task is used as a use-case. A force-based model is designed to combine the robot’s perception of its environment with the force exerted by the human and other factors in an illustrative way. Finally, a total of 58 volunteers participate in two rounds of experiments. In these, it is shown that the human agrees to explicitly state their intention without undue extra effort and that the human understands that this helps to minimize robot errors or misunderstandings. It is also shown that a system that correctly combines inference with explicit elicitation of the human’s intention is the best rated by the human on multiple parameters related to effective Human–Robot Interaction (HRI), such as perceived safety or trust in the robot., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020- ICT-2020-2-101016906), the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 MCIN/ AEI /10.13039/501100011033) and the JST Moonshot R & D Grant No. JPMJMS2011-85. The first author acknowledges Spanish FPU Grant with ref. FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Body gestures recognition for social human-robot interaction

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
In this paper, a solution for human gesture classification is proposed. The solution uses a Deep Learning model and is meant to be useful for non-verbal communication between humans and robots. The research focuses on the creation of the “temPoral bOdy geSTUre REcognition model” (POSTURE) that can recognise continuous gestures performed in real-life situations. The suggested model takes into account spatial and temporal components so as to achieve the recognition of more natural and intuitive gestures. In a first step, a framework extracts from all the images the corresponding landmarks for each of the body joints. Next, some data filtering techniques are applied with the aim of avoiding problems related with the data. And finally, different neural network configurations and approaches are tested to find the optimal performance. The validation of the model is accomplished throughout an extensive set of simulations and real-life experiments., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Social robot navigation tasks: combining machine learning techniques and social force model

  • Gil Viyuela, Óscar|||0000-0001-8812-3168
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2021 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), Social robot navigation in public spaces, buildings or private houses is a difficult problem that is not well solved due to environmental constraints (buildings, static objects etc.), pedestrians and other mobile vehicles. Moreover, robots have to move in a human-aware manner—that is, robots have to navigate in such a way that people feel safe and comfortable. In this work, we present two navigation tasks, social robot navigation and robot accompaniment, which combine machine learning techniques with the Social Force Model (SFM) allowing human-aware social navigation. The robots in both approaches use data from different sensors to capture the environment knowledge as well as information from pedestrian motion. The two navigation tasks make use of the SFM, which is a general framework in which human motion behaviors can be expressed through a set of functions depending on the pedestrians’ relative and absolute positions and velocities. Additionally, in both social navigation tasks, the robot’s motion behavior is learned using machine learning techniques: in the first case using supervised deep learning techniques and, in the second case, using Reinforcement Learning (RL). The machine learning techniques are combined with the SFM to create navigation models that behave in a social manner when the robot is navigating in an environment with pedestrians or accompanying a person. The validation of the systems was performed with a large set of simulations and real-life experiments with a new humanoid robot denominated IVO and with an aerial robot. The experiments show that the combination of SFM and machine learning can solve human-aware robot navigation in complex dynamic environments., This research was supported by the grant MDM-2016-0656 funded by MCIN/AEI / 10.13039/501100011033, the grant ROCOTRANSP PID2019-106702RB-C21 funded by MCIN/AEI/ 10.13039/501100011033 and the grant CANOPIES H2020-ICT-2020-2-101016906 funded by the European Union., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Enhancing robotic collaborative tasks through contextual human motion prediction and intention inference

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License., Predicting human motion based on a sequence of past observations is crucial for various applications in robotics and computer vision. Currently, this problem is typically addressed by training deep learning models using some of the most well-known 3D human motion datasets widely used in the community. However, these datasets generally do not consider how humans behave and move when a robot is nearby, leading to a data distribution different from the real distribution of motion that robots will encounter when collaborating with humans. Additionally, incorporating contextual information related to the interactive task between the human and the robot, as well as information on the human willingness to collaborate with the robot, can improve not only the accuracy of the predicted sequence but also serve as a useful tool for robots to navigate through collaborative tasks successfully. In this research, we propose a deep learning architecture that predicts both 3D human body motion and human intention for collaborative tasks. The model employs a multi-head attention mechanism, taking into account human motion and task context as inputs. The resulting outputs include the predicted motion of the human body and the inferred human intention. We have validated this architecture in two different tasks: collaborative object handover and collaborative grape harvesting. While the architecture remains the same for both tasks, the inputs differ. In the handover task, the architecture considers human motion, robot end effector, and obstacle positions as inputs. Additionally, the model can be conditioned on the desired intention to tailor the output motion accordingly. To assess the performance of the collaborative handover task, we conducted a user study to evaluate human perception of the robot’s sociability, naturalness, security, and comfort. This evaluation was conducted by comparing the robot’s behavior when it utilized the prediction in its planner versus when it did not. Furthermore, we also applied the model to a collaborative grape harvesting task. By integrating human motion prediction and human intention inference, our architecture shows promising results in enhancing the capabilities of robots in collaborative scenarios. The model’s flexibility allows it to handle various tasks with different inputs, making it adaptable to real-world applications., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020-ICT-2020-2-101016906) and the JST Moonshot R & D Grant Number JPMJMS2011-85. The authors declare that they have no conflict of interest., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Efficient hand gesture recognition for human-robot interaction

  • Peral Sánchez, Marc|||0000-0001-6521-6476
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
In this paper, we present an efficient and reliable deep-learning approach that allows users to communicate with robots via hand gesture recognition. Contrary to other works which use external devices such as gloves [1] or joysticks [2] to tele-operate robots, the proposed approach uses only visual information to recognize user's instructions that are encoded in a set of pre-defined hand gestures. Particularly, the method consists of two modules which work sequentially to extract 2D landmarks of hands –ie. joints positions– and to predict the hand gesture based on a temporal representation of them. The approach has been validated in a recent state-of-the-art dataset where it outperformed other methods that use multiple pre-processing steps such as optical flow and semantic segmentation. Our method achieves an accuracy of 87.5% and runs at 10 frames per second. Finally, we conducted real-life experiments with our IVO robot to validate the framework during the interaction process., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Human-robot collaborative minimum time search through sub-priors in ant colony optimization

  • Gil Viyuela, Óscar|||0000-0001-8812-3168
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Human-Robot Collaboration (HRC) has evolved into a highly promising issue owing to the latest breakthroughs in Artificial Intelligence (AI) and Human-Robot Interaction (HRI), among other reasons. This emerging growth increases the need to design multi-agent algorithms that can manage also human preferences. This letter presents an extension of the Ant Colony Optimization (ACO) meta-heuristic to solve the Minimum Time Search (MTS) task, in the case where humans and robots perform an object searching task together. The proposed model consists of two main blocks. The first one is a convolutional neural network (CNN) that provides the prior probabilities about where an object may be from a segmented image. The second one is the Sub-prior MTS-ACO algorithm (SP-MTS-ACO), which takes as inputs the prior probabilities and the particular search preferences of the agents in different sub-priors to generate search plans for all agents. The model has been tested in real experiments for the joint search of an object through a Vizanti web-based visualization in a tablet computer. The designed interface allows the communication between a human and our humanoid robot named IVO. The obtained results show an improvement in the search perception of the users without loss of efficiency., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and JST Moonshot R & D Grant Number JPMJMS2011-85., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Inference vs. explicitness. Do we really need the perfect predictor? The human-robot collaborative object transportation case

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
When robots interact with humans, limitations in their internal models arise due to the uncertainty and even randomness of human behavior. This has led to attempts to predict human future actions and infer their intent. However, some authors argue for combining inference engines with communication systems that explicitly elicit human intention. This work builds on our Perception-Intention-Action (PIA) cycle, a framework that considers human intention at the same level as perception of the environment. The PIA cycle is used in a collaborative task to compare the effect on different human- robot interaction aspects of using a force predictor that infers human implicit intention versus a communication system that explicitly elicits human intention. A study with 18 volunteers shows that allowing humans to directly express themselves can achieve the same improvement as an intention predictor., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and by JST Moonshot R & D Grant Number JPMJMS2011. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

ACHORD: communication-aware multi-robot coordination with intermittent connectivity

  • Saboia Da Silva, Maira
  • Clack, Lillian
  • Thangavelu, Vivek
  • Edlund, Jeffrey
  • Kyohei, Otsu
  • Correa, Gustavo J
  • Varadharajan, Vivek Shankar
  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Touma, Thomas
  • Bouman, Amanda
  • Melikyan, Hovhannes
  • Pailevanian, Torkom
  • Kim, Sung-Kyun
  • Archanian, Avak
  • Vaquero, Tiago Stegun
  • Beltrame, Giovanni
  • Napp, Nils
  • Pessin, Gustavo
  • Agha-mohammadi, Ali-akbar
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Communication is an important capability for multi-robot exploration because (1) inter-robot communication (comms) improves coverage efficiency and (2) robot-to-base comms improves situational awareness. Exploring comms-restricted (e.g., subterranean) environments requires a multi-robot system to tolerate and anticipate intermittent connectivity, and to carefully consider comms requirements, otherwise mission-critical data may be lost. In this paper, we describe and analyze ACHORD (Autonomous & Collaborative High-Bandwidth Operations with Radio Droppables), a multi-layer networking solution which tightly co-designs the network architecture and high-level decision-making for improved comms. ACHORD provides bandwidth prioritization and timely and reliable data transfer despite intermittent connectivity. Furthermore, it exposes low-layer networking metrics to the application layer to enable robots to autonomously monitor, map, and extend the network via droppable radios, as well as restore connectivity to improve collaborative exploration. We evaluate our solution with respect to the comms performance in several challenging underground environments including the DARPA SubT Finals competition environment. Our findings support the use of data stratification and flow control to improve bandwidth-usage., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Improving human-robot interaction effectiveness in human-robot collaborative object transportation using force prediction

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work, we analyse the use of a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We check that this force prediction can improve multiple parameters associated with effective Human-Robot Interaction (HRI) such as percep- tion of the robot’s contribution to the task, comfort or trust in the robot in a physical Human Robot Interaction (pHRI). We present a Deep Learning model that allows to predict the force that a human will exert in the next 1 s using as inputs the force previously exerted by the human, the robot’s velocity and environment information obtained from the robot’s LiDAR. Its success rate is up to 92.3% in testset and up to 89.1% in real experiments. We demonstrate that this force prediction, in addition to being able to be used directly to detect changes in the human’s intention, can be processed to obtain an estimate of the human’s desired trajectory. We have validated this approach with a user study involving 18 volunteers., Work supported under the European project CANOPIES (H2020 ICT-2020-2-101016906) and the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033) and by JST Moonshot R&D Grant Number JPMJMS2011.The first author acknowledges Spanish FPU grant with ref.FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

When the inference meets the explicitness or why multimodality can make us forget about the perfect predictor

  • Domínguez-Vidal, José Enrique
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
The version of record of this article, first published in International journal of social robotics, is available online at Publisher’s website: http://dx.doi.org/10.1007/s12369-025-01303-9, Although in the literature it is common to find predictors and inference systems that try to predict human intentions, the uncertainty of these models due to the randomness of human behaviour has led some authors to start advocating the use of communication systems that explicitly elicit human intention. In this work, it is analysed the use of four different communication systems with a human-robot collaborative object transportation task as experimental testbed: two intention predictors (one based on force prediction and another with an enhanced velocity prediction algorithm) and two explicit communication methods (a button interface and a voice-command recognition system). These systems were integrated into IVO, a custom mobile social robot equipped with force sensor to detect the force exchange between both agents and LiDAR to detect the environment. The collaborative task required transporting an object over a 5–7 meter distance with obstacles in the middle, demanding rapid decisions and precise physical coordination. 75 volunteers perform a total of 255 executions divided into three groups, testing inference systems in the first round, communication systems in the second, and the combined strategies in the third. The results show that, 1) once sufficient performance is achieved, the human no longer notices and positively assesses technical improvements; 2) the human prefers systems that are more natural to them even though they have higher failure rates; and 3) the preferred option is the right combination of both systems., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020- ICT-2020–2-101016906) and the JST Moonshot R&D Grant Number JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Real-life experiment metrics for evaluating human-robot collaborative navigation tasks

  • Repiso Polo, Ely|||0000-0001-6540-8347
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., As robots move from laboratories and industries to the real world, they must develop new abilities to collaborate with humans in various aspects, including human-robot collaborative navigation (HRCN) tasks. Then, it is required to develop general methodologies to evaluate these robots’ behaviors. These methodologies should incorporate objective and subjective measurements. Objective measurements for evaluating a robot’s behavior while navigating with others can be accomplished using social distances in conjunction with task characteristics, people-robot relationships, and physical space. Additionally, the objective evaluation of the task must consider human behavior, which is influenced by changes and the structure of their environment. Subjective evaluations of robot’s behaviors can be conducted using surveys that address various aspects of robot usability. This includes people’s perceptions of their interaction during their collaborative task with the robot, focusing on aspects such as sociability, comfort, and task-intelligence. Moreover, the communicative interaction between the agents (people and robots) involved in the collaborative task should also be evaluated. Therefore, this paper presents a comprehensive methodology for objectively and subjectively evaluating HRCN tasks., This work has been supported by the Artificial Intelligence for Human- Robot Interaction (AI4HRI) project ANR-20-IADJ-0006. Also, this Work has been supported under the EU project grant CANOPIES (H2020- ICT-2020-2-101016906) and by the JST Moonshot R D Grant Number JPMJMS2011. Ely Repiso was also supported during her PhD by Spanish Ministry of Science and Innovation under a FPI-grant (BES-2014-067713) and during her Post Doc by the project AI4HRI, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Attention deep learning based model for predicting the 3D Human Body Pose using the Robot Human Handover Phases

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This work proposes a human motion prediction model for handover operations. We use in this work, the different phases of the handover operation to improve the human motion predictions. Our attention deep learning based model takes into account the position of the robot’s End Effector and the phase in the handover operation to predict future human poses. Our model outputs a distribution of possible positions rather than one deterministic position, a key feature in order to allow robots to collaborate with humans. The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained with an Intel RealSense D435i camera attached inside the robot’s head. The results shown a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906), Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Classification of humans social relations within urban areas

  • Castro Arcusa, Oscar
  • Repiso Polo, Ely|||0000-0001-6540-8347
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
This paper presents the design of deep learning architectures which allow to classify the social relationship existing between two people who are walking in a side-by-side formation into four possible categories --colleagues, couple, family or friendship. The models are developed using Neural Networks or Recurrent Neural Networks to achieve the classification and are trained and evaluated using a database obtained from humans walking together in an urban environment. The best achieved model accomplishes a good accuracy in the classification problem and its results enhance the outcomes from a previous study [1]. In addition, we have developed several models to classify the social interactions in two categories --“intimate" and "acquaintances", where the best model achieves a very good performance, and for a real robot this classification is enough to be able to customize its behavior to its users. Furthermore, the proposed models show their future potential to improve its efficiency and to be implemented in a real robot., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

The human intention: a taxonomy attempt and its applications to robotics

  • Domínguez-Vidal, José Enrique
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
The version of record of this article, first published in International journal of social robotics, is available online at Publisher’s website: http://dx.doi.org/10.1007/s12369-025-01304-8, Despite a surge in robotics research dedicated to inferring and understanding human intent, a universally accepted definition remains elusive since existing works often equate human intention with specific task-related goals. This article seeks to address this gap by examining the multifaceted nature of intention. Drawing on insights from psychology, it attempts to consolidate a definition of intention into a comprehensible framework for a broader audience. The article classifies different types of intention based on psychological and communication studies, offering guidance to researchers shifting from pure technical enhancements to a more human-centric perspective in robotics. It then demonstrates how various robotics studies can be aligned with these intention categories. Finally, through in-depth analyses of collaborative search and object transport use cases, the article underscores the significance of considering the diverse facets of human intention., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020-ICT-2020–2–101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011–85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Human motion trajectory prediction using the social force model for real-time and low computational cost applications

  • Gil Viyuela, Óscar|||0000-0001-8812-3168
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG, Human motion trajectory prediction is a very important functionality for human-robot collaboration, specifically in accompanying, guiding, or approaching tasks, but also in social robotics, self-driving vehicles, or security systems. In this paper, a novel trajectory prediction model, Social Force Generative Adversarial Network (SoFGAN), is proposed. SoFGAN uses a Generative Adversarial Network (GAN) and Social Force Model (SFM) to generate different plausible people trajectories reducing collisions in a scene. Furthermore, a Conditional Variational Autoencoder (CVAE) module is added to emphasize the destination learning. We show that our method is more accurate in making predictions in UCY or BIWI datasets than most of the current state-of-the-art models and also reduces collisions in comparison to other approaches. Through real-life experiments, we demonstrate that the model can be used in real-time without GPU’s to perform good quality predictions with a low computational cost., Ó. Gil—Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and JST Moonshot R & D Grant Number JPMJMS2011-85., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Exploring transformers and visual transformers for force prediction in human-robot collaborative transportation tasks

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we analyze the possibilities offered by Deep Learning State-of-the-Art architectures such as Transformers and Visual Transformers in generating a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We outperform our previous predictor by achieving a success rate of 93.8% in testset and 90.9% in real experiments with 21 volunteers predicting in both cases the force that the human will exert during the next 1 s. A modification in the architecture allows us to obtain a second output from the model with a velocity prediction, which allows us to improve the capabilities of our predictor if it is used to estimate the trajectory that the humanrobot pair will follow. An ablation test is also performed to verify the relative contribution to performance of each input., Work supported under the European project CANOPIES (H2020ICT-2020-2-101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Event-based line SLAM in real-time

  • Chamorro Hernández, William Oswaldo
  • Solà Ortega, Joan|||0000-0002-2933-3381
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Event-based cameras generate asynchronous streams of events, triggered proportionally to the logarithmic change of brightness in the scene. These cameras have very low latency and high dynamic range suitable to address challenging motion scenarios in robotics. In this work, we explore a new event-based line-SLAM approach following a parallel tracking and mapping philosophy. Our fast tracking algorithm, produces accurate camera pose estimates at a high rate by minimizing the event-line reprojection error with an error-state Kalman filter formulated entirely with Lie theory. The mapping thread leverages the natural edge highlighting strength of events to recover and optimize straight lines in human-made scenarios. The proper manipulation of matrix sparsity as well as the information sharing between tracking and mapping nodes allow us to achieve real-time performance on a standard multi-core CPU. This system was tested on several scenarios rich in straight edge objects, and compared against, ground truth and frame and event based state-of-the-art approaches., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Human-robot harvesting plan negotiation: perception, grape mapping and shared planning

  • Dalmasso Blanch, Marc|||0000-0003-1734-7863
  • García Ruiz, Francisco
  • Ciarfuglia, Thomas A.
  • Saraceni, Leonardo
  • Motoi, Ionut M.
  • Gil Moya, Emilio|||0000-0002-3929-5649
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper we present an application of plan negotiation for distributing work in Human-Robot Teams (HRT), engaging in a shared plan generation process. This model has been experimented over data collected from a real-life table grape vineyard field in Corsira (Aprilia, Italy) on the context of the EU CANOPIES project. In this work we describe a complete system up to negotiation agreement, the perception system to detect the grapes position and quality, the grape map generation process and the proposed plan negotiation approach, and protocol used to interactively generate harvest plans. Our proposed approach tackles harvest plan generation as a two step negotiation process on negotiation domains of increasing embedded information. The shared planning process was evaluated using both a subjective user study and teammate quantitative models., This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016906 – Project CANOPIES, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

A survey on socially aware robot navigation: Taxonomy and future challenges

  • Singamaneni, Phani Teja
  • Bachiller Burgos, Pilar
  • Manso, Luis J.
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Spalanzani, Anne
  • Alami, Rachid
Socially aware robot navigation is gaining popularity with the increase in delivery and assistive robots. The research is further fueled by a need for socially aware navigation skills in autonomous vehicles to move safely and appropriately in spaces shared with humans. Although most of these are ground robots, drones are also entering the field. In this paper, we present a literature survey of the works on socially aware robot navigation in the past 10 years. We propose four different faceted taxonomies to navigate the literature and examine the field from four different perspectives. Through the taxonomic review, we discuss the current research directions and the extending scope of applications in various domains. Further, we put forward a list of current research opportunities and present a discussion on possible future challenges that are likely to emerge in the field., This work was supported by the Horizon Europe Framework Programme [Grant: euROBIN 101070596]; Agence Nationale de la Recherche [Grant: ANITI ANR-19-PI3A0004]; European project CANOPIES [Grant: H2020ICT-2020-2-101016906]; Spanish Government (MCIN/AEI) and European Union (“NextGenerationEU”/PRTR) [Grant: TED2021-131739-C22]; Spanish Government (MCIN) [Grant: PDC2022-133597-C41]; Spanish Government (MCIN) [Grant: PID2022-142039NA-I00]., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Force and velocity prediction in human-robot collaborative transportation tasks through video retentive networks

  • Domínguez Vidal, José Enrique|||0000-0002-0397-9248
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, In this article, we propose a generalization of a Deep Learning State-of-the-Art architecture such as Retentive Networks so that it can accept video sequences as input. With this generalization, we design a force/velocity predictor applied to the medium-distance Human-Robot collaborative object transportation task. We achieve better results than with our previous predictor by reaching success rates in testset of up to 93.7% in predicting the force to be exerted by the human and up to 96.5% in the velocity of the human-robot pair during the next 1 s, and up to 91.0% and 95.0% respectively in real experiments. This new architecture also manages to improve inference times by up to 32.8% with different graphics cards. Finally, an ablation test allows us to detect that one of the input variables used so far, such as the position of the task goal, could be discarded allowing this goal to be chosen dynamically by the human instead of being pre-set., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Context and intention for 3D human motion prediction: experimentation and user study in handover tasks

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work we present a novel attention deep learning model that uses context and human intention for 3D human body motion prediction in handover human-robot tasks. This model uses a multi-head attention architecture which incorporates as inputs the human motion, the robot end effector and the position of the obstacles. The outputs of the model are the predicted motion of the human body and the predicted human intention. We use this model to analyze a handover collaborative task with a robot where the robot is able to predict the future motion of the human and use this information in it’s planner. Several experiments are performed where human volunteers fill a standard poll to rate different features, taking into account when the robot uses the prediction versus when the robot doesn’t use the prediction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016- 0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI / 10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT2020-2-101016906)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Context attention: human motion prediction using context information and deep learning attention models

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
This work proposes a human motion prediction model for handover operations. The model uses a multi-headed attention architecture to process the human skeleton data together with contextual data from the operation. This contextual data consists on the position of the robot’s End Effector (REE). The model input is a sequence of 5 seconds skeleton position and it outputs the predicted 2.5 future seconds position. We provide results of the human upper body and the human right hand or Human End Effector (HEE). The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained using OpenPose with an Intel RealSense D435i camera set inside the robot’s head. The results show a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the ROCOTRANSP project (PID2019-106702RB-C21/AEI/10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Body gesture recognition to control a social mobile robot

  • Laplaza Galindo, Javier|||0000-0003-4864-5759
  • Romero Martín, Rut|||0000-0002-9257-3898
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
In this work, we propose a gesture-based language to allow humans to interact with robots using their body in a natural way. We have created a new gesture detection model using neural networks and a new dataset of humans making a collection of body gestures to train this architecture. Furthermore, we compare body gesture communication with other communication channels to demonstrate the importance of adding this knowledge to robots. The presented approach is validated in diverse simulations and real-life experiments with non-trained volunteers. This attains promising results and establishes that it is a valuable framework for social robotic applications, such as human robot collaboration or human-robot interaction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Real-life experiment metrics for evaluating human-robot collaborative navigation tasks

  • Repiso Polo, Ely
  • Garrell Zulueta, Anais
  • Sanfeliu Cortés, Alberto
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., As robots move from laboratories and industries to the real world, they must develop new abilities to collaborate with humans in various aspects, including human-robot collaborative navigation (HRCN) tasks. Then, it is required to develop general methodologies to evaluate these robots’ behaviors. These methodologies should incorporate objective and subjective measurements. Objective measurements for evaluating a robot’s behavior while navigating with others can be accomplished using social distances in conjunction with task characteristics, people-robot relationships, and physical space. Additionally, the objective evaluation of the task must consider human behavior, which is influenced by changes and the structure of their environment. Subjective evaluations of robot’s behaviors can be conducted using surveys that address various aspects of robot usability. This includes people’s perceptions of their interaction during their collaborative task with the robot, focusing on aspects such as sociability, comfort, and task-intelligence. Moreover, the communicative interaction between the agents (people and robots) involved in the collaborative task should also be evaluated. Therefore, this paper presents a comprehensive methodology for objectively and subjectively evaluating HRCN tasks., This work has been supported by the Artificial Intelligence for Human- Robot Interaction (AI4HRI) project ANR-20-IADJ-0006. Also, this Work has been supported under the EU project grant CANOPIES (H2020- ICT-2020-2-101016906) and by the JST Moonshot R D Grant Number JPMJMS2011. Ely Repiso was also supported during her PhD by Spanish Ministry of Science and Innovation under a FPI-grant (BES-2014-067713) and during her Post Doc by the project AI4HRI, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Enhancing robotic collaborative tasks through contextual human motion prediction and intention inference

  • Laplaza Galindo, Javier
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License., Predicting human motion based on a sequence of past observations is crucial for various applications in robotics and computer vision. Currently, this problem is typically addressed by training deep learning models using some of the most well-known 3D human motion datasets widely used in the community. However, these datasets generally do not consider how humans behave and move when a robot is nearby, leading to a data distribution different from the real distribution of motion that robots will encounter when collaborating with humans. Additionally, incorporating contextual information related to the interactive task between the human and the robot, as well as information on the human willingness to collaborate with the robot, can improve not only the accuracy of the predicted sequence but also serve as a useful tool for robots to navigate through collaborative tasks successfully. In this research, we propose a deep learning architecture that predicts both 3D human body motion and human intention for collaborative tasks. The model employs a multi-head attention mechanism, taking into account human motion and task context as inputs. The resulting outputs include the predicted motion of the human body and the inferred human intention. We have validated this architecture in two different tasks: collaborative object handover and collaborative grape harvesting. While the architecture remains the same for both tasks, the inputs differ. In the handover task, the architecture considers human motion, robot end effector, and obstacle positions as inputs. Additionally, the model can be conditioned on the desired intention to tailor the output motion accordingly. To assess the performance of the collaborative handover task, we conducted a user study to evaluate human perception of the robot’s sociability, naturalness, security, and comfort. This evaluation was conducted by comparing the robot’s behavior when it utilized the prediction in its planner versus when it did not. Furthermore, we also applied the model to a collaborative grape harvesting task. By integrating human motion prediction and human intention inference, our architecture shows promising results in enhancing the capabilities of robots in collaborative scenarios. The model’s flexibility allows it to handle various tasks with different inputs, making it adaptable to real-world applications., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020-ICT-2020-2-101016906) and the JST Moonshot R & D Grant Number JPMJMS2011-85. The authors declare that they have no conflict of interest., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Efficient hand gesture recognition for human-robot interaction

  • Peral Sánchez, Marc
  • Sanfeliu Cortés, Alberto
  • Garrell Zulueta, Anais
In this paper, we present an efficient and reliable deep-learning approach that allows users to communicate with robots via hand gesture recognition. Contrary to other works which use external devices such as gloves [1] or joysticks [2] to tele-operate robots, the proposed approach uses only visual information to recognize user's instructions that are encoded in a set of pre-defined hand gestures. Particularly, the method consists of two modules which work sequentially to extract 2D landmarks of hands –ie. joints positions– and to predict the hand gesture based on a temporal representation of them. The approach has been validated in a recent state-of-the-art dataset where it outperformed other methods that use multiple pre-processing steps such as optical flow and semantic segmentation. Our method achieves an accuracy of 87.5% and runs at 10 frames per second. Finally, we conducted real-life experiments with our IVO robot to validate the framework during the interaction process., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Weakly and semi-supervised detection, segmentation and tracking of table grapes with limited and noisy data

  • Ciarfuglia, Thomas A.
  • Motoi, Ionut M.
  • Saraceni, Leonardo
  • Fawakherji, Mulham
  • Sanfeliu Cortés, Alberto
  • Nardi, Daniele
Detection, segmentation and tracking of fruits and vegetables are three fundamental tasks for precision agriculture, enabling robotic harvesting and yield estimation applications. However, modern algorithms are data hungry and it is not always possible to gather enough data to apply the best performing supervised approaches. Since data collection is an expensive and cumbersome task, the enabling technologies for using computer vision in agriculture are often out of reach for small businesses. Following previous work in this context (Ciarfuglia et al., 2022), where we proposed an initial weakly supervised solution to reduce the data needed to get state-of-the-art detection and segmentation in precision agriculture applications, here we improve that system and explore the problem of tracking fruits in orchards. We present the case of vineyards of table grapes in southern Lazio (Italy) since grapes are a difficult fruit to segment due to occlusion, colour and general illumination conditions. We consider the case in which there is some initial labelled data that could work as source data (e.g. wine grape data), but it is considerably different from the target data (e.g. table grape data). To improve detection and segmentation on the target data, we propose to train the segmentation algorithm with a weak bounding box label, while for tracking we leverage 3D Structure from Motion algorithms to generate new labels from already labelled samples. Finally, the two systems are combined in a full semi-supervised approach. Comparisons with state-of-the-art supervised solutions show how our methods are able to train new models that achieve high performances with few labelled images and with very simple labelling., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Body gesture recognition to control a social mobile robot

  • Laplaza Galindo, Javier
  • Romero Martín, Rut
  • Sanfeliu Cortés, Alberto
  • Garrell Zulueta, Anais
In this work, we propose a gesture-based language to allow humans to interact with robots using their body in a natural way. We have created a new gesture detection model using neural networks and a new dataset of humans making a collection of body gestures to train this architecture. Furthermore, we compare body gesture communication with other communication channels to demonstrate the importance of adding this knowledge to robots. The presented approach is validated in diverse simulations and real-life experiments with non-trained volunteers. This attains promising results and establishes that it is a valuable framework for social robotic applications, such as human robot collaboration or human-robot interaction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Body gestures recognition for social human-robot interaction

  • Laplaza Galindo, Javier
  • Sanfeliu Cortés, Alberto
  • Garrell Zulueta, Anais
In this paper, a solution for human gesture classification is proposed. The solution uses a Deep Learning model and is meant to be useful for non-verbal communication between humans and robots. The research focuses on the creation of the “temPoral bOdy geSTUre REcognition model” (POSTURE) that can recognise continuous gestures performed in real-life situations. The suggested model takes into account spatial and temporal components so as to achieve the recognition of more natural and intuitive gestures. In a first step, a framework extracts from all the images the corresponding landmarks for each of the body joints. Next, some data filtering techniques are applied with the aim of avoiding problems related with the data. And finally, different neural network configurations and approaches are tested to find the optimal performance. The validation of the model is accomplished throughout an extensive set of simulations and real-life experiments., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Exploring transformers and visual transformers for force prediction in human-robot collaborative transportation tasks

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper, we analyze the possibilities offered by Deep Learning State-of-the-Art architectures such as Transformers and Visual Transformers in generating a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We outperform our previous predictor by achieving a success rate of 93.8% in testset and 90.9% in real experiments with 21 volunteers predicting in both cases the force that the human will exert during the next 1 s. A modification in the architecture allows us to obtain a second output from the model with a velocity prediction, which allows us to improve the capabilities of our predictor if it is used to estimate the trajectory that the humanrobot pair will follow. An ablation test is also performed to verify the relative contribution to performance of each input., Work supported under the European project CANOPIES (H2020ICT-2020-2-101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Human-robot harvesting plan negotiation: perception, grape mapping and shared planning

  • Dalmasso Blanch, Marc
  • García Ruiz, Francisco
  • Ciarfuglia, Thomas A.
  • Saraceni, Leonardo
  • Motoi, Ionut M.
  • Gil Moya, Emilio
  • Sanfeliu Cortés, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper we present an application of plan negotiation for distributing work in Human-Robot Teams (HRT), engaging in a shared plan generation process. This model has been experimented over data collected from a real-life table grape vineyard field in Corsira (Aprilia, Italy) on the context of the EU CANOPIES project. In this work we describe a complete system up to negotiation agreement, the perception system to detect the grapes position and quality, the grape map generation process and the proposed plan negotiation approach, and protocol used to interactively generate harvest plans. Our proposed approach tackles harvest plan generation as a two step negotiation process on negotiation domains of increasing embedded information. The shared planning process was evaluated using both a subjective user study and teammate quantitative models., This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016906 – Project CANOPIES, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Context attention: human motion prediction using context information and deep learning attention models

  • Laplaza Galindo, Javier
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
This work proposes a human motion prediction model for handover operations. The model uses a multi-headed attention architecture to process the human skeleton data together with contextual data from the operation. This contextual data consists on the position of the robot’s End Effector (REE). The model input is a sequence of 5 seconds skeleton position and it outputs the predicted 2.5 future seconds position. We provide results of the human upper body and the human right hand or Human End Effector (HEE). The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained using OpenPose with an Intel RealSense D435i camera set inside the robot’s head. The results show a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the ROCOTRANSP project (PID2019-106702RB-C21/AEI/10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Context and intention for 3D human motion prediction: experimentation and user study in handover tasks

  • Laplaza Galindo, Javier
  • Garrell Zulueta, Anais
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work we present a novel attention deep learning model that uses context and human intention for 3D human body motion prediction in handover human-robot tasks. This model uses a multi-head attention architecture which incorporates as inputs the human motion, the robot end effector and the position of the obstacles. The outputs of the model are the predicted motion of the human body and the predicted human intention. We use this model to analyze a handover collaborative task with a robot where the robot is able to predict the future motion of the human and use this information in it’s planner. Several experiments are performed where human volunteers fill a standard poll to rate different features, taking into account when the robot uses the prediction versus when the robot doesn’t use the prediction., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016- 0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI / 10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT2020-2-101016906)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Improving human-robot interaction effectiveness in human-robot collaborative object transportation using force prediction

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this work, we analyse the use of a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We check that this force prediction can improve multiple parameters associated with effective Human-Robot Interaction (HRI) such as percep- tion of the robot’s contribution to the task, comfort or trust in the robot in a physical Human Robot Interaction (pHRI). We present a Deep Learning model that allows to predict the force that a human will exert in the next 1 s using as inputs the force previously exerted by the human, the robot’s velocity and environment information obtained from the robot’s LiDAR. Its success rate is up to 92.3% in testset and up to 89.1% in real experiments. We demonstrate that this force prediction, in addition to being able to be used directly to detect changes in the human’s intention, can be processed to obtain an estimate of the human’s desired trajectory. We have validated this approach with a user study involving 18 volunteers., Work supported under the European project CANOPIES (H2020 ICT-2020-2-101016906) and the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033) and by JST Moonshot R&D Grant Number JPMJMS2011.The first author acknowledges Spanish FPU grant with ref.FPU19/06582., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Inference vs. explicitness. Do we really need the perfect predictor? The human-robot collaborative object transportation case

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
When robots interact with humans, limitations in their internal models arise due to the uncertainty and even randomness of human behavior. This has led to attempts to predict human future actions and infer their intent. However, some authors argue for combining inference engines with communication systems that explicitly elicit human intention. This work builds on our Perception-Intention-Action (PIA) cycle, a framework that considers human intention at the same level as perception of the environment. The PIA cycle is used in a collaborative task to compare the effect on different human- robot interaction aspects of using a force predictor that infers human implicit intention versus a communication system that explicitly elicits human intention. A study with 18 volunteers shows that allowing humans to directly express themselves can achieve the same improvement as an intention predictor., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and by JST Moonshot R & D Grant Number JPMJMS2011. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Working with troubles and failures in conversation between humans and robots: workshop report

  • Förster, Frank
  • Wiltschko, Martina
  • Kapetanios, Epaminondas
This paper summarizes the structure and findings from the first Workshop on Troubles and Failures in Conversations between Humans and Robots. The workshop was organized to bring together a small, interdisciplinary group of researchers working on miscommunication from two complementary perspectives. One group of technology-oriented researchers was made up of roboticists, Human-Robot Interaction (HRI) researchers and dialogue system experts. The second group involved experts from conversation analysis, cognitive science, and linguistics. Uniting both groups of researchers is the belief that communication failures between humans and machines need to be taken seriously and that a systematic analysis of such failures may open fruitful avenues in research beyond current practices to improve such systems, including both speech-centric and multimodal interfaces. This workshop represents a starting point for this endeavour. The aim of the workshop was threefold: Firstly, to establish an interdisciplinary network of researchers that share a common interest in investigating communicative failures with a particular view towards robotic speech interfaces; secondly, to gain a partial overview of the 'failure landscape' as experienced by roboticists and HRI researchers; and thirdly, to determine the potential for creating a robotic benchmark scenario for testing future speech interfaces with respect to the identified failures. The present article summarizes both the 'failure landscape' surveyed during the workshop as well as the outcomes of the attempt to define a benchmark scenario., The workshop, the outcomes of which are described in this paper, was funded by the UK Engineering and Physical Science Research Council (EPSRC) Robotics & Autonomous Systems Network (UK-RAS) Pump Priming programme under the project title 'Charting the Limits and Developing Future Directions of Speech Interfaces for Robotics'. DG is supported under the EPSRC projects NLG for low-resource domains (EP/T024917/1) and CiViL (EP/T014598/1). Some of the authors are supported by the Engineering and Physical Sciences Research Council [grant numbers EP/V00784X/1, EP/X009343/1] including through the Trustworthy Autonomous Systems (TAS) Hub. One of the authors has been supported by the H2020 EU projects CANOPIES'A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems, Grant Agreement 101016906. DK is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy-EXC 2002/1 'Science of Intelligence'-project number 390523135.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Classification of humans social relations within urban areas

  • Castro Arcusa, Oscar
  • Repiso Polo, Ely
  • Garrell Zulueta, Anais
  • Sanfeliu Cortés, Alberto
This paper presents the design of deep learning architectures which allow to classify the social relationship existing between two people who are walking in a side-by-side formation into four possible categories --colleagues, couple, family or friendship. The models are developed using Neural Networks or Recurrent Neural Networks to achieve the classification and are trained and evaluated using a database obtained from humans walking together in an urban environment. The best achieved model accomplishes a good accuracy in the classification problem and its results enhance the outcomes from a previous study [1]. In addition, we have developed several models to classify the social interactions in two categories --“intimate" and "acquaintances", where the best model achieves a very good performance, and for a real robot this classification is enough to be able to customize its behavior to its users. Furthermore, the proposed models show their future potential to improve its efficiency and to be implemented in a real robot., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Human-robot collaborative minimum time search through sub-priors in ant colony optimization

  • Gil Viyuela, Óscar
  • Sanfeliu Cortés, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Human-Robot Collaboration (HRC) has evolved into a highly promising issue owing to the latest breakthroughs in Artificial Intelligence (AI) and Human-Robot Interaction (HRI), among other reasons. This emerging growth increases the need to design multi-agent algorithms that can manage also human preferences. This letter presents an extension of the Ant Colony Optimization (ACO) meta-heuristic to solve the Minimum Time Search (MTS) task, in the case where humans and robots perform an object searching task together. The proposed model consists of two main blocks. The first one is a convolutional neural network (CNN) that provides the prior probabilities about where an object may be from a segmented image. The second one is the Sub-prior MTS-ACO algorithm (SP-MTS-ACO), which takes as inputs the prior probabilities and the particular search preferences of the agents in different sub-priors to generate search plans for all agents. The model has been tested in real experiments for the joint search of an object through a Vizanti web-based visualization in a tablet computer. The designed interface allows the communication between a human and our humanoid robot named IVO. The obtained results show an improvement in the search perception of the users without loss of efficiency., Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and JST Moonshot R & D Grant Number JPMJMS2011-85., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

The human intention: a taxonomy attempt and its applications to robotics

  • Domínguez-Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
The version of record of this article, first published in International journal of social robotics, is available online at Publisher’s website: http://dx.doi.org/10.1007/s12369-025-01304-8, Despite a surge in robotics research dedicated to inferring and understanding human intent, a universally accepted definition remains elusive since existing works often equate human intention with specific task-related goals. This article seeks to address this gap by examining the multifaceted nature of intention. Drawing on insights from psychology, it attempts to consolidate a definition of intention into a comprehensible framework for a broader audience. The article classifies different types of intention based on psychological and communication studies, offering guidance to researchers shifting from pure technical enhancements to a more human-centric perspective in robotics. It then demonstrates how various robotics studies can be aligned with these intention categories. Finally, through in-depth analyses of collaborative search and object transport use cases, the article underscores the significance of considering the diverse facets of human intention., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020-ICT-2020–2–101016906) and by JST Moonshot R & D Grant Number: JPMJMS2011–85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Social robot navigation tasks: combining machine learning techniques and social force model

  • Gil Viyuela, Óscar
  • Garrell Zulueta, Anais
  • Sanfeliu Cortés, Alberto
© 2021 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), Social robot navigation in public spaces, buildings or private houses is a difficult problem that is not well solved due to environmental constraints (buildings, static objects etc.), pedestrians and other mobile vehicles. Moreover, robots have to move in a human-aware manner—that is, robots have to navigate in such a way that people feel safe and comfortable. In this work, we present two navigation tasks, social robot navigation and robot accompaniment, which combine machine learning techniques with the Social Force Model (SFM) allowing human-aware social navigation. The robots in both approaches use data from different sensors to capture the environment knowledge as well as information from pedestrian motion. The two navigation tasks make use of the SFM, which is a general framework in which human motion behaviors can be expressed through a set of functions depending on the pedestrians’ relative and absolute positions and velocities. Additionally, in both social navigation tasks, the robot’s motion behavior is learned using machine learning techniques: in the first case using supervised deep learning techniques and, in the second case, using Reinforcement Learning (RL). The machine learning techniques are combined with the SFM to create navigation models that behave in a social manner when the robot is navigating in an environment with pedestrians or accompanying a person. The validation of the systems was performed with a large set of simulations and real-life experiments with a new humanoid robot denominated IVO and with an aerial robot. The experiments show that the combination of SFM and machine learning can solve human-aware robot navigation in complex dynamic environments., This research was supported by the grant MDM-2016-0656 funded by MCIN/AEI / 10.13039/501100011033, the grant ROCOTRANSP PID2019-106702RB-C21 funded by MCIN/AEI/ 10.13039/501100011033 and the grant CANOPIES H2020-ICT-2020-2-101016906 funded by the European Union., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Force and velocity prediction in human-robot collaborative transportation tasks through video retentive networks

  • Domínguez Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, In this article, we propose a generalization of a Deep Learning State-of-the-Art architecture such as Retentive Networks so that it can accept video sequences as input. With this generalization, we design a force/velocity predictor applied to the medium-distance Human-Robot collaborative object transportation task. We achieve better results than with our previous predictor by reaching success rates in testset of up to 93.7% in predicting the force to be exerted by the human and up to 96.5% in the velocity of the human-robot pair during the next 1 s, and up to 91.0% and 95.0% respectively in real experiments. This new architecture also manages to improve inference times by up to 32.8% with different graphics cards. Finally, an ablation test allows us to detect that one of the input variables used so far, such as the position of the task goal, could be discarded allowing this goal to be chosen dynamically by the human instead of being pre-set., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Perception-intention-action cycle as a human acceptable way for improving human-robot collaborative tasks

  • Domínguez Vidal, José Enrique
  • Rodríguez Linares, Nicolás Adrián
  • Sanfeliu Cortés, Alberto
© ACM 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in HRI '23: Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, https://doi.org/10.1145/3568294.3580149., In Human-Robot Collaboration (HRC) tasks, the classical Perception-Action cycle can not fully explain the collaborative behaviour of the human-robot pair until it is extended to Perception-Intention-Action (PIA) cycle, giving to the human's intention a key role at the same level of the robot's perception and not as a subblock of this. Although part of the human's intention can be perceived or inferred by the other agent, this is prone to misunderstandings so the true intention has to be explicitly informed in some cases to fulfill the task. Here, we explore both types of intention and we combine them with the robot's perception through the concept of Situation Awareness (SA). We validate the PIA cycle and its acceptance by the user with a preliminary experiment in an object transportation task showing that its usage can increase trust in the robot., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Human motion trajectory prediction using the social force model for real-time and low computational cost applications

  • Gil Viyuela, Óscar
  • Sanfeliu Cortés, Alberto
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG, Human motion trajectory prediction is a very important functionality for human-robot collaboration, specifically in accompanying, guiding, or approaching tasks, but also in social robotics, self-driving vehicles, or security systems. In this paper, a novel trajectory prediction model, Social Force Generative Adversarial Network (SoFGAN), is proposed. SoFGAN uses a Generative Adversarial Network (GAN) and Social Force Model (SFM) to generate different plausible people trajectories reducing collisions in a scene. Furthermore, a Conditional Variational Autoencoder (CVAE) module is added to emphasize the destination learning. We show that our method is more accurate in making predictions in UCY or BIWI datasets than most of the current state-of-the-art models and also reduces collisions in comparison to other approaches. Through real-life experiments, we demonstrate that the model can be used in real-time without GPU’s to perform good quality predictions with a low computational cost., Ó. Gil—Work supported under the European project CANOPIES with grant number H2020- ICT-2020-2-101016906 and JST Moonshot R & D Grant Number JPMJMS2011-85., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Event-based line SLAM in real-time

  • Chamorro Hernández, William Oswaldo
  • Solà Ortega, Joan
  • Andrade-Cetto, Juan
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Event-based cameras generate asynchronous streams of events, triggered proportionally to the logarithmic change of brightness in the scene. These cameras have very low latency and high dynamic range suitable to address challenging motion scenarios in robotics. In this work, we explore a new event-based line-SLAM approach following a parallel tracking and mapping philosophy. Our fast tracking algorithm, produces accurate camera pose estimates at a high rate by minimizing the event-line reprojection error with an error-state Kalman filter formulated entirely with Lie theory. The mapping thread leverages the natural edge highlighting strength of events to recover and optimize straight lines in human-made scenarios. The proper manipulation of matrix sparsity as well as the information sharing between tracking and mapping nodes allow us to achieve real-time performance on a standard multi-core CPU. This system was tested on several scenarios rich in straight edge objects, and compared against, ground truth and frame and event based state-of-the-art approaches., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Attention deep learning based model for predicting the 3D Human Body Pose using the Robot Human Handover Phases

  • Laplaza Galindo, Javier
  • Pumarola Peris, Albert
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This work proposes a human motion prediction model for handover operations. We use in this work, the different phases of the handover operation to improve the human motion predictions. Our attention deep learning based model takes into account the position of the robot’s End Effector and the phase in the handover operation to predict future human poses. Our model outputs a distribution of possible positions rather than one deterministic position, a key feature in order to allow robots to collaborate with humans. The attention deep learning based model has been trained and evaluated with a dataset created using human volunteers and an anthropomorphic robot, simulating handover operations where the robot is the giver and the human the receiver. For each operation, the human skeleton is obtained with an Intel RealSense D435i camera attached inside the robot’s head. The results shown a great improvement of the human’s right hand prediction and 3D body compared with other methods., Work supported under the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI (MDM-2016-0656), the ROCOTRANSP project (PID2019-106702RB-C21 / AEI /10.13039/501100011033)) and the EU project CANOPIES (H2020- ICT-2020-2-101016906), Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A survey on socially aware robot navigation: Taxonomy and future challenges

  • Singamaneni, Phani Teja
  • Bachiller Burgos, Pilar
  • Manso, Luis J.
  • Garrell Zulueta, Anais
  • Sanfeliu Cortés, Alberto
  • Spalanzani, Anne
  • Alami, Rachid
Socially aware robot navigation is gaining popularity with the increase in delivery and assistive robots. The research is further fueled by a need for socially aware navigation skills in autonomous vehicles to move safely and appropriately in spaces shared with humans. Although most of these are ground robots, drones are also entering the field. In this paper, we present a literature survey of the works on socially aware robot navigation in the past 10 years. We propose four different faceted taxonomies to navigate the literature and examine the field from four different perspectives. Through the taxonomic review, we discuss the current research directions and the extending scope of applications in various domains. Further, we put forward a list of current research opportunities and present a discussion on possible future challenges that are likely to emerge in the field., This work was supported by the Horizon Europe Framework Programme [Grant: euROBIN 101070596]; Agence Nationale de la Recherche [Grant: ANITI ANR-19-PI3A0004]; European project CANOPIES [Grant: H2020ICT-2020-2-101016906]; Spanish Government (MCIN/AEI) and European Union (“NextGenerationEU”/PRTR) [Grant: TED2021-131739-C22]; Spanish Government (MCIN) [Grant: PDC2022-133597-C41]; Spanish Government (MCIN) [Grant: PID2022-142039NA-I00]., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Perception–intention–action cycle in human–robot collaborative tasks: the collaborative lightweight object transportation use-case

  • Domínguez Vidal, José Enrique
  • Rodríguez Linares, Nicolás Adrián
  • Sanfeliu Cortés, Alberto
This study proposes to improve the reliability, robustness and human-like nature of Human–Robot Collaboration (HRC). For that, the classical Perception–Action cycle is extended to a Perception–Intention–Action (PIA) cycle, which includes an Intention stage at the same level as the Perception one, being in charge of obtaining both the implicit and the explicit intention of the human, opposing to classical approaches based on inferring everything from perception. This complete cycle is exposed theoretically including its use of the concept of Situation Awareness, which is shown as a key element for the correct understanding of the current situation and future action prediction. This enables the assignment of roles to the agents involved in a collaborative task and the building of collaborative plans. To visualize the cycle, a collaborative transportation task is used as a use-case. A force-based model is designed to combine the robot’s perception of its environment with the force exerted by the human and other factors in an illustrative way. Finally, a total of 58 volunteers participate in two rounds of experiments. In these, it is shown that the human agrees to explicitly state their intention without undue extra effort and that the human understands that this helps to minimize robot errors or misunderstandings. It is also shown that a system that correctly combines inference with explicit elicitation of the human’s intention is the best rated by the human on multiple parameters related to effective Human–Robot Interaction (HRI), such as perceived safety or trust in the robot., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020- ICT-2020-2-101016906), the MINECO/AEI ROCOTRANSP project (PID2019-106702RB-C21 MCIN/ AEI /10.13039/501100011033) and the JST Moonshot R & D Grant No. JPMJMS2011-85. The first author acknowledges Spanish FPU Grant with ref. FPU19/06582., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

When the inference meets the explicitness or why multimodality can make us forget about the perfect predictor

  • Domínguez-Vidal, José Enrique
  • Sanfeliu Cortés, Alberto
The version of record of this article, first published in International journal of social robotics, is available online at Publisher’s website: http://dx.doi.org/10.1007/s12369-025-01303-9, Although in the literature it is common to find predictors and inference systems that try to predict human intentions, the uncertainty of these models due to the randomness of human behaviour has led some authors to start advocating the use of communication systems that explicitly elicit human intention. In this work, it is analysed the use of four different communication systems with a human-robot collaborative object transportation task as experimental testbed: two intention predictors (one based on force prediction and another with an enhanced velocity prediction algorithm) and two explicit communication methods (a button interface and a voice-command recognition system). These systems were integrated into IVO, a custom mobile social robot equipped with force sensor to detect the force exchange between both agents and LiDAR to detect the environment. The collaborative task required transporting an object over a 5–7 meter distance with obstacles in the middle, demanding rapid decisions and precise physical coordination. 75 volunteers perform a total of 255 executions divided into three groups, testing inference systems in the first round, communication systems in the second, and the combined strategies in the third. The results show that, 1) once sufficient performance is achieved, the human no longer notices and positively assesses technical improvements; 2) the human prefers systems that are more natural to them even though they have higher failure rates; and 3) the preferred option is the right combination of both systems., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Work supported under the European project CANOPIES (H2020- ICT-2020–2-101016906) and the JST Moonshot R&D Grant Number JPMJMS2011-85. The first author acknowledges Spanish FPU grant with ref. FPU19/06582., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

ACHORD: communication-aware multi-robot coordination with intermittent connectivity

  • Saboia Da Silva, Maira
  • Clack, Lillian
  • Thangavelu, Vivek
  • Edlund, Jeffrey
  • Kyohei, Otsu
  • Correa, Gustavo J
  • Varadharajan, Vivek Shankar
  • Santamaria Navarro, Àngel
  • Touma, Thomas
  • Bouman, Amanda
  • Melikyan, Hovhannes
  • Pailevanian, Torkom
  • Kim, Sung-Kyun
  • Archanian, Avak
  • Vaquero, Tiago Stegun
  • Beltrame, Giovanni
  • Napp, Nils
  • Pessin, Gustavo
  • Agha-mohammadi, Ali-akbar
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Communication is an important capability for multi-robot exploration because (1) inter-robot communication (comms) improves coverage efficiency and (2) robot-to-base comms improves situational awareness. Exploring comms-restricted (e.g., subterranean) environments requires a multi-robot system to tolerate and anticipate intermittent connectivity, and to carefully consider comms requirements, otherwise mission-critical data may be lost. In this paper, we describe and analyze ACHORD (Autonomous & Collaborative High-Bandwidth Operations with Radio Droppables), a multi-layer networking solution which tightly co-designs the network architecture and high-level decision-making for improved comms. ACHORD provides bandwidth prioritization and timely and reliable data transfer despite intermittent connectivity. Furthermore, it exposes low-layer networking metrics to the application layer to enable robots to autonomously monitor, map, and extend the network via droppable radios, as well as restore connectivity to improve collaborative exploration. We evaluate our solution with respect to the comms performance in several challenging underground environments including the DARPA SubT Finals competition environment. Our findings support the use of data stratification and flow control to improve bandwidth-usage., Peer Reviewed, Postprint (author's final draft)