Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 116
Encontrada(s) 2 página(s)

Repositorio Digital de la UPF
Publicación Artículo científico (JournalArticle)

MSClique: multiple structure discovery through the maximum weighted clique problem

  • Sanromà, Gerard
  • Penate-Sanchez, Adrian
  • Alquézar, René
  • Serratosa, Francesc
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan
  • González Ballester, Miguel Ángel, 1973-
We present a novel approach for feature correspondence and multiple structure discovery in computer vision. In contrast to existing methods, we exploit the fact that point-sets on the same structure usually lie close to each other, thus forming clusters in the image. Given a pair of input images, we initially extract points of interest and extract hierarchical representations by agglomerative clustering. We use the maximum weighted clique problem to find the set of corresponding clusters with maximum number of inliers representing the multiple structures at the correct scales. Our method is parameter-free and only needs two sets of points along with their tentative correspondences, thus being extremely easy to use. We demonstrate the effectiveness of our method in multiple-structure fitting experiments in both publicly available and in-house datasets. As shown in the experiments, our approach finds a higher number of structures containing fewer outliers compared to state-of-the-art methods., This work has been partially supported by the Spanish Ministry of Economy and Competitiveness under Project DPI-2011-27510 and the ERA-Net Chistera project ViSen PCIN-2013-047. This work has also been partially supported by EU H2020, Call H2020-ICT-23-2014-1 (RIA) under Project 644271. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Los primeros robots manipuladores aéreos

  • Ollero Baturone, Aníbal
  • Viguria, Antidio
  • Maza, Ivan
  • Trujillo Soto, Miguel Ángel
  • Heredia, Guillermo
  • Cano, Raúl
  • Caballero, Fernando
  • Lasagni, Fernando
  • Acosta, J. Á.
  • Santos, Silvia de los
  • Cobano-Suárez, José-Antonio
[Resumen] En este artículo se describen los primeros robots manipuladores aéreos dotados con brazos robóticos de seis grados de libertad y capacidades de percepción y planificación que se han desarrollado en el mundo. Estos robots son el resultado del proyecto ARCAS (Aerial Robotics Cooperative Assembly System) coordinado por el primero de los autores y financiado por el Séptimo Programa Marco de la Unión Europea. Los resultados de este proyecto han comenzado a emplearse, entre otros, en el proyecto H2020 AEROARMS y en el español AEROMAIN. El artículo introduce las diferentes plataformas utilizadas en el proyecto y a continuación resume las principales características de los sistemas de control, percepción y planificación desarrollados. Se resume también la integración y validación del sistema., Este trabajo ha sido financiado parcialmente por el Proyecto Europeo del FP7 ARCAS (ICT-2011-287617) y por el H2020 AEROARMS (ICT-23-2014-644271), así como por el proyecto nacional AEROMAIN DPI2014-59383-C2-1-R del Ministerio de Economía y Competitividad, https://doi.org/10.17979/spudc.9788497498081
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

MSClique: Multiple structure discovery through the maximum weighted clique problem

  • Sanroma, Gerard
  • Peñate-Sánchez, Adrián
  • Alquézar Mancho, Renato
  • Serratosa, Francesc
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan
  • González Ballester, Miguel Angel
This is an open access article distributed under the terms of the Creative Commons Attribution License., We present a novel approach for feature correspondence and multiple structure discovery in computer vision. In contrast to existing methods, we exploit the fact that point-sets on the same structure usually lie close to each other, thus forming clusters in the image. Given a pair of input images, we initially extract points of interest and extract hierarchical representations by agglomerative clustering. We use the maximum weighted clique problem to find the set of corresponding clusters with maximum number of inliers representing the multiple structures at the correct scales. Our method is parameter-free and only needs two sets of points along with their tentative correspondences, thus being extremely easy to use. We demonstrate the effectiveness of our method in multiple-structure fitting experiments in both publicly available and in-house datasets. As shown in the experiments, our approach finds a higher number of structures containing fewer outliers compared to state-of-the-art methods., This work has been partially supported by the Spanish Ministry of Economy and Competitiveness under Project DPI-2011-27510 and the ERA-Net Chistera project ViSen PCIN-2013-047. This work has also been partially supported by EU H2020, Call H2020-ICT-23-2014-1 (RIA) under Project 644271., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Hybrid visual servoing with hierarchical task composition for aerial manipulation

  • Lippiello, Vincenzo
  • Cacace, Jonathan
  • Santamaria-Navarro, Àngel
  • Andrade-Cetto, Juan
  • Trujillo, Miguel Ángel
  • Rodríguez Esteves, Yamnia
  • Viguria, Antidio
In this letter, a hybrid visual servoing with a hierarchical task-composition control framework is described for aerial manipulation, i.e., for the control of an aerial vehicle endowed with a robot arm. The proposed approach suitably combines into a unique hybrid-control framework the main benefits of both image-based and position-based control schemes. Moreover, the underactuation of the aerial vehicle has been explicitly taken into account in a general formulation, together with a dynamic smooth activation mechanism. Both simulation case studies and experiments are presented to demonstrate the performance of the proposed technique., This work was supported by the ARCAS and AEROARMS large-scale integrating projects, funded by the European Community’s under Grants FP7-ICT-287617 and H2020-ICT-644271, respectively., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Discriminative learning of deep convolutional feature point descriptors

  • Simo-Serra, Edgar
  • Trulls, Eduard
  • Ferraz, Luis
  • Kokkinos, Iasonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Trabajo presentado a la International Conference on Computer Vision celebrada en Santiago (España) del 7 al 13 de diciembre de 2015., Deep learning has revolutionalized image-level tasks such as classification, but patch-level tasks, such as correspondence, still rely on hand-crafted features, e.g. SIFT. In this paper we use Convolutional Neural Networks (CNNs) to learn discriminant patch representations and in particular train a Siamese network with pairs of (non-)corresponding patches. We deal with the large number of potential pairs with the combination of a stochastic sampling of the training set and an aggressive mining strategy biased towards patches that are hard to classify. By using the L2 distance during both training and testing we develop 128-D descriptors whose euclidean distances reflect patch similarity, and which can be used as a drop-in replacement for any task involving SIFT. We demonstrate consistent performance gains over the state of the art, and generalize well against scaling and rotation, perspective transformation, non-rigid deformation, and illumination changes. Our descriptors are efficient to compute and amenable to modern GPUs, and are publicly available., This work was partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project ViSen PCIN-2013-047, by EU projects AEROARMS H2020-ICT-2014-1-644271, ISUPPORT H2020-ICT-2014-1-643666 and MOBOT FP7-ICT-2011-600796, and by the ERC project MicroNano., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Matchability prediction for full-search template matching algorithms

  • Peñate-Sánchez, Adrián
  • Porzi, Lorenzo
  • Moreno-Noguer, Francesc
Trabajo presentado a la International Conference on 3D Vision, celebrada en Lyon (Francia) del 19 al 22 de octubre de 2015., While recent approaches have shown that it is possible to do template matching by exhaustively scanning the parameter space, the resulting algorithms are still quite demanding. In this paper we alleviate the computational load of these algorithms by proposing an efficient approach for predicting the match ability of a template, before it is actually performed. This avoids large amounts of unnecessary computations. We learn the match ability of templates by using dense convolutional neural network descriptors that do not require ad-hoc criteria to characterize a template. By using deep learning descriptions of patches we are able to predict match ability over the whole image quite reliably. We will also show how no specific training data is required to solve problems like panorama stitching in which you usually require data from the scene in question. Due to the highly parallelizable nature of this tasks we offer an efficient technique with a negligible computational cost at test time., This work has been partially funded by Spanish Ministry of Economy and Competitiveness under project RobInstruct TIN2014-58178-R and ERA-Net Chistera project ViSen PCIN-2013-047; and by the EU project AEROARMS H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

BASS: Boundary-aware superpixel segmentation

  • Rubio Romano, Antonio
  • Yu, LongLong
  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
Trabajo presentado a la 23rd International Conference on Pattern Recognition, celebrada en Cancún (México) del 5 al 8 de diciembre de 2016., We propose a new superpixel algorithm based on exploiting the boundary information of an image, as objects in images can generally be described by their boundaries. Our proposed approach initially estimates the boundaries and uses them to place superpixel seeds in the areas in which they are more dense. Afterwards, we minimize an energy function in order to expand the seeds into full superpixels. In addition to standard terms such as color consistency and compactness, we propose using the geodesic distance which concentrates small superpixels in regions of the image with more information, while letting larger superpixels cover more homogeneous regions. By both improving the initialization using the boundaries and coherency of the superpixels with geodesic distances, we are able to maintain the coherency of the image structure with fewer superpixels than other approaches. We show the resulting algorithm to yield smaller Variation of Information metrics in seven different datasets while maintaining Undersegmentation Error values similar to the state-of-the-art methods., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project I-DRESS PCIN-2015-147 and by the EU project AEROARMS H2020-ICT-2014-1-644271. A. Rubio is supported by the industrial doctorate grant 2015-DI-010 of the AGAUR., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Learning the hidden human knowledge of UAV pilots when navigating in a cluttered environament for improving Path Plannig

  • Alzugaray, Ignacio
  • Sanfeliu, Alberto
  • Sanfeliu, Alberto
Trabajo presentado a la IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), celebrada en Daejeon (Korea) del 9 al 14 de octubre de 2016., We propose in this work a new model of how the hidden human knowledge (HHK) of UAV pilots can be incorporated in the UAVs path planning generation. We intuitively know that human's pilots barely manage or even attempt to drive the UAV through a path that is optimal attending to some criteria as an optimal planner would suggest. Although human pilots might get close but not reach the optimal path proposed by some planner that optimizes over time or distance, the final effect of this differentiation could be not only surprisingly better, but also desirable. In the best scenario for optimality, the path that human pilots generate would deviate from the optimal path as much as the hidden knowledge that its perceives is injected into the path. The aim of our work is to use real human pilot paths to learn the hidden knowledge using repulsion fields and to incorporate this knowledge afterwards in the environment obstacles as cause of the deviation from optimality. We present a strategy of learning this knowledge based on attractor and repulsors, the learning method and a modified RRT* that can use this knowledge for path planning. Finally we do real-life tests and we compare the resulting paths with and without this knowledge., This work was supported by the Spanish Ministry of Science and Innovation, project Rob-Int-Coop DPI2013-42458-P and AEROARMS European project H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Interactive multiple object learning with scanty human supervision

  • Villamizar, Michael
  • Garrell, Anaís
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
We present a fast and online human-robot interaction approach that progressively learns multiple object classifiers using scanty human supervision. Given an input video stream recorded during the human-robot interaction, the user just needs to annotate a small fraction of frames to compute object specific classifiers based on random ferns which share the same features. The resulting methodology is fast (in a few seconds, complex object appearances can be learned), versatile (it can be applied to unconstrained scenarios), scalable (real experiments show we can model up to 30 different object classes), and minimizes the amount of human intervention by leveraging the uncertainty measures associated to each classifier. We thoroughly validate the approach on synthetic data and on real sequences acquired with a mobile platform in indoor and outdoor scenarios containing a multitude of different objects. We show that with little human assistance, we are able to build object classifiers robust to viewpoint changes, partial occlusions, varying lighting and cluttered backgrounds., This work has been partially funded by the Spanish Ministry of Economy and Competitiveness under projects ERA-Net Chistera project ViSen PCIN-2013-047, RobInstruct TIN2014-58178-R, ROBOT-INT-COOP DPI2013-42458-P, and by the EU project AEROARMS H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Trajectory generation for unmanned aerial manipulators through quadratic programming

  • Rossi, Roberto
  • Santamaria-Navarro, Àngel
  • Andrade-Cetto, Juan
  • Rocco, Paolo
In this paper a trajectory generation approach using quadratic programming is described for aerial manipulation, i.e. for the control of an aerial vehicle equipped with a robot arm. The proposed approach applies the online active set strategy to generate a feasible trajectory of the joints, in order to accomplish a set of tasks with defined bounds and constraint inequalities. The definition of the problem in the acceleration domain allows to integrate and perform a large set of tasks and, as a result, to obtain smooth motion of the joints. A weighting strategy, associated with a normalization procedure, allows to easily define the relative importance of the tasks. This approach is useful to accomplish different phases of a mission with different redundancy resolution strategies. The performance of the proposed technique is demonstrated through real experiments with all the algorithms running onboard in real time. In particular, the aerial manipulator can successfully perform navigation and interaction phases, while keeping motion within prescribed bounds and avoiding collisions with external obstacles., Their work has been partially funded by the EU project AEROARMS H2020-ICT2014-1-644271 and by the Spanish Ministry of Economy and Competitiveness project ROBINSTRUCT TIN2014-58178-R., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Uncalibrated visual servo for unmanned aerial manipulation

  • Santamaria-Navarro, Àngel
  • Grosch, Patrick
  • Lippiello, Vincenzo
  • Solà, Joan
  • Andrade-Cetto, Juan
This paper addresses the problem of autonomous servoing an unmanned redundant aerial manipulator using computer vision. The overactuation of the system is exploited by means of a hierarchical control law, which allows to prioritize several tasks during flight. We propose a safety-related primary task to avoid possible collisions. As a secondary task, we present an uncalibrated image-based visual servo strategy to drive the arm end-effector to a desired position and orientation by using a camera attached to it. In contrast to the previous visual servo approaches, a known value of camera focal length is not strictly required. To further improve flight behavior, we hierarchically add one task to reduce dynamic effects by vertically aligning the arm center of gravity to the multirotor gravitational vector, and another one that keeps the arm close to a desired configuration of high manipulability and avoiding arm joint limits. The performance of the hierarchical control law, with and without activation of each of the tasks, is shown in simulations and in real experiments confirming the viability of such prioritized control scheme for aerial manipulation., This work has been funded by the EU project AEROARMS H2020-ICT2014-1-644271 and by the Spanish Ministry of Economy and Competitiveness project ROBINSTRUCT TIN2014-58178-R., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Searching and tracking people in urban environments with static and dynamic obstacles

  • Goldhoorn, Alex
  • Garrell, Anaís
  • Alquézar Mancho, Renato
  • Sanfeliu, Alberto
Searching and tracking people in crowded urban areas where they can be occluded by static or dynamic obstacles is an important behavior for social robots which assist humans in urban outdoor environments. In this work, we propose a method that can handle in real-time searching and tracking people using a Highest Belief Particle Filter Searcher and Tracker. It makes use of a modified Particle Filter (PF), which, in contrast to other methods, can do both searching and tracking of a person under uncertainty, with false negative detections, lack of a person detection, in continuous space and real-time. Moreover, this method uses dynamic obstacles to improve the predicted possible location of the person. Comparisons have been made with our previous method, the Adaptive Highest Belief Continuous Real-time POMCP Follower, in different conditions and with dynamic obstacles. Real-life experiments have been done during two weeks with a mobile service robot in two urban environments of Barcelona with other people walking around., This work has been partially funded by the EU project AEROARMS European projectH2020-ICT-2014-1-644271 and the CICYT project DPI2013-42458-P., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Nonlinear model predictive control for aerial manipulation

  • Lunni, Dario
  • Santamaria-Navarro, Àngel
  • Rossi, Roberto
  • Rocco, Paolo
  • Bascetta, Luca
  • Andrade-Cetto, Juan
Trabajo presentado a la International Conference on Unmanned Aircraft Systems (ICUAS), celebrada en Miami (US) del 13 al 16 de junio de 2017., This paper presents a nonlinear model predictive controller to follow desired 3D trajectories with the end effector of an unmanned aerial manipulator (i.e., a multirotor with a serial arm attached). To the knowledge of the authors, this is the first time that such controller runs online and on board a limited computational unit to drive a kinematically augmented aerial vehicle. Besides the trajectory following target, we explore the possibility of accomplishing other tasks during flight by taking advantage of the system redundancy. We define several tasks designed for aerial manipulators and show in simulation case studies how they can be achieved by either a weighting strategy, within a main optimization process, or a hierarchical approach consisting on nested optimizations. Moreover, experiments are presented to demonstrate the performance of such controller in a real robot., Their work was partially funded by the EU project AEROARMS H2020-ICT-2014-1-644271, and by the Spanish Ministry of Economy and Competitiveness project ROBINSTRUCT TIN2014-58178-R., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

A flexible hardware-in-the-loop architecture for UAVs

  • Lepej, Peter
  • Santamaria-Navarro, Àngel
  • Solà, Joan
Trabajo presentado a la International Conference on Unmanned Aircraft Systems (ICUAS), celebrada en Miami (US) del 13 al 16 de junio de 2017., As robotic technology matures, fully autonomous robots become a realistic possibility, but demand very complex solutions to be rapidly engineered. In order to be able to quickly set up a working autonomous system, and to reduce the gap between simulated and real experiments, we propose a modular, upgradeable and flexible hardware-in-the-loop (HIL) architecture, which hybridizes the simulated and real settings. We take as use case the autonomous exploration of dense forests with UAVs, with the aim of creating useful maps for forest inspection, cataloging, or to compute other metrics such as total wood volume. As the first step in the development of the full system, in this paper we implement a fraction of this architecture, comprising assisted localization, and automatic methods for mapping, planning and motion execution. Specifically we are able to simulate the use of a 3D LIDAR endowed below an actual UAV autonomously navigating among simulated obstacles, thus the platform safety is not compromised. The full system is modular and takes profit of pieces either publicly available or easily programmed. We highlight the flexibility of the proposed HIL architecture to rapidly configure different experimental setups with a UAV in challenging terrain. Moreover, it can be extended to other robotic fields without further design. The HIL system uses the multi-platform ROS capabilities and only needs a motion capture system as external extra hardware, which is becoming standard equipment in all research labs dealing with mobile robots., This work has been funded by the EU project AEROARMS H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Teaching robot’s proactive behavior using human assistance

  • Garrell, Anaís
  • Villamizar, Michael
  • Moreno-Noguer, Francesc
  • Sanfeliu, Alberto
In recent years, there has been a growing interest in enabling autonomous social robots to interact with people. However, many questions remain unresolved regarding the social capabilities robots should have in order to perform this interaction in an ever more natural manner. In this paper, we tackle this problem through a comprehensive study of various topics involved in the interaction between a mobile robot and untrained human volunteers for a variety of tasks. In particular, this work presents a framework that enables the robot to proactively approach people and establish friendly interaction. To this end, we provided the robot with several perception and action skills, such as that of detecting people, planning an approach and communicating the intention to initiate a conversation while expressing an emotional status. We also introduce an interactive learning system that uses the person’s volunteered assistance to incrementally improve the robot’s perception skills. As a proof of concept, we focus on the particular task of online face learning and recognition. We conducted real-life experiments with our Tibi robot to validate the framework during the interaction process. Within this study, several surveys and user studies have been realized to reveal the social acceptability of the robot within the context of different tasks., This work was supported by the Spanish Ministry of Science and Innovation, project Rob-Int-Coop DPI2013-42458-P and AEROARMS European project H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

PL-SLAM: Real-time monocular visual SLAM with points and lines

  • Pumarola, Albert
  • Vakhitov, Alexander
  • Agudo Martínez, Antonio
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Trabajo presentado a la IEEE International Conference on Robotics and Automation (ICRA), celebrada en Singapur del 29 de mayo al 3 de junio de 2017., Low textured scenes are well known to be one of the main Achilles heels of geometric computer vision algorithms relying on point correspondences, and in particular for visual SLAM. Yet, there are many environments in which, despite being low textured, one can still reliably estimate line-based geometric primitives, for instance in city and indoor scenes, or in the so-called ``Manhattan worlds'', where structured edges are predominant. In this paper we propose a solution to handle these situations. Specifically, we build upon ORB-SLAM, presumably the current state-of-the-art solution both in terms of accuracy as efficiency, and extend its formulation to simultaneously handle both point and line correspondences. We propose a solution that can even work when most of the points are vanished out from the input images, and, interestingly it can be initialized from solely the detection of line correspondences in three consecutive frames. We thoroughly evaluate our approach and the new initialization strategy on the TUM RGB-D benchmark and demonstrate that the use of lines does not only improve the performance of the original ORB-SLAM solution in poorly textured frames, but also systematically improves it in sequence frames combining points and lines, without compromising the efficiency., This work has been partially supported by the EU project AEROARMS H2020-ICT-2014-1-644271, by the MINECO projects RobInstruct TIN2014-58178-R and Rob-Int-Coop DPI2013-42458-P, by the ERA-Net Chistera project IDRESS PCIN-2015-147 and by the Russian MES grant RFMEFI61516X0003., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Learning depth-aware deep representations for robotic perception

  • Porzi, Lorenzo
  • Rota Buló, Samuel
  • Peñate-Sánchez, Adrián
  • Ricci, Elisa
  • Moreno-Noguer, Francesc
Exploiting RGB-D data by means of Convolutional Neural Networks (CNNs) is at the core of a number of robotics applications, including object detection, scene semantic segmentation and grasping. Most existing approaches, however, exploit RGB-D data by simply considering depth as an additional input channel for the network. In this paper we show that the performance of deep architectures can be boosted by introducing DaConv, a novel, general-purpose CNN block which exploits depth to learn scale-aware feature representations. We demonstrate the benefits of DaConv on a variety of robotics oriented tasks, involving affordance detection, object coordinate regression and contour detection in RGB-D images. In each of these experiments we show the potential of the proposed block and how it can be readily integrated into existing CNN architectures., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project I-DRESS PCIN-2015-147, by the EU project AEROARMS H2020-ICT-2014-1-644271 and by the EU project SECOND HANDS H2020-ICT-2014-1-643950., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Depth-aware convolutional neural networks for accurate 3D pose estimation in RGB-D images

  • Porzi, Lorenzo
  • Peñate-Sánchez, Adrián
  • Ricci, Elisa
  • Moreno-Noguer, Francesc
Trabajo presentado a la IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), celebrada en Vancouver (Canada) del 24 al 28 de septiembre de 2017., Most recent approaches to 3D pose estimation from RGB-D images address the problem in a two-stage pipeline. First, they learn a classifier-typically a random forest-to predict the position of each input pixel on the object surface. These estimates are then used to define an energy function that is minimized w.r.t. the object pose. In this paper, we focus on the first stage of the problem and propose a novel classifier based on a depth-aware Convolutional Neural Network. This classifier is able to learn a scale-adaptive regression model that yields very accurate pixel-level predictions, allowing to finally estimate the pose using a simple RANSAC-based scheme, with no need to optimize complex ad hoc energy functions. Our experiments on publicly available datasets show that our approach achieves remarkable improvements over state-of-the-art methods., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project I-DRESS PCIN-2015-147, by the EU project AEROARMS H2020-ICT-2014-1-644271, by the EU project SECOND HANDS H2020-ICT-2014-1-643950 and by the Spanish State Research Agency through the Mara de Maeztu Seal of Excellence to IRI MDM-2016-0656., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Aerial social force model: A new framework to accompany people using autonomous flying robots

  • Garrell, Anaís
  • Garza Elizondo, Luis Alberto
  • Villamizar, Michael
  • Herrero, Fernando
  • Sanfeliu, Alberto
Trabajo presentado a la IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), celebrada en Vancouver (Canada) del 24 al 28 de septiembre de 2017., In this paper, we proposed a novel Aerial Social Force Model (ASFM) that allows autonomous flying robots to accompany humans in urban environments in a safe and comfortable manner. To date, we are not aware of other state-of-the-art method that accomplish this task. The proposed approach is a 3D version of the Social Force Model (SFM) for the field of aerial robots which includes an interactive human-robot navigation scheme capable of predicting human motions and intentions so as to safely accompany them to their final destination. ASFM also introduces a new metric to fine-tune the parameters of the force model, and to evaluate the performance of the aerial robot companion based on comfort and distance between the robot and humans. The presented approach is extensively validated in diverse simulations and real experiments, and compared against other similar works in the literature. ASFM attains remarkable results and shows that it is a valuable framework for social robotics applications, such as guiding people or human-robot interaction., This work is supported by the Spanish State Research Agency through the Mara de Maeztu Seal of Excellence to IRI MDM-2016-0656. This work was supported by the Spanish Ministry of Science and Innovation, project ColRobTransp DPI2016-78957-R (AEI/FEDER, UE) and AEROARMS European project H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Multi-modal joint embedding for fashion product retrieval

  • Rubio Romano, Antonio
  • Yu, LongLong
  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
Trabajo presentado a la IEEE International Conference on Image Processing (ICIP), celebrada en Beijing (China) del 17 al 20 de septiembre de 2017., Finding a product in the fashion world can be a daunting task. Everyday, e-commerce sites are updating with thousands of images and their associated metadata (textual information), deepening the problem, akin to finding a needle in a haystack. In this paper, we leverage both the images and textual meta-data and propose a joint multi-modal embedding that maps both the text and images into a common latent space. Distances in the latent space correspond to similarity between products, allowing us to effectively perform retrieval in this latent space, which is both efficient and accurate. We train this embedding using large-scale real world e-commerce data by both minimizing the similarity between related products and using auxiliary classification networks to that encourage the embedding to have semantic meaning. We compare against existing approaches and show significant improvements in retrieval tasks on a large-scale e-commerce dataset. We also provide an analysis of the different metadata., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project I-DRESS PCIN-2015-147 and by the EU project iAEROARMS H2020-ICT-2014-1-644271. A.Rubio is supported by the industrial doctorate grant 2015-DI-010 of the AGAUR., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Joint coarse-and-fine reasoning for deep optical flow

  • Vaquero, Victor
  • Ros, German
  • Moreno-Noguer, Francesc
  • López, Antonio M.
  • Sanfeliu, Alberto
Trabajo presentado a la IEEE International Conference on Image Processing (ICIP), celebrada en Beijing (China) del 17 al 20 de septiembre de 2017., We propose a novel representation for dense pixel-wise estimation tasks using CNNs that boosts accuracy and reduces training time, by explicitly exploiting joint coarse-and-fine reasoning. The coarse reasoning is performed over a discrete classification space to obtain a general rough solution, while the fine details of the solution are obtained over a continuous regression space. In our approach both components are jointly estimated, which proved to be beneficial for improving estimation accuracy. Additionally, we propose a new network architecture, which combines coarse and fine components by treating the fine estimation as a refinement built on top of the coarse solution, and therefore adding details to the general prediction. We apply our approach to the challenging problem of optical flow estimation and empirically validate it against state-of-the-art CNN-based solutions trained from scratch and tested on large optical flow datasets., This work was partially supported by European AEROARMS project (H2020-ICT-2014-1-644271) and CICYT projects ColRobTransp (DPI2016-78957-R), ROBINSTRUCT (TIN2014-58178-R)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Random clustering ferns for multimodal object recognition

  • Villamizar, Michael
  • Garrell, Anaís
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
We propose an efficient and robust method for the recognition of objects exhibiting multiple intra-class modes, where each one is associated with a particular object appearance. The proposed method, called random clustering ferns, combines synergically a single and real-time classifier, based on the boosted assembling of extremely randomized trees (ferns), with an unsupervised and probabilistic approach in order to recognize efficiently object instances in images and discover simultaneously the most prominent appearance modes of the object through tree-structured visual words. In particular, we use boosted random ferns and probabilistic latent semantic analysis to obtain a discriminative and multimodal classifier that automatically clusters the response of its randomized trees in function of the visual object appearance. The proposed method is validated extensively in synthetic and real experiments, showing that the method is capable of detecting objects with diverse and complex appearance distributions in real-time performance., This work has been partially funded by the Spanish Ministry of Economy and Competitiveness under projects ERA-Net Chistera project ViSen PCIN-2013-047, RobInstruct TIN2014-58178-R, ROBOT-INT-COOP DPI2013-42458-P, and by the EU project AEROARMS H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Boosted Random Ferns for object detection

  • Villamizar, Michael
  • Andrade-Cetto, Juan
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
In this paper we introduce the Boosted Random Ferns (BRFs) to rapidly build discriminative classifiers for learning and detecting object categories. At the core of our approach we use standard random ferns, but we introduce four main innovations that let us bring ferns from an instance to a category level, and still retain efficiency. First, we define binary features on the histogram of oriented gradients-domain (as opposed to intensity-), allowing for a better representation of intra-class variability. Second, both the positions where ferns are evaluated within the sliding window, and the location of the binary features for each fern are not chosen completely at random, but instead we use a boosting strategy to pick the most discriminative combination of them. This is further enhanced by our third contribution, that is to adapt the boosting strategy to enable sharing of binary features among different ferns, yielding high recognition rates at a low computational cost. And finally, we show that training can be performed online, for sequentially arriving images. Overall, the resulting classifier can be very efficiently trained, densely evaluated for all image locations in about 0.1 seconds, and provides detection rates similar to competing approaches that require expensive and significantly slower processing times. We demonstrate the effectiveness of our approach by thorough experimentation in publicly available datasets in which we compare against state-of-the-art, and for tasks of both 2D detection and 3D multi-view estimation., This work was partially supported by the Catalan Agency for Management of University and Research Grants for the Consolidated Group VIS (2014 SGR 897). A.S. acknowledges support also from project ColRobTransp (DPI2016-78957-R), and J.A. and F.M. from project RobInstruct (TIN2014-58178-R), both funded by the Spanish Ministry of Economy, Industry and Competitiveness. J.A., A.S., and F.M. were also partially supported by the European Union projects AEROARMS (H2020-ICT-2014-1-644271) and LOGIMATIC (H2020-Galileo-2015-1-687534)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Planar PØP: Feature-less pose estimation with applications in UAV localization

  • Amor-Martinez, Adrian
  • Santamaria-Navarro, Àngel
  • Herrero, Fernando
  • Ruiz Moreno, Alberto
  • Sanfeliu, Alberto
Trabajo presentado al IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), celebrado en Lausanne (Suiza) del 23 al 27 de octubre de 2016., We present a featureless pose estimation method that, in contrast to current Perspective-n-Point (PnP) approaches, it does not require n point correspondences to obtain the camera pose, allowing for pose estimation from natural shapes that do not necessarily have distinguished features like corners or intersecting edges. Instead of using n correspondences (e.g. extracted with a feature detector) we will use the raw polygonal representation of the observed shape and directly estimate the pose in the pose-space of the camera. This method compared with a general PnP method, does not require n point correspondences neither a priori knowledge of the object model (except the scale), which is registered with a picture taken from a known robot pose. Moreover, we achieve higher precision because all the information of the shape contour is used to minimize the area between the projected and the observed shape contours. To emphasize the non-use of n point correspondences between the projected template and observed contour shape, we call the method Planar PØP. The method is shown both in simulation and in a real application consisting on a UAV localization where comparisons with a precise ground-truth are provided., This work has been partially funded by the AEROARMS EU project H2020-ICT-2014-1-644271, the CICYT project DPI2013-42458-P, and by Spanish MINECO and EU FEDER grant TIN2015-66972-C5-3-R., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Tesis doctoral (DoctoralThesis)

Visual guidance of unmanned aerial manipulators

  • Santamaria-Navarro, Àngel
A thesis submitted to the Universitat Politècnica de Catalunya for the degree of Doctor of Philosophy. Doctoral Programme: Automatic Control, Robotics and Computer Vision. This thesis has been completed at: Institut de Robòtica i Informàtica Industrial, CSIC-UPC., [EN]: The ability to fly has greatly expanded the possibilities for robots to perform surveillance, inspection or map generation tasks. Yet it was only in recent years that research in aerial robotics was mature enough to allow active interactions with the environment. The robots responsible for these interactions are called aerial manipulators and usually combine a multirotor platform and one or more robotic arms. The main objective of this thesis is to formalize the concept of aerial manipulator and present guidance methods, using visual information, to provide them with autonomous functionalities. A key competence to control an aerial manipulator is the ability to localize it in the environment. Traditionally, this localization has required external infrastructure of sensors (e.g., GPS or IR cameras), restricting the real applications. Furthermore, localization methods with on-board sensors, exported from other robotics fields such as simultaneous localization and mapping (SLAM), require large computational units becoming a handicap in vehicles where size, load, and power consumption are important restrictions. In this regard, this thesis proposes a method to estimate the state of the vehicle (i.e., position, orientation, velocity and acceleration) by means of on-board, low-cost, light-weight and high-rate sensors. With the physical complexity of these robots, it is required to use advanced control techniques during navigation. Thanks to their redundancy on degrees-of-freedom, they offer the possibility to accomplish not only with mobility requirements but with other tasks simultaneously and hierarchically, prioritizing them depending on their impact to the overall mission success. In this work we present such control laws and define a number of these tasks to drive the vehicle using visual information, guarantee the robot integrity during flight, and improve the platform stability or increase arm operability. The main contributions of this research work are threefold: (1) Present a localization technique to allow autonomous navigation, this method is specifically designed for aerial platforms with size, load and computational burden restrictions. (2) Obtain control commands to drive the vehicle using visual information (visual servo). (3) Integrate the visual servo commands into a hierarchical control law by exploiting the redundancy of the robot to accomplish secondary tasks during flight. These tasks are specific for aerial manipulators and they are also provided. All the techniques presented in this document have been validated throughout extensive experimentation with real robotic platforms., [CA]: La capacitat de volar ha incrementat molt les possibilitats dels robots per a realitzar tasques de vigilància, inspecció o generació de mapes. Tot i això, no és fins fa pocs anys que la recerca en robòtica aèria ha estat prou madura com per començar a permetre interaccions amb l’entorn d’una manera activa. Els robots per a fer-ho s’anomenen manipuladors aeris i habitualment combinen una plataforma multirotor i un braç robòtic. L’objectiu d’aquesta tesi és formalitzar el concepte de manipulador aeri i presentar mètodes de guiatge, utilitzant informació visual, per dotar d’autonomia aquest tipus de vehicles. Una competència clau per controlar un manipulador aeri és la capacitat de localitzar-se en l’entorn. Tradicionalment aquesta localització ha requerit d’infraestructura sensorial externa (GPS, càmeres IR, etc.), limitant així les aplicacions reals. Pel contrari, sistemes de localització exportats d’altres camps de la robòtica basats en sensors a bord, com per exemple mètodes de localització i mapejat simultànis (SLAM), requereixen de gran capacitat de còmput, característica que penalitza molt en vehicles on la mida, pes i consum elèctric son grans restriccions. En aquest sentit, aquesta tesi proposa un mètode d’estimació d’estat del robot (posició, velocitat, orientació i acceleració) a partir de sensors instal·lats a bord, de baix cost, baix consum computacional i que proporcionen mesures a alta freqüència. Degut a la complexitat física d’aquests robots, és necessari l’ús de tècniques de control avançades. Gràcies a la seva redundància de graus de llibertat, aquests robots ens ofereixen la possibilitat de complir amb els requeriments de mobilitat i, simultàniament, realitzar tasques de manera jeràrquica, ordenant-les segons l’impacte en l’acompliment de la missió. En aquest treball es presenten aquestes lleis de control, juntament amb la descripció de tasques per tal de guiar visualment el vehicle, garantir la integritat del robot durant el vol, millorar de l’estabilitat del vehicle o augmentar la manipulabilitat del braç. Aquesta tesi es centra en tres aspectes fonamentals: (1) Presentar una tècnica de localització per dotar d’autonomia el robot. Aquest mètode està especialment dissenyat per a plataformes amb restriccions de capacitat computacional, mida i pes. (2) Obtenir les comandes de control necessàries per guiar el vehicle a partir d’informació visual. (3) Integrar aquestes accions dins una estructura de control jeràrquica utilitzant la redundància del robot per complir altres tasques durant el vol. Aquestes tasques son específiques per a manipuladors aeris i també es defineixen en aquest document. Totes les tècniques presentades en aquesta tesi han estat avaluades de manera experimental amb plataformes robòtiques reals., This work has been partially supported by an FPI-UPC grant (87-915) of the Universitat Politècnica de Catalunya under the EU project ARCAS (FP7-ICT-287617), and also funded from the EU project AEROARMS (H2020-ICT-2014-1-644271), Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Autonomous navigation of micro aerial vehicles using high-rate and low-cost sensors

  • Santamaria-Navarro, Àngel
  • Loianno, Giuseppe
  • Solà, Joan
  • Kumar, Vijay
  • Andrade-Cetto, Juan
The combination of visual and inertial sensors for state estimation has recently found wide echo in the robotics community, especially in the aerial robotics field, due to the lightweight and complementary characteristics of the sensors data. However, most state estimation systems based on visual-inertial sensing suffer from severe processor requirements, which in many cases make them impractical. In this paper, we propose a simple, low-cost and high rate method for state estimation enabling autonomous flight of micro aerial vehicles, which presents a low computational burden. The proposed state estimator fuses observations from an inertial measurement unit, an optical flow smart camera and a time-of-flight range sensor. The smart camera provides optical flow measurements up to a rate of 200 Hz, avoiding the computational bottleneck to the main processor produced by all image processing requirements. To the best of our knowledge, this is the first example of extending the use of these smart cameras from hovering-like motions to odometry estimation, producing estimates that are usable during flight times of several minutes. In order to validate and defend the simplest algorithmic solution, we investigate the performances of two Kalman filters, in the extended and error-state flavors, alongside with a large number of algorithm modifications defended in earlier literature on visual-inertial odometry, showing that their impact on filter performance is minimal. To close the control loop, a non-linear controller operating in the special Euclidean group SE(3) is able to drive, based on the estimated vehicle’s state, a quadrotor platform in 3D space guaranteeing the asymptotic stability of 3D position and heading. All the estimation and control tasks are solved on board and in real time on a limited computational unit. The proposed approach is validated through simulations and experimental results, which include comparisons with ground-truth data provided by a motion capture system. For the benefit of the community, we make the source code public., This work was funded by the project ROBINSTRUCT (TIN2014-58178-R) and the Ramón y Cajal posdoctoral fellowship (RYC-2012-11604) from the Spanish Ministry of Economy and Competitiveness, by the Spanish State Research Agency through the María de Maeztu Seal of Excellence to IRI (MDM-2016-0656) and by the EU H2020 project AEROARMS (H2020-ICT-2014-1-644271)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Geometry-aware network for non-rigid shape prediction from a single view

  • Pumarola, Albert
  • Agudo Martínez, Antonio
  • Porzi, Lorenzo
  • Sanfeliu, Alberto
  • Lepetit, Vincent
  • Moreno-Noguer, Francesc
Trabajo presentado en la IEEE/CVF Conference on Computer Vision and Pattern Recognition, celebrada en Salt Lake City (UT, USA), del 18 al 23 de junio de 2018, We propose a method for predicting the 3D shape of a deformable surface from a single view. By contrast with previous approaches, we do not need a pre-registered template of the surface, and our method is robust to the lack of texture and partial occlusions. At the core of our approach is a geometry-aware deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image. We train this architecture in an end-to-end manner using a large dataset of synthetic renderings of shapes under different levels of deformation, material properties, textures and lighting conditions. We evaluate our approach on a test split of this dataset and available real benchmarks, consistently improving state-of-the-art solutions with a significantly lower computational time., This work is supported in part by a Google Faculty Research Award, by the Spanish Ministry of Science and Innovation under projects HuMoUR TIN2017- 90086-R, ColRobTransp DPI2016-78957 and Mar´ıa de Maeztu Seal of Excellence MDM-2016-0656; and by the EU project AEROARMS ICT-2014-1-644271. We also thank Nvidia for hardware donation., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Unsupervised person image synthesis in arbitrary poses

  • Pumarola, Albert
  • Agudo Martínez, Antonio
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Trabajo presentado en la IEEE/CVF Conference on Computer Vision and Pattern Recognition, celebrada en Salt Lake City (UT, USA), del 18 al 23 de junio de 2018, We present a novel approach for synthesizing photorealistic images of people in arbitrary poses using generative adversarial learning. Given an input image of a person and a desired pose represented by a 2D skeleton, our model renders the image of the same person under the new pose, synthesizing novel views of the parts visible in the input image and hallucinating those that are not seen. This problem has recently been addressed in a supervised manner [16, 35], i.e., during training the ground truth images under the new poses are given to the network. We go beyond these approaches by proposing a fully unsupervised strategy. We tackle this challenging scenario by splitting the problem into two principal subtasks. First, we consider a pose conditioned bidirectional generator that maps back the initially rendered image to the original pose, hence being directly comparable to the input image without the need to resort to any training image. Second, we devise a novel loss function that incorporates content and style terms, and aims at producing images of high perceptual quality. Extensive experiments conducted on the DeepFashion dataset demonstrate that the images rendered by our model are very close in appearance to those obtained by fully supervised approaches., This work is supported in part by a Google Faculty Research Award, by the Spanish Ministry of Science and Innovation under projects HuMoUR TIN2017- 90086-R, ColRobTransp DPI2016-78957 and Mar´ıa de Maeztu Seal of Excellence MDM-2016-0656; and by the EU project AEROARMS ICT-2014-1-644271. We also thank Nvidia for hardware donation under the GPU Grant Program., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

GANimation: Anatomically-aware facial animation from a single image

  • Pumarola, Albert
  • Agudo Martínez, Antonio
  • Martínez, Aleix M.
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Trabajo presentado en la 15th European Conference, celebrada en Munich, (Germany) del 8 al 14 de septiembre de 2018, Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for task of facial expression synthesis. The most successful architecture is StarGAN, that conditions GANs’ generation process with images of a specific domain, namely a set of images of persons sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combine several of them. Additionally, we propose a fully unsupervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit attention mechanisms that make our network robust to changing backgrounds and lighting conditions. Extensive evaluation show that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild., This work is partially supported by the Spanish Ministry of Economy and Competitiveness under projects HuMoUR TIN2017-90086-R, ColRobTransp DPI2016-78957 and María de Maeztu Seal of Excellence MDM-2016-0656; by the EU project AEROARMS ICT-2014-1-644271; and by the Grant R01-DC- 014498 of the National Institute of Health. We also thank Nvidia for hardware donation under the GPU Grant Program., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

GANimation: One-Shot Anatomically Consistent Facial Animation

  • Pumarola, Albert
  • Agudo Martínez, Antonio
  • Martínez, Aleix M.
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Recent advances in generative adversarial networks (GANs) have shown impressive results for the task of facial expression synthesis. The most successful architecture is StarGAN (Choi et al. in CVPR, 2018), that conditions GANs’ generation process with images of a specific domain, namely a set of images of people sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content and granularity of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on action units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combining several of them. Additionally, we propose a weakly supervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit a novel self-learned attention mechanism that makes our network robust to changing backgrounds, lighting conditions and occlusions. Extensive evaluation shows that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild. The code of this work is publicly available at https://github.com/albertpumarola/GANimation., This work is partially supported by an Amazon Research Award, by the Spanish Ministry of Economy and Competitiveness under Projects HuMoUR TIN2017-90086-R, ColRobTransp DPI2016-78957 and María de Maeztu Seal of Excellence MDM-2016-0656; by the EU Project AEROARMS ICT-2014-1-644271; and by the Grant R01-DC- 014498 of the National Institute of Health. We also thank Nvidia for hardware donation under the GPU Grant Program.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Tesis doctoral (DoctoralThesis)

Bridging the gap between reconstruction and synthesis

  • Pumarola, Albert
Tesis doctoral presentada para lograr el título de Doctor por la Universidad Politécnica de Cataluña, programa de doctorado: Automatic Control, Robotics and Computer Vision.--2021-10-13, [EN] 3D reconstruction and image synthesis are two of the main pillars in computer vision. Early works focused on simple tasks such as multi-view reconstruction and texture synthesis. With the spur of Deep Learning, the field has rapidly progressed, making it possible to achieve more complex and high level tasks. For example, the 3D reconstruction results of traditional multi-view approaches are currently obtained with single view methods. Similarly, early pattern based texture synthesis works have resulted in techniques that allow generating novel high-resolution images. In this thesis we have developed a hierarchy of tools that cover all these range of problems, lying at the intersection of computer vision, graphics and machine learning. We tackle the problem of 3D reconstruction and synthesis in the wild. Importantly, we advocate for a paradigm in which not everything should be learned. Instead of applying Deep Learning naively we propose novel representations, layers and architectures that directly embed prior 3D geometric knowledge for the task of 3D reconstruction and synthesis. We apply these techniques to problems including scene/person reconstruction and photo-realistic rendering. We first address methods to reconstruct a scene and the clothed people in it while estimating the camera position. Then, we tackle image and video synthesis for clothed people in the wild. Finally, we bridge the gap between reconstruction and synthesis under the umbrella of a unique novel formulation. Extensive experiments conducted along this thesis show that the proposed techniques improve the performance of Deep Learning models in terms of the quality of the reconstructed 3D shapes / synthesised images, while reducing the amount of supervision and training data required to train them. In summary, we provide a variety of low, mid and high level algorithms that can be used to incorporate prior knowledge into different stages of the Deep Learning pipeline and improve performance in tasks of 3D reconstruction and image synthesis., [CAT] La reconstrucció 3D i la síntesi d'imatges són dos dels pilars fonamentals en visió per computador. Els estudis previs es centren en tasques senzilles com la reconstrucció amb informació multi-càmera i la síntesi de textures. Amb l'aparició del "Deep Learning", aquest camp ha progressat ràpidament, fent possible assolir tasques molt més complexes. Per exemple, per obtenir una reconstrucció 3D, tradicionalment s'utilitzaven mètodes multi-càmera, en canvi ara, es poden obtenir a partir d'una sola imatge. De la mateixa manera, els primers treballs de síntesi de textures basats en patrons han donat lloc a tècniques que permeten generar noves imatges completes en alta resolució. En aquesta tesi, hem desenvolupat una sèrie d'eines que cobreixen tot aquest ventall de problemes, situats en la intersecció entre la visió per computador, els gràfics i l'aprenentatge automàtic. Abordem el problema de la reconstrucció i la síntesi 3D en el món real. És important destacar que defensem un paradigma on no tot s'ha d'aprendre. Enlloc d'aplicar el "Deep Learning" de forma naïve, proposem representacions novedoses i arquitectures que incorporen directament els coneixements geomètrics ja existents per a aconseguir la reconstrucció 3D i la síntesi d'imatges. Nosaltres apliquem aquestes tècniques a problemes com ara la reconstrucció d'escenes/persones i a la renderització d'imatges fotorealistes. Primer abordem els mètodes per reconstruir una escena, les persones vestides que hi ha i la posició de la càmera. A continuació, abordem la síntesi d'imatges i vídeos de persones vestides en situacions quotidianes. I finalment, aconseguim, a través d'una nova formulació única, connectar la reconstrucció amb la síntesi. Els experiments realitzats al llarg d'aquesta tesi demostren que les tècniques proposades milloren el rendiment dels models de "Deepp Learning" pel que fa a la qualitat de les reconstruccions i les imatges sintetitzades alhora que redueixen la quantitat de dades necessàries per entrenar-los. En resum, proporcionem una varietat d'algoritmes de baix, mitjà i alt nivell que es poden utilitzar per incorporar els coneixements previs a les diferents etapes del "Deep Learning" i millorar el rendiment en tasques de reconstrucció 3D i síntesi d'imatges., This work has been partially supported by: - EU project AEROARMS - Aerial Robotics Inspection System, 2015-2019. Code: H2020-ICT-2014- 1-644271 - National Project ColRobTransp - Robotics Last Mile Transportation in Urban Areas, 2016-2019. Code: DPI2016-78957-R - Google Research Award - Geometry-aware CNNs for Non-Rigid Reconstruction, 2017-2018 - Amazon Research Award - Geometry-aware 3D Human Body Animation from Still Photos, 2018- 2019 - Google Research Award - GANimation3D: Unsupervised 3D Face Animation from Monocular Images, 2019-2020
DIGITAL.CSIC. Repositorio Institucional del CSIC
Datos de investigación Set de datos (Dataset)

AEROARMS - Image Dataset for the Crawler Indirect Detection through its Cage

  • Laplaza Galindo, Javier
  • Pumarola, Albert
  • Sanfeliu, Alberto
Dataset containing images and ground-truth position of the crawler's cage used in the AEROARMS project experiments, European Commission: TERRINet - The European Robotics Research Infrastructure Network (730994) AEROARMS - AErial RObotic system integrating multiple ARMS and advanced manipulation capabilities for inspection and maintenance (644271), Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Datos de investigación Set de datos (Dataset)

AEROARMS - Crawler Direct Detection Image Dataset

  • Pumarola, Albert
  • Andrade-Cetto, Juan
  • Sanfeliu, Alberto
Dataset containing images and ground-truth position of the crawler used in the AEROARMS project experiments, European Commission: TERRINet - The European Robotics Research Infrastructure Network (730994) AEROARMS - AErial RObotic system integrating multiple ARMS and advanced manipulation capabilities for inspection and maintenance (644271), Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Unsupervised person image synthesis in arbitrary poses

  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Moreno-Noguer, Francesc
© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, We present a novel approach for synthesizing photo-realistic images of people in arbitrary poses using generative adversarial learning. Given an input image of a person and a desired pose represented by a 2D skeleton, our model renders the image of the same person under the new pose, synthesizing novel views of the parts visible in the input image and hallucinating those that are not seen. This problem has recently been addressed in a supervised manner, i.e., during training the ground truth images under the new poses are given to the network. We go beyond these approaches by proposing a fully unsupervised strategy. We tackle this challenging scenario by splitting the problem into two principal subtasks. First, we consider a pose conditioned bidirectional generator that maps back the initially rendered image to the original pose, hence being directly comparable to the input image without the need to resort to any training image. Second, we devise a novel loss function that incorporates content and style terms, and aims at producing images of high perceptual quality. Extensive experiments conducted on the DeepFashion dataset demonstrate that the images rendered by our model are very close in appearance to those obtained by fully supervised approaches., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

BASS: boundary-aware superpixel segmentation

  • Rubio Romano, Antonio
  • Yu, Longlong
  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We propose a new superpixel algorithm based on exploiting the boundary information of an image, as objects in images can generally be described by their boundaries. Our proposed approach initially estimates the boundaries and uses them to place superpixel seeds in the areas in which they are more dense. Afterwards, we minimize an energy function in order to expand the seeds into full superpixels. In addition to standard terms such as color consistency and compactness, we propose using the geodesic distance which concentrates small superpixels in regions of the image with more information, while letting larger superpixels cover more homogeneous regions. By both improving the initialization using the boundaries and coherency of the superpixels with geodesic distances, we are able to maintain the coherency of the image structure with fewer superpixels than other approaches. We show the resulting algorithm to yield smaller Variation of Information metrics in seven different datasets while maintaining Undersegmentation Error values similar to the state-of-the-art methods., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

A flexible hardware-in-the-loop architecture for UAVs

  • Lepej, Peter
  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Solà, Joan
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., As robotic technology matures, fully autonomous robots become a realistic possibility, but demand very complex solutions to be rapidly engineered. In order to be able to quickly set up a working autonomous system, and to reduce the gap between simulated and real experiments, we propose a modular, upgradeable and flexible hardware-in-the-loop (HIL) architecture, which hybridizes the simulated and real settings. We take as use case the autonomous exploration of dense forests with UAVs, with the aim of creating useful maps for forest inspection, cataloging, or to compute other metrics such as total wood volume. As the first step in the development of the full system, in this paper we implement a fraction of this architecture, comprising assisted localization, and automatic methods for mapping, planning and motion execution. Specifically we are able to simulate the use of a 3D LIDAR endowed below an actual UAV autonomously navigating among simulated obstacles, thus the platform safety is not compromised. The full system is modular and takes profit of pieces either publicly available or easily programmed. We highlight the flexibility of the proposed HIL architecture to rapidly configure different experimental setups with a UAV in challenging terrain. Moreover, it can be extended to other robotic fields without further design. The HIL system uses the multi-platform ROS capabilities and only needs a motion capture system as external extra hardware, which is becoming standard equipment in all research labs dealing with mobile robots., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

MSClique: Multiple structure discovery through the maximum weighted clique problem

  • Sanromà Güell, Gerard
  • Peñate Sánchez, Adrián
  • Alquézar Mancho, René|||0000-0002-6420-0517
  • Serratosa Casanelles, Francesc|||0000-0001-6112-5913
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
  • González Ballester, Miguel Ángel
We present a novel approach for feature correspondence and multiple structure discovery in computer vision. In contrast to existing methods, we exploit the fact that point-sets on the same structure usually lie close to each other, thus forming clusters in the image. Given a pair of input images, we initially extract points of interest and extract hierarchical representations by agglomerative clustering. We use the maximum weighted clique problem to find the set of corresponding clusters with maximum number of inliers representing the multiple structures at the correct scales. Our method is parameter-free and only needs two sets of points along with their tentative correspondences, thus being extremely easy to use. We demonstrate the effectiveness of our method in multiple-structure fitting experiments in both publicly available and in-house datasets. As shown in the experiments, our approach finds a higher number of structures containing fewer outliers compared to state-of-the-art methods., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Aerial robotics in building inspection and maintenance

  • Grau Saldes, Antoni|||0000-0003-4112-3325
  • Guerra Paradas, Edmundo|||0000-0002-6696-0982
  • Bolea Monte, Yolanda|||0000-0002-7075-9878
  • Puig-Pey Clavería, Ana María|||0000-0002-6073-7448
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
Buildings need periodic revision about their state, materials degrade with time and repairs or renewals have to be made driven by maintenance needs or safety requirements. That happens with any kind of buildings and constructions: housing, architecture masterpieces, old and ancient buildings and industrial buildings. Currently, nearly all of these tasks are carried out by human intervention. In order to carry out the inspection or maintenance, humans need to access to roofs, façades or other areas hard to reach and otherwise potentially hazardous location to perform the task. In some cases, it might not be feasible to access for inspection. For instance, in industry buildings operation must be often interrupted to allow for safe execution of such tasks; these shutdowns not only lead to substantial production loss, but the shutdown and start-up operation itself causes risks to human and environment. In touristic buildings, access has to be restricted with the consequent losses and inconveniences to visitors. The use of aerial robots can help to perform this kind of hazardous operations in an autonomous way, not only teleoperated. Robots are able to carry sensors to detect failures of many types and to locate them in a previously generated map, which the robot uses to navigate. Some of those sensors are cameras in different spectra (visual, near-infrared, UV), laser, LIDAR, ultrasounds and inertial sensory system. If the sensory part is crucial to inspect hazardous areas in buildings, the actuation is also important: the aerial robot can carry small robots (mainly crawler) to be deployed to perform more in-depth operation where the contact between the sensors and the material is basic (any kind of metallic part: pipes, roofs, panels…). The aerial robot has the ability to recover the deployed small crawler to be reused again. In this paper, authors will explain the research that they are conducting in this area and propose future research areas and applications with aerial, ground, submarine and other autonomous robots within the construction field., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Capítulo o parte de un libro (BookPart)

Odometry estimation for aerial manipulation

  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Solà Ortega, Joan|||0000-0002-2933-3381
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
The final publication is available at link.springer.com, This chapter explains a fast and low-cost state localization estimation method for small-sized UAVs, that uses an IMU, a smart camera and an infrared time-of-flight range sensor that act as an odometer providing absolute attitude, velocity, orientation, angular rate and acceleration at a rate higher than 100 Hz. This allows estimating almost continuously the localization of the aerial robot, when GPS or other methods can at most reach 5 Hz. This technique does not require creating a map for localization., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Detection and positioning of pipes and columns with autonomous multicopter drones

  • Guerra Paradas, Edmundo|||0000-0002-6696-0982
  • Munguía Alcalá, Rodrigo Francisco
  • Bolea Monte, Yolanda|||0000-0002-7075-9878
  • Grau Saldes, Antoni|||0000-0003-4112-3325
A multimodal sensory array to accurately position aerial multicopter drones with respect to pipes has been studied, and a solution exploiting both LiDAR and vision sensors has been proposed. Several challenges, including detection of pipes and other cylindrical elements in sensor space and validation of the elements detected, have been studied. A probabilistic parametric method has been applied to segment and position cylinders with LIDAR, while several vision-based techniques have been tested to find the contours of the pipe, combined with conic estimation cylinder pose recovery. Multiple solutions have been studied and analyzed, evaluating their results. This allowed proposing an approach that combines both LiDAR and vision to produce robust and accurate pipe detection. This combined solution is validated with real experimental data., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Multi-modal joint embedding for fashion product retrieval

  • Rubio Romano, Antonio
  • Yu, Longlong
  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Finding a product in the fashion world can be a daunting task. Everyday, e-commerce sites are updating with thousands of images and their associated metadata (textual information), deepening the problem, akin to finding a needle in a haystack. In this paper, we leverage both the images and textual meta-data and propose a joint multi-modal embedding that maps both the text and images into a common latent space. Distances in the latent space correspond to similarity between products, allowing us to effectively perform retrieval in this latent space, which is both efficient and accurate. We train this embedding using large-scale real world e-commerce data by both minimizing the similarity between related products and using auxiliary classification networks to that encourage the embedding to have semantic meaning. We compare against existing approaches and show significant improvements in retrieval tasks on a large-scale e-commerce dataset. We also provide an analysis of the different metadata., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

UAV visual and laser sensors fusion for detection and positioning in industrial applications

  • Guerra Paradas, Edmundo|||0000-0002-6696-0982
  • Munguía Alcalá, Rodrigo Francisco
  • Grau Saldes, Antoni|||0000-0003-4112-3325
This work presents a solution to localize Unmanned Autonomous Vehicles with respect to pipes and other cylindrical elements found in inspection and maintenance tasks both in industrial and civilian infrastructures. The proposed system exploits the different features of vision and laser based sensors, combining them to obtain accurate positioning of the robot with respect to the cylindrical structures. A probabilistic (RANSAC-based) procedure is used to segment possible cylinders found in the laser scans, and this is used as a seed to accurately determine the robot position through a computer vision system. The priors obtained from the laser scan registration help to solve the problem of determining the apparent contour of the cylinders. In turn this apparent contour is used in a degenerate quadratic conic estimation, enabling to visually estimate the pose of the cylinder.
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Delayed monocular SLAM approach applied to unmanned aerial vehicles

  • Munguía Alcalá, Rodrigo Francisco
  • Urzua, Sarquis
  • Grau Saldes, Antoni|||0000-0003-4112-3325
In recent years, many researchers have addressed the issue of making Unmanned Aerial Vehicles (UAVs) more and more autonomous. In this context, the state estimation of the vehicle position is a fundamental necessity for any application involving autonomy. However, the problem of position estimation could not be solved in some scenarios, even when a GPS signal is available, for instance, an application requiring performing precision manoeuvres in a complex environment. Therefore, some additional sensory information should be integrated into the system in order to improve accuracy and robustness. In this work, a novel vision-based simultaneous localization and mapping (SLAM) method with application to unmanned aerial vehicles is proposed. One of the contributions of this work is to design and develop a novel technique for estimating features depth which is based on a stochastic technique of triangulation. In the proposed method the camera is mounted over a servo-controlled gimbal that counteracts the changes in attitude of the quadcopter. Due to the above assumption, the overall problem is simplified and it is focused on the position estimation of the aerial vehicle. Also, the tracking process of visual features is made easier due to the stabilized video. Another contribution of this work is to demonstrate that the integration of very noisy GPS measurements into the system for an initial short period of time is enough to initialize the metric scale. The performance of this proposed method is validated by means of experiments with real data carried out in unstructured outdoor environments. A comparative study shows that, when compared with related methods, the proposed approach performs better in terms of accuracy and computational time., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Capítulo o parte de un libro (BookPart)

Perception for detection and grasping

  • Guerra Paradas, Edmundo|||0000-0002-6696-0982
  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Grau Saldes, Antoni|||0000-0003-4112-3325
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
The final publication is available at link.springer.com, This research presents a methodology for the detection of the crawler used in the project AEROARMS. The approach consisted on using a two-step progressive strategy, going from rough detection and tracking, for approximation maneuvers, to an accurate positioning step based on fiducial markers. Two different methods are explained for the first step, one using efficient image segmentation approach; and the second one using Deep Learning techniques to detect the center of the crawler. The fiducial markers are used for precise localization of the crawler in a similar way as explained in earlier chapters. The methods can run in real-time., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Autonomous navigation of micro aerial vehicles using high-rate and low-cost sensors

  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Loianno, Giuseppe
  • Solà Ortega, Joan|||0000-0002-2933-3381
  • Kumar, Vijay
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
The final publication is available at link.springer.com, The combination of visual and inertial sensors for state estimation has recently found wide echo in the robotics community, especially in the aerial robotics field, due to the lightweight and complementary characteristics of the sensors data. However, most state estimation systems based on visual-inertial sensing suffer from severe processor requirements, which in many cases make them impractical. In this paper, we propose a simple, low-cost and high rate method for state estimation enabling autonomous flight of micro aerial vehicles, which presents a low computational burden. The proposed state estimator fuses observations from an inertial measurement unit, an optical flow smart camera and a time-of-flight range sensor. The smart camera provides optical flow measurements up to a rate of 200 Hz, avoiding the computational bottleneck to the main processor produced by all image processing requirements. To the best of our knowledge, this is the first example of extending the use of these smart cameras from hovering-like motions to odometry estimation, producing estimates that are usable during flight times of several minutes. In order to validate and defend the simplest algorithmic solution, we investigate the performances of two Kalman filters, in the extended and error-state flavors, alongside with a large number of algorithm modifications defended in earlier literature on visual-inertial odometry, showing that their impact on filter performance is minimal. To close the control loop, a non-linear controller operating in the special Euclidean group SE(3) is able to drive, based on the estimated vehicle’s state, a quadrotor platform in 3D space guaranteeing the asymptotic stability of 3D position and heading. All the estimation and control tasks are solved on board and in real time on a limited computational unit. The proposed approach is validated through simulations and experimental results, which include comparisons with ground-truth data provided by a motion capture system. For the benefit of the community, we make the source code public., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Interactive multiple object learning with scanty human supervision

  • Villamizar Vergel, Michael Alejandro
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Moreno-Noguer, Francesc
© 2016. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/, We present a fast and online human-robot interaction approach that progressively learns multiple object classifiers using scanty human supervision. Given an input video stream recorded during the human robot interaction, the user just needs to annotate a small fraction of frames to compute object specific classifiers based on random ferns which share the same features. The resulting methodology is fast (in a few seconds, complex object appearances can be learned), versatile (it can be applied to unconstrained scenarios), scalable (real experiments show we can model up to 30 different object classes), and minimizes the amount of human intervention by leveraging the uncertainty measures associated to each classifier.; We thoroughly validate the approach on synthetic data and on real sequences acquired with a mobile platform in indoor and outdoor scenarios containing a multitude of different objects. We show that with little human assistance, we are able to build object classifiers robust to viewpoint changes, partial occlusions, varying lighting and cluttered backgrounds. (C) 2016 Elsevier Inc. All rights reserved., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Capítulo o parte de un libro (BookPart)

Relative localization for aerial manipulation with PL-SLAM

  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Vakhitov, A.
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
The final publication is available at link.springer.com, This chapter explains a precise SLAM technique, PL-SLAM, that allows to simultaneously process points and lines and tackle situations where point-only based methods are prone to fail, like poorly textured scenes or motion blurred images where feature points are vanished out. The method is remarkably robust against image noise, and that it outperforms state-of-the-art methods for point based contour alignment. The method can run in real-time and in a low cost hardware., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Nonlinear model predictive control for aerial manipulation

  • Lunni, Dario
  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Rossi, Roberto
  • Rocco, Paolo
  • Bascetta, Luca
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This paper presents a nonlinear model predictive controller to follow desired 3D trajectories with the end effector of an unmanned aerial manipulator (i.e., a multirotor with a serial arm attached). To the knowledge of the authors, this is the first time that such controller runs online and on board a limited computational unit to drive a kinematically augmented aerial vehicle. Besides the trajectory following target, we explore the possibility of accomplishing other tasks during flight by taking advantage of the system redundancy. We define several tasks designed for aerial manipulators and show in simulation case studies how they can be achieved by either a weighting strategy, within a main optimization process, or a hierarchical approach consisting on nested optimizations. Moreover, experiments are presented to demonstrate the performance of such controller in a real robot., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Teaching robot's proactive behavior using human assistance

  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Villamizar Vergel, Michael Alejandro
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
The final publication is available at link.springer.com, In recent years, there has been a growing interest in enabling autonomous social robots to interact with people. However, many questions remain unresolved regarding the social capabilities robots should have in order to perform this interaction in an ever more natural manner. In this paper, we tackle this problem through a comprehensive study of various topics involved in the interaction between a mobile robot and untrained human volunteers for a variety of tasks. In particular, this work presents a framework that enables the robot to proactively approach people and establish friendly interaction. To this end, we provided the robot with several perception and action skills, such as that of detecting people, planning an approach and communicating the intention to initiate a conversation while expressing an emotional status. We also introduce an interactive learning system that uses the person’s volunteered assistance to incrementally improve the robot’s perception skills. As a proof of concept, we focus on the particular task of online face learning and recognition. We conducted real-life experiments with our Tibi robot to validate the framework during the interaction process. Within this study, several surveys and user studies have been realized to reveal the social acceptability of the robot within the context of different tasks., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Robotic monocular SLAM in wastewater treatment plants with a sampling device

  • Guerra Paradas, Edmundo|||0000-0002-6696-0982
  • Bolea Monte, Yolanda|||0000-0002-7075-9878
  • Munguía Alcalá, Rodrigo Francisco
  • Grau Saldes, Antoni|||0000-0003-4112-3325
The final publication is available at link.springer.com, A novel application is presented in this paper. Wastewater treatment plants need to sample water to follow the cleaned process continuously. This task is very tedious for laboratory workers and, mainly, can be done in reduced areas of the large basins that form the whole plant, specifically only in the edges of those basins. The new proposal is to enlarge the sampling area and reduce the load for workers. A new unmanned aerial vehicle has been designed together with a novel tool to take water samples. Moreover, a new architecture for the mission plan is presented in form of multi-agents. Experiments have been done in order to test the feasibility of the proposal., This research has been funded by AEROARMS EU Project H2020-ICT-2014-1-644271, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

GANimation: anatomically-aware facial animation from a single image

  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Martinez, Aleix M.
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Moreno-Noguer, Francesc
The final publication is available at link.springer.com, Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for task of facial expression synthesis. The most successful architecture is StarGAN, that conditions GANs' generation process with images of a specific domain, namely a set of images of persons sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combine several of them. Additionally, we propose a fully unsupervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit attention mechanisms that make our network robust to changing backgrounds and lighting conditions. Extensive evaluation show that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild., Peer Reviewed, Award-winning
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Cooperative monocular-based SLAM for multi-UAV systems in GPS-denied environments

  • Trujillo, Juan Carlos
  • Munguía Alcalá, Rodrigo Francisco
  • Guerra Paradas, Edmundo|||0000-0002-6696-0982
  • Grau Saldes, Antoni|||0000-0003-4112-3325
This work presents a cooperative monocular-based SLAM approach for multi-UAV systems that can operate in GPS-denied environments. The main contribution of the work is to show that, using visual information obtained from monocular cameras mounted onboard aerial vehicles flying in formation, the observability properties of the whole system are improved. This fact is especially notorious when compared with other related visual SLAM configurations. In order to improve the observability properties, some measurements of the relative distance between the UAVs are included in the system. These relative distances are also obtained from visual information. The proposed approach is theoretically validated by means of a nonlinear observability analysis. Furthermore, an extensive set of computer simulations is presented in order to validate the proposed approach. The numerical simulation results show that the proposed system is able to provide a good position and orientation estimation of the aerial vehicles flying in formation., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Geometry-aware network for non-rigid shape prediction from a single view

  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Porzi, Lorenzo
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Lepetit, Vincent
  • Moreno-Noguer, Francesc
© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, We propose a method for predicting the 3D shape of a deformable surface from a single view. By contrast with previous approaches, we do not need a pre-registered template of the surface, and our method is robust to the lack of texture and partial occlusions. At the core of our approach is a {it geometry-aware} deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image. We train this architecture in an end-to-end manner using a large dataset of synthetic renderings of shapes under different levels of deformation, material properties, textures and lighting conditions. We evaluate our approach on a test split of this dataset and available real benchmarks, consistently improving state-of-the-art solutions with a significantly lower computational time., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Matchability prediction for full-search template matching algorithms

  • Peñate Sánchez, Adrián
  • Porzi, Lorenzo
  • Moreno-Noguer, Francesc
While recent approaches have shown that it is possible to do template matching by exhaustively scanning the parameter space, the resulting algorithms are still quite demanding. In this paper we alleviate the computational load of these algorithms by proposing an efficient approach for predicting the match ability of a template, before it is actually performed. This avoids large amounts of unnecessary computations. We learn the match ability of templates by using dense convolutional neural network descriptors that do not require ad-hoc criteria to characterize a template. By using deep learning descriptions of patches we are able to predict match ability over the whole image quite reliably. We will also show how no specific training data is required to solve problems like panorama stitching in which you usually require data from the scene in question. Due to the highly parallelizable nature of this tasks we offer an efficient technique with a negligible computational cost at test time., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Aerial social force model: a new framework to accompany people using autonomous flying robots

  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Garza Elizondo, Luís Alberto
  • Villamizar Vergel, Michael Alejandro
  • Herrero Cotarelo, Fernando|||0000-0003-4116-0265
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We proposed a novel Aerial Social Force Model (ASFM) that allows autonomous flying robots to accompany humans in urban environments in a safe and comfortable manner. To date, we are not aware of other state-of-the-art method that accomplish this task. The proposed approach is a 3D version of the Social Force Model (SFM) for the field of aerial robots which includes an interactive human-robot navigation scheme capable of predicting human motions and intentions so as to safely accompany them to their final destination. ASFM also introduces a new metric to fine-tune the parameters of the force model, and to evaluate the performance of the aerial robot companion based on comfort and distance between the robot and humans. The presented approach is extensively validated in diverse simulations and real experiments, and compared against other similar works in the literature. ASFM attains remarkable results and shows that it is a valuable framework for social robotics applications, such as guiding people or human-robot interaction., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Capítulo o parte de un libro (BookPart)

Precise localization for aerial inspection using augmented reality markers

  • Amor Martínez, Adrián
  • Ruiz García, Alberto
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
The final publication is available at link.springer.com, This chapter is devoted to explaining a method for precise localization using augmented reality markers. This method can achieve precision of less of 5 mm in position at a distance of 0.7 m, using a visual mark of 17 mm × 17 mm, and it can be used by controller when the aerial robot is doing a manipulation task. The localization method is based on optimizing the alignment of deformable contours from textureless images working from the raw vertexes of the observed contour. The algorithm optimizes the alignment of the XOR area computed by means of computer graphics clipping techniques. The method can run at 25 frames per second., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Vision-based SLAM system for unmanned aerial vehicles

  • Munguía Alcalá, Rodrigo Francisco
  • Urzua, Sarquis
  • Bolea Monte, Yolanda|||0000-0002-7075-9878
  • Grau Saldes, Antoni|||0000-0003-4112-3325
The present paper describes a vision-based simultaneous localization and mapping system to be applied to Unmanned Aerial Vehicles (UAVs). The main contribution of this work is to propose a novel estimator relying on an Extended Kalman Filter. The estimator is designed in order to fuse the measurements obtained from: (i) an orientation sensor (AHRS); (ii) a position sensor (GPS); and (iii) a monocular camera. The estimated state consists of the full state of the vehicle: position and orientation and their first derivatives, as well as the location of the landmarks observed by the camera. The position sensor will be used only during the initialization period in order to recover the metric scale of the world. Afterwards, the estimated map of landmarks will be used to perform a fully vision-based navigation when the position sensor is not available. Experimental results obtained with simulations and real data show the benefits of the inclusion of camera measurements into the system. In this sense the estimation of the trajectory of the vehicle is considerably improved, compared with the estimates obtained using only the measurements from the position sensor, which are commonly low-rated and highly noisy.
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

PL-SLAM: real-time monocular visual SLAM with points and lines

  • Pumarola Peris, Albert|||0000-0003-4185-6991
  • Vakhitov, A.
  • Agudo Martínez, Antonio|||0000-0001-6845-4998
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Low textured scenes are well known to be one of the main Achilles heels of geometric computer vision algorithms relying on point correspondences, and in particular for visual SLAM. Yet, there are many environments in which, despite being low textured, one can still reliably estimate line-based geometric primitives, for instance in city and indoor scenes, or in the so-called, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Random clustering ferns for multimodal object recognition

  • Villamizar Vergel, Michael Alejandro
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Moreno-Noguer, Francesc
The final publication is available at link.springer.com, We propose an efficient and robust method for the recognition of objects exhibiting multiple intra-class modes, where each one is associated with a particular object appearance. The proposed method, called random clustering ferns, combines synergically a single and real-time classifier, based on the boosted assembling of extremely randomized trees (ferns), with an unsupervised and probabilistic approach in order to recognize efficiently object instances in images and discover simultaneously the most prominent appearance modes of the object through tree-structured visual words. In particular, we use boosted random ferns and probabilistic latent semantic analysis to obtain a discriminative and multimodal classifier that automatically clusters the response of its randomized trees in function of the visual object appearance. The proposed method is validated extensively in synthetic and real experiments, showing that the method is capable of detecting objects with diverse and complex appearance distributions in real-time performance., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Visual-based SLAM configurations for cooperative multi-UAV systems with a lead agent: an observability-based approach

  • Trujillo, Juan Carlos
  • Munguía Alcalá, Rodrigo Francisco
  • Guerra Paradas, Edmundo|||0000-0002-6696-0982
  • Grau Saldes, Antoni|||0000-0003-4112-3325
In this work, the problem of the cooperative visual-based SLAM for the class of multi-UA systems that integrates a lead agent has been addressed. In these kinds of systems, a team of aerial robots flying in formation must follow a dynamic lead agent, which can be another aerial robot, vehicle or even a human. A fundamental problem that must be addressed for these kinds of systems has to do with the estimation of the states of the aerial robots as well as the state of the lead agent. In this work, the use of a cooperative visual-based SLAM approach is studied in order to solve the above problem. In this case, three different system configurations are proposed and investigated by means of an intensive nonlinear observability analysis. In addition, a high-level control scheme is proposed that allows to control the formation of the UAVs with respect to the lead agent. In this work, several theoretical results are obtained, together with an extensive set of computer simulations which are presented in order to numerically validate the proposal and to show that it can perform well under different circumstances (e.g., GPS-challenging environments). That is, the proposed method is able to operate robustly under many conditions providing a good position estimation of the aerial vehicles and the lead agent as well., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Boosted Random ferns for object detection

  • Villamizar Vergel, Michael Alejandro
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper we introduce the Boosted Random Ferns (BRFs) to rapidly build discriminative classifiers for learning and detecting object categories. At the core of our approach we use standard random ferns, but we introduce four main innovations that let us bring ferns from an instance to a category level, and still retain efficiency. First, we define binary features on the histogram of oriented gradients-domain (as opposed to intensity-), allowing for a better representation of intra-class variability. Second, both the positions where ferns are evaluated within the sliding window, and the location of the binary features for each fern are not chosen completely at random, but instead we use a boosting strategy to pick the most discriminative combination of them. This is further enhanced by our third contribution, that is to adapt the boosting strategy to enable sharing of binary features among different ferns, yielding high recognition rates at a low computational cost. And finally, we show that training can be performed online, for sequentially arriving images. Overall, the resulting classifier can be very efficiently trained, densely evaluated for all image locations in about 0.1 seconds, and provides detection rates similar to competing approaches that require expensive and significantly slower processing times. We demonstrate the effectiveness of our approach by thorough experimentation in publicly available datasets in which we compare against state-of-the-art, and for tasks of both 2D detection and 3D multi-view estimation., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Teaching robot’s proactive behavior using human assistance

  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Villamizar Vergel, Michael Alejandro
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
The final publication is available at link.springer.com, In recent years, there has been a growing interest in enabling autonomous social robots to interact with people. However, many questions remain unresolved regarding the social capabilities robots should have in order to perform this interaction in an ever more natural manner. In this paper, we tackle this problem through a comprehensive study of various topics involved in the interaction between a mobile robot and untrained human volunteers for a variety of tasks. In particular, this work presents a framework that enables the robot to proactively approach people and establish friendly interaction. To this end, we provided the robot with several perception and action skills, such as that of detecting people, planning an approach and communicating the intention to initiate a conversation while expressing an emotional status.We also introduce an interactive learning system that uses the person’s volunteered assistance to incrementally improve the robot’s perception skills. As a proof of concept, we focus on the particular task of online face learning and recognition. We conducted real-life experiments with our Tibi robot to validate the framework during the interaction process. Within this study, several surveys and user studies have been realized to reveal the social acceptability of the robot within the context of different tasks., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Searching and tracking people in urban environments with static and dynamic obstacles

  • Goldhoorn, Alex|||0000-0001-9257-2564
  • Garrell Zulueta, Anais|||0000-0002-4629-0723
  • Alquézar Mancho, René|||0000-0002-6420-0517
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© <year>. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/, Searching and tracking people in crowded urban areas where they can be occluded by static or dynamic obstacles is an important behavior for social robots which assist humans in urban outdoor environments. In this work, we propose a method that can handle in real-time searching and tracking people using a Highest Belief Particle Filter Searcher and Tracker. It makes use of a modified Particle Filter (PF), which, in contrast to other methods, can do both searching and tracking of a person under uncertainty, with false negative detections, lack of a person detection, in continuous space and real-time. Moreover, this method uses dynamic obstacles to improve the predicted possible location of the person. Comparisons have been made with our previous method, the Adaptive Highest Belief Continuous Real-time POMCP Follower, in different conditions and with dynamic obstacles. Real-life experiments have been done during two weeks with a mobile service robot in two urban environments of Barcelona with other people walking around., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Trajectory generation for unmanned aerial manipulators through quadratic programming

  • Rossi, Roberto
  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
  • Rocco, Paolo
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper a trajectory generation approach using quadratic programming is described for aerial manipulation, i.e. for the control of an aerial vehicle equipped with a robot arm. The proposed approach applies the online active set strategy to generate a feasible trajectory of the joints, in order to accomplish a set of tasks with defined bounds and constraint inequalities. The definition of the problem in the acceleration domain allows to integrate and perform a large set of tasks and, as a result, to obtain smooth motion of the joints. A weighting strategy, associated with a normalization procedure, allows to easily define the relative importance of the tasks. This approach is useful to accomplish different phases of a mission with different redundancy resolution strategies. The performance of the proposed technique is demonstrated through real experiments with all the algorithms running onboard in real time. In particular, the aerial manipulator can successfully perform navigation and interaction phases, while keeping motion within prescribed bounds and avoiding collisions with external obstacles., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Temporal segmentation of human actions in video sequences

  • Carmona Leyva, José María
  • Climent Vilaró, Joan|||0000-0002-8593-6603
Most of the published works concerning action recognition, usually assume that the action sequences have been previously segmented in time, that is, the action to be recognized starts with the first sequence frame and ends with the last one. However, temporal segmentation of actions in sequences is not an easy task, and is always prone to errors. In this paper, we present a new technique to automatically extract human actions from a video sequence. Our approach presents several contributions. First of all, we use a projection template scheme and find spatio-temporal features and descriptors within the projected surface, rather than extracting them in the whole sequence. For projecting the sequence we use a variant of the R transform, which has never been used before for temporal action segmentation. Instead of projecting the original video sequence, we project its optical flow components, preserving important information about action motion. We test our method on a publicly available action dataset, and the results show that it performs very well segmenting human actions compared with the state-of-the-art methods., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Hybrid visual servoing with hierarchical task composition for aerial manipulation

  • Lippiello, Vincenzo
  • Cacace, Jonathan
  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
  • Trujillo, Miguel Angel
  • Rodrigez Esteves, Yamnia
  • Viguria, Antidio
© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper a hybrid visual servoing with a hierarchical task-composition control framework is described for aerial manipulation, i.e. for the control of an aerial vehicle endowed with a robot arm. The proposed approach suitably combines into a unique hybrid-control framework the main benefits of both image-based and position-based control schemes. Moreover, the underactuation of the aerial vehicle has been explicitly taken into account in a general formulation, together with a dynamic smooth activation mechanism. Both simulation case studies and experiments are presented to demonstrate the performance of the proposed technique., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Learning depth-aware deep representations for robotic perception

  • Porzi, Lorenzo
  • Rota Bulò, Samuel
  • Peñate Sánchez, Adrián
  • Ricci, Elisa
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Exploiting RGB-D data by means of Convolutional Neural Networks (CNNs) is at the core of a number of robotics applications, including object detection, scene semantic segmentation and grasping. Most existing approaches, however, exploit RGB-D data by simply considering depth as an additional input channel for the network. In this paper we show that the performance of deep architectures can be boosted by introducing DaConv, a novel, general-purpose CNN block which exploits depth to learn scale-aware feature representations. We demonstrate the benefits of DaConv on a variety of robotics oriented tasks, involving affordance detection, object coordinate regression and contour detection in RGB-D images. In each of these experiments we show the potential of the proposed block and how it can be readily integrated into existing CNN architectures., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Tesis doctoral (DoctoralThesis)

Visual guidance of unmanned aerial manipulators

  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
Finalista del Georges Giralt PhD Award 2018, a la millor tesi europea sobre robòtica, The ability to fly has greatly expanded the possibilities for robots to perform surveillance, inspection or map generation tasks. Yet it was only in recent years that research in aerial robotics was mature enough to allow active interactions with the environment. The robots responsible for these interactions are called aerial manipulators and usually combine a multirotor platform and one or more robotic arms. The main objective of this thesis is to formalize the concept of aerial manipulator and present guidance methods, using visual information, to provide them with autonomous functionalities. A key competence to control an aerial manipulator is the ability to localize it in the environment. Traditionally, this localization has required external infrastructure of sensors (e.g., GPS or IR cameras), restricting the real applications. Furthermore, localization methods with on-board sensors, exported from other robotics fields such as simultaneous localization and mapping (SLAM), require large computational units becoming a handicap in vehicles where size, load, and power consumption are important restrictions. In this regard, this thesis proposes a method to estimate the state of the vehicle (i.e., position, orientation, velocity and acceleration) by means of on-board, low-cost, light-weight and high-rate sensors. With the physical complexity of these robots, it is required to use advanced control techniques during navigation. Thanks to their redundancy on degrees-of-freedom, they offer the possibility to accomplish not only with mobility requirements but with other tasks simultaneously and hierarchically, prioritizing them depending on their impact to the overall mission success. In this work we present such control laws and define a number of these tasks to drive the vehicle using visual information, guarantee the robot integrity during flight, and improve the platform stability or increase arm operability. The main contributions of this research work are threefold: (1) Present a localization technique to allow autonomous navigation, this method is specifically designed for aerial platforms with size, load and computational burden restrictions. (2) Obtain control commands to drive the vehicle using visual information (visual servo). (3) Integrate the visual servo commands into a hierarchical control law by exploiting the redundancy of the robot to accomplish secondary tasks during flight. These tasks are specific for aerial manipulators and they are also provided. All the techniques presented in this document have been validated throughout extensive experimentation with real robotic platforms., La capacitat de volar ha incrementat molt les possibilitats dels robots per a realitzar tasques de vigilància, inspecció o generació de mapes. Tot i això, no és fins fa pocs anys que la recerca en robòtica aèria ha estat prou madura com per començar a permetre interaccions amb l’entorn d’una manera activa. Els robots per a fer-ho s’anomenen manipuladors aeris i habitualment combinen una plataforma multirotor i un braç robòtic. L’objectiu d’aquesta tesi és formalitzar el concepte de manipulador aeri i presentar mètodes de guiatge, utilitzant informació visual, per dotar d’autonomia aquest tipus de vehicles. Una competència clau per controlar un manipulador aeri és la capacitat de localitzar-se en l’entorn. Tradicionalment aquesta localització ha requerit d’infraestructura sensorial externa (GPS, càmeres IR, etc.), limitant així les aplicacions reals. Pel contrari, sistemes de localització exportats d’altres camps de la robòtica basats en sensors a bord, com per exemple mètodes de localització i mapejat simultànis (SLAM), requereixen de gran capacitat de còmput, característica que penalitza molt en vehicles on la mida, pes i consum elèctric son grans restriccions. En aquest sentit, aquesta tesi proposa un mètode d’estimació d’estat del robot (posició, velocitat, orientació i acceleració) a partir de sensors instal·lats a bord, de baix cost, baix consum computacional i que proporcionen mesures a alta freqüència. Degut a la complexitat física d’aquests robots, és necessari l’ús de tècniques de control avançades. Gràcies a la seva redundància de graus de llibertat, aquests robots ens ofereixen la possibilitat de complir amb els requeriments de mobilitat i, simultàniament, realitzar tasques de manera jeràrquica, ordenant-les segons l’impacte en l’acompliment de la missió. En aquest treball es presenten aquestes lleis de control, juntament amb la descripció de tasques per tal de guiar visualment el vehicle, garantir la integritat del robot durant el vol, millorar de l’estabilitat del vehicle o augmentar la manipulabilitat del braç. Aquesta tesi es centra en tres aspectes fonamentals: (1) Presentar una tècnica de localització per dotar d’autonomia el robot. Aquest mètode està especialment dissenyat per a plataformes amb restriccions de capacitat computacional, mida i pes. (2) Obtenir les comandes de control necessàries per guiar el vehicle a partir d’informació visual. (3) Integrar aquestes accions dins una estructura de control jeràrquica utilitzant la redundància del robot per complir altres tasques durant el vol. Aquestes tasques son específiques per a manipuladors aeris i també es defineixen en aquest document. Totes les tècniques presentades en aquesta tesi han estat avaluades de manera experimental amb plataformes robòtiques reals, Award-winning
UPCommons. Portal del coneixement obert de la UPC
Publicación Capítulo o parte de un libro (BookPart)

Robust perception for aerial inspection: Adaptive and on-line techniques

  • Villamizar Vergel, Michael Alejandro
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
This chapter explains an adaptive on-line object detection and classification technique for robust perception due to varying scene conditions, for example partial cast shadows, change on the illumination conditions or changes in the angle of the object target view. This approach continuously updates the target model upon arrival of new data, being able to adapt to dynamic situations. The method uses an on-line learning technique that works on real-time and it is continuously updated in order to adapt to potential changes undergone by the target object. The method can run in real-time., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Discriminative learning of deep convolutional feature point descriptors

  • Simó Serra, Edgar
  • Trulls Fortuny, Eduard
  • Ferraz, Luis
  • Kokkinos, Iasonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Deep learning has revolutionalized image-level tasks such as classification, but patch-level tasks, such as correspondence, still rely on hand-crafted features, e.g. SIFT. In this paper we use Convolutional Neural Networks (CNNs) to learn discriminant patch representations and in particular train a Siamese network with pairs of (non)corresponding patches. We deal with the large number of potential pairs with the combination of a stochastic sampling of the training set and an aggressive mining strategy biased towards patches that are hard to classify. By using the L2 distance during both training and test- ing we develop 128-D descriptors whose euclidean distances reflect patch similarity, and which can be used as a drop-in replacement for any task involving SIFT. We demonstrate consistent performance gains over the state of the art, and generalize well against scaling and rotation, perspective transformation, non-rigid deformation, and illumination changes. Our descriptors are efficient to compute and amenable to modern GPUs, and are publicly available., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Joint coarse-and-fine reasoning for deep optical flow

  • Vaquero Gómez, Víctor|||0000-0003-2516-2551
  • Ros, German
  • Moreno-Noguer, Francesc
  • López, Antonio Manuel
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We propose a novel representation for dense pixel-wise estimation tasks using CNNs that boosts accuracy and reduces training time, by explicitly exploiting joint coarse-and-fine reasoning. The coarse reasoning is performed over a discrete classification space to obtain a general rough solution, while the fine details of the solution are obtained over a continuous regression space. In our approach both components are jointly estimated, which proved to be beneficial for improving estimation accuracy. Additionally, we propose a new network architecture, which combines coarse and fine components by treating the fine estimation as a refinement built on top of the coarse solution, and therefore adding details to the general prediction. We apply our approach to the challenging problem of optical flow estimation and empirically validate it against state-of-the-art CNN-based solutions trained from scratch and tested on large optical flow datasets., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Quadrotor multi-model for control purposes

  • Grau Saldes, Antoni|||0000-0003-4112-3325
  • Bolea Monte, Yolanda|||0000-0002-7075-9878
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
In this work, a multi-model of a quadrotor is developed in order to control this system. The kinematic model of each part of the quadrotor will be derived using the Euler angles, and also the dynamics model of the quadrotor will be calculated based on the first principles of a rigid body using the Newton-Euler formulation. Furthermore, the following assumptions are used :1) The structure is completely rigid and perfectly symmetric. 2) The center of mass is in the origin of the quadrotor fixed frame. 3) The thrusts are proportional to the square of the motors rotational speed. A state-space model (kinematics and dynamics) is developed by physical laws. But, this deduced model presents several no linearities that are produced by three factors: the orientation (Pitch, Roll and Yaw), the control action and the angular velocities. To be able to control the quadrotor system in simple, linear and manageable way, it is necessary to linearize the system. Two method are possible: a classical linearization around several set-points and a multi-model linearization. In this case, a multi-model linearization is proposed due to the obtained control model will be used to compute a multi-model controller using fuzzy techniques. Fuzzy control techniques are suitable for linear parameter varying systems with no linearities, as our quadrotor., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Planar PØP: feature-less pose estimation with applications in UAV localization

  • Amor Martínez, Adrián
  • Santamaria Navarro, Àngel|||0000-0002-6328-1423
  • Herrero Cotarelo, Fernando|||0000-0003-4116-0265
  • Ruiz, Alberto
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We present a featureless pose estimation method that, in contrast to current Perspective-n-Point (PnP) approaches, it does not require n point correspondences to obtain the camera pose, allowing for pose estimation from natural shapes that do not necessarily have distinguished features like corners or intersecting edges. Instead of using n correspondences (e.g. extracted with a feature detector) we will use the raw polygonal representation of the observed shape and directly estimate the pose in the pose-space of the camera. This method compared with a general PnP method, does not require n point correspondences neither a priori knowledge of the object model (except the scale), which is registered with a picture taken from a known robot pose. Moreover, we achieve higher precision because all the information of the shape contour is used to minimize the area between the projected and the observed shape contours. To emphasize the non-use of n point correspondences between the projected template and observed contour shape, we call the method Planar PØP. The method is shown both in simulation and in a real application consisting on a UAV localization where comparisons with a precise ground-truth are provided., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Learning the hidden human knowledge of UAV pilots when navigating in a cluttered environment for improving path planning

  • Alzugaray López, Ignacio
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, We propose in this work a new model of how the hidden human knowledge (HHK) of UAV pilots can be incorporated in the UAVs path planning generation. We intuitively know that human’s pilots barely manage or even attempt to drive the UAV through a path that is optimal attending to some criteria as an optimal planner would suggest. Although human pilots might get close but not reach the optimal path proposed by some planner that optimizes over time or distance, the final effect of this differentiation could be not only surprisingly better, but also desirable. In the best scenario for optimality, the path that human pilots generate would deviate from the optimal path as much as the hidden knowledge that its perceives is injected into the path. The aim of our work is to use real human pilot paths to learn the hidden knowledge using repulsion fields and to incorporate this knowledge afterwards in the environment obstacles as cause of the deviation from optimality. We present a strategy of learning this knowledge based on attractor and repulsors, the learning method and a modified RRT* that can use this knowledge for path planning. Finally we do real-life tests and we compare the resulting paths with and without this knowledge.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Vision-based SLAM system for unmanned aerial vehicles

  • Munguía Alcalá, Rodrigo Francisco
  • Urzua, Sarquis
  • Bolea Monte, Yolanda
  • Grau Saldes, Antoni
The present paper describes a vision-based simultaneous localization and mapping system to be applied to Unmanned Aerial Vehicles (UAVs). The main contribution of this work is to propose a novel estimator relying on an Extended Kalman Filter. The estimator is designed in order to fuse the measurements obtained from: (i) an orientation sensor (AHRS); (ii) a position sensor (GPS); and (iii) a monocular camera. The estimated state consists of the full state of the vehicle: position and orientation and their first derivatives, as well as the location of the landmarks observed by the camera. The position sensor will be used only during the initialization period in order to recover the metric scale of the world. Afterwards, the estimated map of landmarks will be used to perform a fully vision-based navigation when the position sensor is not available. Experimental results obtained with simulations and real data show the benefits of the inclusion of camera measurements into the system. In this sense the estimation of the trajectory of the vehicle is considerably improved, compared with the estimates obtained using only the measurements from the position sensor, which are commonly low-rated and highly noisy., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Multi-modal joint embedding for fashion product retrieval

  • Rubio Romano, Antonio
  • Yu, Longlong
  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Finding a product in the fashion world can be a daunting task. Everyday, e-commerce sites are updating with thousands of images and their associated metadata (textual information), deepening the problem, akin to finding a needle in a haystack. In this paper, we leverage both the images and textual meta-data and propose a joint multi-modal embedding that maps both the text and images into a common latent space. Distances in the latent space correspond to similarity between products, allowing us to effectively perform retrieval in this latent space, which is both efficient and accurate. We train this embedding using large-scale real world e-commerce data by both minimizing the similarity between related products and using auxiliary classification networks to that encourage the embedding to have semantic meaning. We compare against existing approaches and show significant improvements in retrieval tasks on a large-scale e-commerce dataset. We also provide an analysis of the different metadata., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Aerial social force model: a new framework to accompany people using autonomous flying robots

  • Garrell Zulueta, Anais
  • Garza Elizondo, Luís Alberto
  • Villamizar Vergel, Michael Alejandro
  • Herrero Cotarelo, Fernando
  • Sanfeliu Cortés, Alberto
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We proposed a novel Aerial Social Force Model (ASFM) that allows autonomous flying robots to accompany humans in urban environments in a safe and comfortable manner. To date, we are not aware of other state-of-the-art method that accomplish this task. The proposed approach is a 3D version of the Social Force Model (SFM) for the field of aerial robots which includes an interactive human-robot navigation scheme capable of predicting human motions and intentions so as to safely accompany them to their final destination. ASFM also introduces a new metric to fine-tune the parameters of the force model, and to evaluate the performance of the aerial robot companion based on comfort and distance between the robot and humans. The presented approach is extensively validated in diverse simulations and real experiments, and compared against other similar works in the literature. ASFM attains remarkable results and shows that it is a valuable framework for social robotics applications, such as guiding people or human-robot interaction., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Unsupervised person image synthesis in arbitrary poses

  • Pumarola Peris, Albert
  • Agudo Martínez, Antonio
  • Sanfeliu Cortés, Alberto
  • Moreno-Noguer, Francesc
© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, We present a novel approach for synthesizing photo-realistic images of people in arbitrary poses using generative adversarial learning. Given an input image of a person and a desired pose represented by a 2D skeleton, our model renders the image of the same person under the new pose, synthesizing novel views of the parts visible in the input image and hallucinating those that are not seen. This problem has recently been addressed in a supervised manner, i.e., during training the ground truth images under the new poses are given to the network. We go beyond these approaches by proposing a fully unsupervised strategy. We tackle this challenging scenario by splitting the problem into two principal subtasks. First, we consider a pose conditioned bidirectional generator that maps back the initially rendered image to the original pose, hence being directly comparable to the input image without the need to resort to any training image. Second, we devise a novel loss function that incorporates content and style terms, and aims at producing images of high perceptual quality. Extensive experiments conducted on the DeepFashion dataset demonstrate that the images rendered by our model are very close in appearance to those obtained by fully supervised approaches., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Learning the hidden human knowledge of UAV pilots when navigating in a cluttered environment for improving path planning

  • Alzugaray López, Ignacio
  • Sanfeliu Cortés, Alberto
© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, We propose in this work a new model of how the hidden human knowledge (HHK) of UAV pilots can be incorporated in the UAVs path planning generation. We intuitively know that human’s pilots barely manage or even attempt to drive the UAV through a path that is optimal attending to some criteria as an optimal planner would suggest. Although human pilots might get close but not reach the optimal path proposed by some planner that optimizes over time or distance, the final effect of this differentiation could be not only surprisingly better, but also desirable. In the best scenario for optimality, the path that human pilots generate would deviate from the optimal path as much as the hidden knowledge that its perceives is injected into the path. The aim of our work is to use real human pilot paths to learn the hidden knowledge using repulsion fields and to incorporate this knowledge afterwards in the environment obstacles as cause of the deviation from optimality. We present a strategy of learning this knowledge based on attractor and repulsors, the learning method and a modified RRT* that can use this knowledge for path planning. Finally we do real-life tests and we compare the resulting paths with and without this knowledge., Accepted version
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Searching and tracking people in urban environments with static and dynamic obstacles

  • Goldhoorn, Alex
  • Garrell Zulueta, Anais
  • Alquézar Mancho, René
  • Sanfeliu Cortés, Alberto
© <year>. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/, Searching and tracking people in crowded urban areas where they can be occluded by static or dynamic obstacles is an important behavior for social robots which assist humans in urban outdoor environments. In this work, we propose a method that can handle in real-time searching and tracking people using a Highest Belief Particle Filter Searcher and Tracker. It makes use of a modified Particle Filter (PF), which, in contrast to other methods, can do both searching and tracking of a person under uncertainty, with false negative detections, lack of a person detection, in continuous space and real-time. Moreover, this method uses dynamic obstacles to improve the predicted possible location of the person. Comparisons have been made with our previous method, the Adaptive Highest Belief Continuous Real-time POMCP Follower, in different conditions and with dynamic obstacles. Real-life experiments have been done during two weeks with a mobile service robot in two urban environments of Barcelona with other people walking around., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación

Perception for detection and grasping

  • Guerra Paradas, Edmundo
  • Pumarola Peris, Albert
  • Grau Saldes, Antoni
  • Sanfeliu Cortés, Alberto
The final publication is available at link.springer.com, This research presents a methodology for the detection of the crawler used in the project AEROARMS. The approach consisted on using a two-step progressive strategy, going from rough detection and tracking, for approximation maneuvers, to an accurate positioning step based on fiducial markers. Two different methods are explained for the first step, one using efficient image segmentation approach; and the second one using Deep Learning techniques to detect the center of the crawler. The fiducial markers are used for precise localization of the crawler in a similar way as explained in earlier chapters. The methods can run in real-time., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Boosted Random ferns for object detection

  • Villamizar Vergel, Michael Alejandro
  • Andrade-Cetto, Juan
  • Sanfeliu Cortés, Alberto
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper we introduce the Boosted Random Ferns (BRFs) to rapidly build discriminative classifiers for learning and detecting object categories. At the core of our approach we use standard random ferns, but we introduce four main innovations that let us bring ferns from an instance to a category level, and still retain efficiency. First, we define binary features on the histogram of oriented gradients-domain (as opposed to intensity-), allowing for a better representation of intra-class variability. Second, both the positions where ferns are evaluated within the sliding window, and the location of the binary features for each fern are not chosen completely at random, but instead we use a boosting strategy to pick the most discriminative combination of them. This is further enhanced by our third contribution, that is to adapt the boosting strategy to enable sharing of binary features among different ferns, yielding high recognition rates at a low computational cost. And finally, we show that training can be performed online, for sequentially arriving images. Overall, the resulting classifier can be very efficiently trained, densely evaluated for all image locations in about 0.1 seconds, and provides detection rates similar to competing approaches that require expensive and significantly slower processing times. We demonstrate the effectiveness of our approach by thorough experimentation in publicly available datasets in which we compare against state-of-the-art, and for tasks of both 2D detection and 3D multi-view estimation., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

MSClique: multiple structure discovery through the maximum weighted clique problem

  • Sanromà, Gerard
  • Penate-Sanchez, Adrian
  • Alquézar, René
  • Serratosa, Francesc
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan
  • González Ballester, Miguel Ángel, 1973-
We present a novel approach for feature correspondence and multiple structure discovery in computer vision. In contrast to existing methods, we exploit the fact that point-sets on the same structure usually lie close to each other, thus forming clusters in the image. Given a pair of input images, we initially extract points of interest and extract hierarchical representations by agglomerative clustering. We use the maximum weighted clique problem to find the set of corresponding clusters with maximum number of inliers representing the multiple structures at the correct scales. Our method is parameter-free and only needs two sets of points along with their tentative correspondences, thus being extremely easy to use. We demonstrate the effectiveness of our method in multiple-structure fitting experiments in both publicly available and in-house datasets. As shown in the experiments, our approach finds a higher number of structures containing fewer outliers compared to state-of-the-art methods., This work has been partially supported by the Spanish Ministry of Economy and Competitiveness under Project DPI-2011-27510 and the ERA-Net Chistera project ViSen PCIN-2013-047. This work has also been partially supported by EU H2020, Call H2020-ICT-23-2014-1 (RIA) under Project 644271. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Hybrid visual servoing with hierarchical task composition for aerial manipulation

  • Lippiello, Vincenzo
  • Cacace, Jonathan
  • Santamaria Navarro, Àngel
  • Andrade-Cetto, Juan
  • Trujillo, Miguel Angel
  • Rodrigez Esteves, Yamnia
  • Viguria, Antidio
© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper a hybrid visual servoing with a hierarchical task-composition control framework is described for aerial manipulation, i.e. for the control of an aerial vehicle endowed with a robot arm. The proposed approach suitably combines into a unique hybrid-control framework the main benefits of both image-based and position-based control schemes. Moreover, the underactuation of the aerial vehicle has been explicitly taken into account in a general formulation, together with a dynamic smooth activation mechanism. Both simulation case studies and experiments are presented to demonstrate the performance of the proposed technique., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación

Relative localization for aerial manipulation with PL-SLAM

  • Pumarola Peris, Albert
  • Vakhitov, A.
  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
  • Sanfeliu Cortés, Alberto
The final publication is available at link.springer.com, This chapter explains a precise SLAM technique, PL-SLAM, that allows to simultaneously process points and lines and tackle situations where point-only based methods are prone to fail, like poorly textured scenes or motion blurred images where feature points are vanished out. The method is remarkably robust against image noise, and that it outperforms state-of-the-art methods for point based contour alignment. The method can run in real-time and in a low cost hardware., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

A flexible hardware-in-the-loop architecture for UAVs

  • Lepej, Peter
  • Santamaria Navarro, Àngel
  • Solà, Joan
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., As robotic technology matures, fully autonomous robots become a realistic possibility, but demand very complex solutions to be rapidly engineered. In order to be able to quickly set up a working autonomous system, and to reduce the gap between simulated and real experiments, we propose a modular, upgradeable and flexible hardware-in-the-loop (HIL) architecture, which hybridizes the simulated and real settings. We take as use case the autonomous exploration of dense forests with UAVs, with the aim of creating useful maps for forest inspection, cataloging, or to compute other metrics such as total wood volume. As the first step in the development of the full system, in this paper we implement a fraction of this architecture, comprising assisted localization, and automatic methods for mapping, planning and motion execution. Specifically we are able to simulate the use of a 3D LIDAR endowed below an actual UAV autonomously navigating among simulated obstacles, thus the platform safety is not compromised. The full system is modular and takes profit of pieces either publicly available or easily programmed. We highlight the flexibility of the proposed HIL architecture to rapidly configure different experimental setups with a UAV in challenging terrain. Moreover, it can be extended to other robotic fields without further design. The HIL system uses the multi-platform ROS capabilities and only needs a motion capture system as external extra hardware, which is becoming standard equipment in all research labs dealing with mobile robots., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Nonlinear model predictive control for aerial manipulation

  • Lunni, Dario
  • Santamaria Navarro, Àngel
  • Rossi, Roberto
  • Rocco, Paolo
  • Bascetta, Luca
  • Andrade-Cetto, Juan
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This paper presents a nonlinear model predictive controller to follow desired 3D trajectories with the end effector of an unmanned aerial manipulator (i.e., a multirotor with a serial arm attached). To the knowledge of the authors, this is the first time that such controller runs online and on board a limited computational unit to drive a kinematically augmented aerial vehicle. Besides the trajectory following target, we explore the possibility of accomplishing other tasks during flight by taking advantage of the system redundancy. We define several tasks designed for aerial manipulators and show in simulation case studies how they can be achieved by either a weighting strategy, within a main optimization process, or a hierarchical approach consisting on nested optimizations. Moreover, experiments are presented to demonstrate the performance of such controller in a real robot., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Joint coarse-and-fine reasoning for deep optical flow

  • Vaquero Gómez, Víctor
  • Ros, German
  • Moreno-Noguer, Francesc
  • López, Antonio Manuel
  • Sanfeliu Cortés, Alberto
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We propose a novel representation for dense pixel-wise estimation tasks using CNNs that boosts accuracy and reduces training time, by explicitly exploiting joint coarse-and-fine reasoning. The coarse reasoning is performed over a discrete classification space to obtain a general rough solution, while the fine details of the solution are obtained over a continuous regression space. In our approach both components are jointly estimated, which proved to be beneficial for improving estimation accuracy. Additionally, we propose a new network architecture, which combines coarse and fine components by treating the fine estimation as a refinement built on top of the coarse solution, and therefore adding details to the general prediction. We apply our approach to the challenging problem of optical flow estimation and empirically validate it against state-of-the-art CNN-based solutions trained from scratch and tested on large optical flow datasets., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Interactive multiple object learning with scanty human supervision

  • Villamizar Vergel, Michael Alejandro
  • Garrell Zulueta, Anais
  • Sanfeliu Cortés, Alberto
  • Moreno-Noguer, Francesc
© 2016. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/, We present a fast and online human-robot interaction approach that progressively learns multiple object classifiers using scanty human supervision. Given an input video stream recorded during the human robot interaction, the user just needs to annotate a small fraction of frames to compute object specific classifiers based on random ferns which share the same features. The resulting methodology is fast (in a few seconds, complex object appearances can be learned), versatile (it can be applied to unconstrained scenarios), scalable (real experiments show we can model up to 30 different object classes), and minimizes the amount of human intervention by leveraging the uncertainty measures associated to each classifier.; We thoroughly validate the approach on synthetic data and on real sequences acquired with a mobile platform in indoor and outdoor scenarios containing a multitude of different objects. We show that with little human assistance, we are able to build object classifiers robust to viewpoint changes, partial occlusions, varying lighting and cluttered backgrounds. (C) 2016 Elsevier Inc. All rights reserved., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Temporal segmentation of human actions in video sequences

  • Carmona Leyva, José María
  • Climent Vilaró, Joan
Most of the published works concerning action recognition, usually assume that the action sequences have been previously segmented in time, that is, the action to be recognized starts with the first sequence frame and ends with the last one. However, temporal segmentation of actions in sequences is not an easy task, and is always prone to errors. In this paper, we present a new technique to automatically extract human actions from a video sequence. Our approach presents several contributions. First of all, we use a projection template scheme and find spatio-temporal features and descriptors within the projected surface, rather than extracting them in the whole sequence. For projecting the sequence we use a variant of the R transform, which has never been used before for temporal action segmentation. Instead of projecting the original video sequence, we project its optical flow components, preserving important information about action motion. We test our method on a publicly available action dataset, and the results show that it performs very well segmenting human actions compared with the state-of-the-art methods., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación

Robust perception for aerial inspection: Adaptive and on-line techniques

  • Villamizar Vergel, Michael Alejandro
  • Sanfeliu Cortés, Alberto
This chapter explains an adaptive on-line object detection and classification technique for robust perception due to varying scene conditions, for example partial cast shadows, change on the illumination conditions or changes in the angle of the object target view. This approach continuously updates the target model upon arrival of new data, being able to adapt to dynamic situations. The method uses an on-line learning technique that works on real-time and it is continuously updated in order to adapt to potential changes undergone by the target object. The method can run in real-time., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

BASS: boundary-aware superpixel segmentation

  • Rubio Romano, Antonio
  • Yu, Longlong
  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., We propose a new superpixel algorithm based on exploiting the boundary information of an image, as objects in images can generally be described by their boundaries. Our proposed approach initially estimates the boundaries and uses them to place superpixel seeds in the areas in which they are more dense. Afterwards, we minimize an energy function in order to expand the seeds into full superpixels. In addition to standard terms such as color consistency and compactness, we propose using the geodesic distance which concentrates small superpixels in regions of the image with more information, while letting larger superpixels cover more homogeneous regions. By both improving the initialization using the boundaries and coherency of the superpixels with geodesic distances, we are able to maintain the coherency of the image structure with fewer superpixels than other approaches. We show the resulting algorithm to yield smaller Variation of Information metrics in seven different datasets while maintaining Undersegmentation Error values similar to the state-of-the-art methods., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Matchability prediction for full-search template matching algorithms

  • Peñate Sánchez, Adrián
  • Porzi, Lorenzo
  • Moreno-Noguer, Francesc
While recent approaches have shown that it is possible to do template matching by exhaustively scanning the parameter space, the resulting algorithms are still quite demanding. In this paper we alleviate the computational load of these algorithms by proposing an efficient approach for predicting the match ability of a template, before it is actually performed. This avoids large amounts of unnecessary computations. We learn the match ability of templates by using dense convolutional neural network descriptors that do not require ad-hoc criteria to characterize a template. By using deep learning descriptions of patches we are able to predict match ability over the whole image quite reliably. We will also show how no specific training data is required to solve problems like panorama stitching in which you usually require data from the scene in question. Due to the highly parallelizable nature of this tasks we offer an efficient technique with a negligible computational cost at test time., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Learning depth-aware deep representations for robotic perception

  • Porzi, Lorenzo
  • Rota Bulò, Samuel
  • Peñate Sánchez, Adrián
  • Ricci, Elisa
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Exploiting RGB-D data by means of Convolutional Neural Networks (CNNs) is at the core of a number of robotics applications, including object detection, scene semantic segmentation and grasping. Most existing approaches, however, exploit RGB-D data by simply considering depth as an additional input channel for the network. In this paper we show that the performance of deep architectures can be boosted by introducing DaConv, a novel, general-purpose CNN block which exploits depth to learn scale-aware feature representations. We demonstrate the benefits of DaConv on a variety of robotics oriented tasks, involving affordance detection, object coordinate regression and contour detection in RGB-D images. In each of these experiments we show the potential of the proposed block and how it can be readily integrated into existing CNN architectures., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Trajectory generation for unmanned aerial manipulators through quadratic programming

  • Rossi, Roberto
  • Santamaria Navarro, Àngel
  • Andrade-Cetto, Juan
  • Rocco, Paolo
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this paper a trajectory generation approach using quadratic programming is described for aerial manipulation, i.e. for the control of an aerial vehicle equipped with a robot arm. The proposed approach applies the online active set strategy to generate a feasible trajectory of the joints, in order to accomplish a set of tasks with defined bounds and constraint inequalities. The definition of the problem in the acceleration domain allows to integrate and perform a large set of tasks and, as a result, to obtain smooth motion of the joints. A weighting strategy, associated with a normalization procedure, allows to easily define the relative importance of the tasks. This approach is useful to accomplish different phases of a mission with different redundancy resolution strategies. The performance of the proposed technique is demonstrated through real experiments with all the algorithms running onboard in real time. In particular, the aerial manipulator can successfully perform navigation and interaction phases, while keeping motion within prescribed bounds and avoiding collisions with external obstacles., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

UAV visual and laser sensors fusion for detection and positioning in industrial applications

  • Guerra Paradas, Edmundo
  • Munguía Alcalá, Rodrigo Francisco
  • Grau Saldes, Antoni
This work presents a solution to localize Unmanned Autonomous Vehicles with respect to pipes and other cylindrical elements found in inspection and maintenance tasks both in industrial and civilian infrastructures. The proposed system exploits the different features of vision and laser based sensors, combining them to obtain accurate positioning of the robot with respect to the cylindrical structures. A probabilistic (RANSAC-based) procedure is used to segment possible cylinders found in the laser scans, and this is used as a seed to accurately determine the robot position through a computer vision system. The priors obtained from the laser scan registration help to solve the problem of determining the apparent contour of the cylinders. In turn this apparent contour is used in a degenerate quadratic conic estimation, enabling to visually estimate the pose of the cylinder., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Aerial robotics in building inspection and maintenance

  • Grau Saldes, Antoni
  • Guerra Paradas, Edmundo
  • Bolea Monte, Yolanda
  • Puig-Pey Clavería, Ana María
  • Sanfeliu Cortés, Alberto
Buildings need periodic revision about their state, materials degrade with time and repairs or renewals have to be made driven by maintenance needs or safety requirements. That happens with any kind of buildings and constructions: housing, architecture masterpieces, old and ancient buildings and industrial buildings. Currently, nearly all of these tasks are carried out by human intervention. In order to carry out the inspection or maintenance, humans need to access to roofs, façades or other areas hard to reach and otherwise potentially hazardous location to perform the task. In some cases, it might not be feasible to access for inspection. For instance, in industry buildings operation must be often interrupted to allow for safe execution of such tasks; these shutdowns not only lead to substantial production loss, but the shutdown and start-up operation itself causes risks to human and environment. In touristic buildings, access has to be restricted with the consequent losses and inconveniences to visitors. The use of aerial robots can help to perform this kind of hazardous operations in an autonomous way, not only teleoperated. Robots are able to carry sensors to detect failures of many types and to locate them in a previously generated map, which the robot uses to navigate. Some of those sensors are cameras in different spectra (visual, near-infrared, UV), laser, LIDAR, ultrasounds and inertial sensory system. If the sensory part is crucial to inspect hazardous areas in buildings, the actuation is also important: the aerial robot can carry small robots (mainly crawler) to be deployed to perform more in-depth operation where the contact between the sensors and the material is basic (any kind of metallic part: pipes, roofs, panels…). The aerial robot has the ability to recover the deployed small crawler to be reused again. In this paper, authors will explain the research that they are conducting in this area and propose future research areas and applications with aerial, ground, submarine and other autonomous robots within the construction field., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Robotic monocular SLAM in wastewater treatment plants with a sampling device

  • Guerra Paradas, Edmundo
  • Bolea Monte, Yolanda
  • Munguía Alcalá, Rodrigo Francisco
  • Grau Saldes, Antoni
The final publication is available at link.springer.com, A novel application is presented in this paper. Wastewater treatment plants need to sample water to follow the cleaned process continuously. This task is very tedious for laboratory workers and, mainly, can be done in reduced areas of the large basins that form the whole plant, specifically only in the edges of those basins. The new proposal is to enlarge the sampling area and reduce the load for workers. A new unmanned aerial vehicle has been designed together with a novel tool to take water samples. Moreover, a new architecture for the mission plan is presented in form of multi-agents. Experiments have been done in order to test the feasibility of the proposal., This research has been funded by AEROARMS EU Project H2020-ICT-2014-1-644271, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

PL-SLAM: real-time monocular visual SLAM with points and lines

  • Pumarola Peris, Albert
  • Vakhitov, A.
  • Agudo Martínez, Antonio
  • Sanfeliu Cortés, Alberto
  • Moreno-Noguer, Francesc
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works, Low textured scenes are well known to be one of the main Achilles heels of geometric computer vision algorithms relying on point correspondences, and in particular for visual SLAM. Yet, there are many environments in which, despite being low textured, one can still reliably estimate line-based geometric primitives, for instance in city and indoor scenes, or in the so-called, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Quadrotor multi-model for control purposes

  • Grau Saldes, Antoni
  • Bolea Monte, Yolanda
  • Sanfeliu Cortés, Alberto
In this work, a multi-model of a quadrotor is developed in order to control this system. The kinematic model of each part of the quadrotor will be derived using the Euler angles, and also the dynamics model of the quadrotor will be calculated based on the first principles of a rigid body using the Newton-Euler formulation. Furthermore, the following assumptions are used :1) The structure is completely rigid and perfectly symmetric. 2) The center of mass is in the origin of the quadrotor fixed frame. 3) The thrusts are proportional to the square of the motors rotational speed. A state-space model (kinematics and dynamics) is developed by physical laws. But, this deduced model presents several no linearities that are produced by three factors: the orientation (Pitch, Roll and Yaw), the control action and the angular velocities. To be able to control the quadrotor system in simple, linear and manageable way, it is necessary to linearize the system. Two method are possible: a classical linearization around several set-points and a multi-model linearization. In this case, a multi-model linearization is proposed due to the obtained control model will be used to compute a multi-model controller using fuzzy techniques. Fuzzy control techniques are suitable for linear parameter varying systems with no linearities, as our quadrotor., Peer Reviewed, Postprint (published version)