Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 117
Encontrada(s) 2 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous pose, correspondence and non-rigid shape

  • Sánchez-Riera, Jordi
  • Ostlund, Jonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Trabajo presentado al CVPR 2010 celebrada en San Francisco (EE.UU.) del 13 al 18 de junio., Recent works have shown that 3D shape of non-rigid surfaces can be accurately retrieved from a single image given a set of 3D-to-2D correspondences between that image and another one for which the shape is known. However, existing approaches assume that such correspondences can be readily established, which is not necessarily true when large deformations produce significant appearance changes between the input and the reference images. Furthermore, it is either assumed that the pose of the camera is known, or the estimated solution is pose-ambiguous. In this paper we relax all these assumptions and, given a set of 3D and 2D unmatched points, we present an approach to simultaneously solve their correspondences, compute the camera pose and retrieve the shape of the surface in the input image. This is achieved by introducing weak priors on the pose and shape that we model as Gaussian Mixtures. By combining them into a Kalman filter we can progressively reduce the number of 2D candidates that can be potentially matched to each 3D point, while pose and shape are refined. This lets us to perform a complete and efficient exploration of the solution space and retain the best solution., This work was supported by projects: 'CONSOLIDER-INGENIO 2010 Multimodal interaction in pattern recognition and computer vision' (V-00069), 'Percepción y acción ante incertidumbre' (4803), 'Priors para la detección 3D en tiempo real de objetos rígidos y deformables' (4805), 'Gardening with a cognitive system' (4920). This work has been partially funded by the Spanish Ministry of Science and Innovation under projects 200850I055, DPI2008-06022, and Consolider Ingenio 2010 CSD2007-00018, by EU GARNICS project FP7-247947, and by the Swiss National Science Foundation., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Shared random Ferns for efficient detection of multiple categories

  • Villamizar, Michael
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan
  • Sanfeliu, Alberto
Trabajo presentado al 20th ICPR celebrado en Estambul (Turquía) del 23 al 26 de agosto de 2010., We propose a new algorithm for detecting multiple object categories that exploits the fact that different categories may share common features but with different geometric distributions. This yields an efficient detector which, in contrast to existing approaches, considerably reduces the computation cost at runtime, where the feature computation step is traditionally the most expensive. More specifically, at the learning stage we compute common features by applying the same Random Ferns over the Histograms of Oriented Gradients on the training images. We then apply a boosting step to build discriminative weak classifiers, and learn the specific geometric distribution of the Random Ferns for each class. At runtime, only a few Random Ferns have to be densely computed over each input image, and their geometric distribution allows performing the detection. The proposed method has been validated in public datasets achieving competitive detection results, which are comparable with state-of-the-art methods that use specific features per class., This work was supported by projects: 'CONSOLIDER-INGENIO 2010 Multimodal interaction in pattern recognition and computer vision' (V-00069), 'Robotica ubicua para entornos urbanos' (J-01225), 'Percepción y acción ante incertidumbre' (4803), 'Gardening with a cognitive system' (4920). This work has been partially funded by the Spanish Ministry of Science and Innovation under projects UbROB DPI2007-61452, PAU DPI2008-06022, and MIPRCV Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947. The first au- thor is funded by the Technical University of Catalonia (UPC)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Learning force-based robot skills from haptic demonstration

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Trabajo presentado a la International Conference of the Catalan Association for Artificial Intelligence (CCIA) celebrada en Espluga de Francolí (Tarragona) del 20 al 22 de octubre de 2010., Locally weighted as well as Gaussian mixtures learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration. Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. For the first time, force/torque feedback through a haptic device has been used for teaching a teleoperated robot to empty a rigid container. The memory-based LWPLS and the non-memory-based LWPR algorithms, as well as both the batch and the incremental versions of GMM/GMR were implemented, their comparison leading to very similar results, with the same pattern as regards to both the involved robot joints and the different initial experimental conditions. Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages, where the taught motion will be refined by autonomous robot rehearsal through reinforcement learning., This work has been partially supported by the European projects “PACO-PLUS” (IST-4-27657) and “GARNICS: Gardening with a Cognitive System” ( FP7-247947), the Spanish project “Multimodal Interaction in Pattern Recognition and Computer Vision (MIPRCV)” (Consolider Ingenio 2010 project CSD2007-00018) and the consolidated Robotics group of the Generalitat de Catalunya. L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Combining Geometric and Appearance Priors for Robust Homography Estimation

  • Serradell, Eduard
  • Özuysal, Mustafa
  • Lepetit, Vincent
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Trabajo presentado al 11th European Conference on Computer Vision celebrado en Creta del 5 al 11 de septiembre de 2010., The homography between pairs of images are typically computed from the correspondence of keypoints, which are established by using image descriptors. When these descriptors are not reliable, either because of repetitive patterns or large amounts of clutter, additional priors need to be considered. The Blind PnP algorithm makes use of geometric priors to guide the search for matches while computing camera pose. Inspired by this, we propose a novel approach for homography estimation that combines geometric priors with appearance priors of ambiguous descriptors. More specifically, for each point we retain its best candidates according to appearance. We then prune the set of potential matches by iteratively shrinking the regions of the image that are consistent with the geometric prior. We can then successfully compute homographies between pairs of images containing highly repetitive patterns and even under oblique viewing conditions., This work has been partially funded by the Spanish Ministry of Science and Innovation under CICYT projects PAU DPI2008-06022 and UbROB DPI2007-61452; by MIPRCV Consolider Ingenio 2010 CSD2007-00018; by the EU project GARNICS FP7-247947 and by the Swiss National Science Foundation., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Sharpening haptic inputs for teaching a manipulation skill to a robot

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
8 páginas.-- Comunicación presentada al 1st International Conference on Applied Bionics and Biomechanics celebrado en Venecia (Italia) en Octubre de 2010., Gaussian mixtures-based learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration (PbD). Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. In this work we have used force/torque feedback through a haptic device for teaching a teleoperated robot to empty a rigid container. Structure vibrations and container inertia appeared to considerably disrupt the sensing process, so a filtering algorithm had to be devised. Moreover, some input variables seemed much more relevant to the particular task to be learned than others, which lead us to analyze the training data in order to select those relevant features through principal component analysis and a mutual information criterion. Then, a batch version of GMM/GMR [1], [2] was implemented using different training datasets (original, pre-processed data through PCA and MI). Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages., This work has been partially supported by the European projects PACO-PLUS (IST-4-27657) and GARNICS (FP7-247947), the Spanish project Multimodal Interaction in Pattern Recognition and Computer Vision (MIPRCV) (Consolider Ingenio 2010 project CSD2007-00018) and the Robotics group of the Generalitat de Catalunya. L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration in the force domain

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
6 páginas, 6 figuras.-- Presentado al Workshop on Agents Learning Interactively from Human Teachers(ALIHT) celebrado en Barcelona del 16 al 22 de julio de 2011., Researchers are becoming aware of the importance of other information sources besides visual data in robot learning by demonstration (LbD). Force-based perceptions are shown to convey very relevant information – missed by visual and position sensors – for learning specific tasks. In this paper, we review some recent works using forces as input data in LbD and Human-Robot interaction (HRI) scenarios, and propose a complete learning framework for teaching force-based manipulation skills to a robot through a haptic device. We suggest to use haptic interfaces not only as a demonstration tool but also as a communication channel between the human and the robot, getting the teacher more involved in the teaching process by experiencing the force signals sensed by the robot. Within the proposed framework, we provide solutions for treating force signals, extracting relevant information about the task, encoding the training data and generalizing to perform successfully under unknown conditions., This research is partially sponsored by the European projects GARNICS (FP7-247947) and IntellAct (FP7-269959), the Spanish projects MIPRCV (Consolider Ingenio 2010 project CSD2007-00018) and PAU (DPI2008-06022). L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration of force-based tasks with multiple solution trajectories

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Presentado al ICAR 2011 celebrado en Tallin (Estonia) del 20 al 23 de junio de 2011., A learning framework with a bidirectional communication channel is proposed, where a human performs several demonstrations of a task using a haptic device (providing him/her with force-torque feedback) while a robot captures these executions using only its force-based perceptive system. Our work departs from the usual approaches to learning by demonstration in that the robot has to execute the task blindly, relying only on force-torque perceptions, and, more essential, we address goal-driven manipulation tasks with multiple solution trajectories, whereas most works tackle tasks that can be learned by just finding a generalization at the trajectory level. To cope with these multiple-solution tasks, in our framework demonstrations are represented by means of a Hidden Markov Model (HMM) and the robot reproduction of the task is performed using a modified version of Gaussian Mixture Regression that incorporates temporal information (GMRa) through the forward variable of the HMM. Also, we exploit the haptic device as a teaching and communication tool in a human-robot interaction context, as an alternative to kinesthetic-based teaching systems. Results show that the robot is able to learn a container-emptying task relying only on force-based perceptions and to achieve the goal from several non-trained initial conditions., This research is partially sponsored by the European projects GARNICS (FP7-247947) and IntellAct (FP7-269959), the Spanish projects Multimodal Interaction in Pattern Recognition and Computer Vision (MIPRCV) (Consolider Ingenio 2010 project CSD2007-00018) and PAU (DPI2008-06022). L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Depth-supported real-time video segmentation with the Kinect

  • Alexey, Abramov
  • Pauwels, Karl
  • Papon, Jeremie
  • Wörgötter, Florentin
  • Dellen, Babette
Trabajo presentado al WACV celebrado en Breckenridge (USA) del 9 al 11 de enero de 2012., We present a real-time technique for the spatiotemporal segmentation of color/depth movies. Images are segmented using a parallel Metropolis algorithm implemented on a GPU utilizing both color and depth information, acquired with the Microsoft Kinect. Segments represent the equilibrium states of a Potts model, where tracking of segments is achieved by warping obtained segment labels to the next frame using real-time optical flow, which reduces the number of iterations required for the Metropolis method to encounter the new equilibrium state. By including depth information into the framework, true objects boundaries can be found more easily, improving also the temporal coherency of the method. The algorithm has been tested for videos of medium resolutions showing human manipulations of objects. The framework provides an inexpensive visual front end for visual preprocessing of videos in industrial settings and robot labs which can potentially be used in various applications., This research has received funding by the EU GARNICS project FP7-247947 and the EU IntellAct project FP7-269959. B.Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship. K. Pauwels acknowledges support from CEI BioTIC GENIL (CEB09-0010) of the MICINN CEI program., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Depth-supported real-time video segmentation with the Kinect

  • Alexey, Abramov
  • Pauwels, Karl
  • Papon, Jeremie
  • Wörgötter, Florentin
  • Dellen, Babette
This research has received funding by the EU GARNICS project FP7-247947 and the EU IntellAct project FP7-269959. B.Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship. K. Pauwels acknowledges support from CEI BioTIC GENIL (CEB09-0010) of the MICINN CEI program., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Outdoor view recognition based on landmark grouping and logistic regression

  • Todt, Eduardo
  • Torras, Carme
Vision-based robot localization outdoors has remained more elusive than its indoors counterpart. Drastic illumination changes and the scarceness of suitable landmarks are the main difficulties. This paper attempts to surmount them by deviating from the main trend of using local features. Instead, a global descriptor called landmark-view is defined, which aggregates the most visually-salient landmarks present in each scene. Thus, landmark co-occurrence and spatial and saliency relationships between them are added to the single landmark characterization, based on saliency and color distribution. A suitable framework to compare landmark-views is developed, and it is shown how this remarkably enhances the recognition performance, compared against single landmark recognition. A view-matching model is constructed using logistic regression. Experimentation using 45 views, acquired outdoors, containing 273 landmarks, yielded good recognition results. The overall percentage of correct view classification obtained was 80.6%, indicating the adequacy of the approach. © 2013 World Scientific Publishing Company., This work was partially funded by the GARNICS (Gardening with a Cognitive System) project FP7-ICT-247947, Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Robotized plant probing: Leaf segmentation utilizing time-of-flight data

  • Alenyà, Guillem
  • Dellen, Babette
  • Foix, Sergi
  • Torras, Carme
Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modeling for this application has strong demands, particularly in what concerns three-dimensional (3-D) information gathering and speed. © 2013 IEEE., This research is partially funded by the EU GARNICS project FP7-247947, the project PAU+ (DPI2011-27510), and the Grup consolidat SGR155. B.Dellen acknowledges support from the Spanish Ministry of Science and Innovation through a Ramon y Cajal program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

General robot kinematics decomposition without intermediate markers

  • Ulbrich, Stefan
  • Ruiz de Angulo, Vicente
  • Asfour, Tamim
  • Torras, Carme
  • Dillman, Rudiger
The calibration of serial manipulators with high numbers of degrees of freedom by means of machine learning is a complex and time-consuming task. With the help of a simple strategy, this complexity can be drastically reduced and the speed of the learning procedure can be increased: When the robot is virtually divided into shorter kinematic chains, these subchains can be learned separately and, hence, much more efficiently than the complete kinematics. Such decompositions, however, require either the possibility to capture the poses of all end- effectors of all subchains at the same time, or they are limited to robots that fulfill special constraints. In this work, an alternative decomposition is presented that does not suffer from these limitations. An offline training algorithm is provided in which the composite subchains are learned sequentially with dedicated movements. A second training scheme is provided to train composite chains simultaneously and online. Both schemes can be used together with many machine learning algorithms. In the simulations, an algorithm using Parameterized Self-Organizing Maps (PSOM) modified for online learning and Gaussian Mixture Models (GMM) were chosen to show the correctness of the approach. The experimental results show that, using a two-fold decomposition, the number of samples required to reach a given precision is reduced to twice the square root of the original number., The work described in this paper was partially conducted within the EU Cognitive Systems projects Xperience (FP-7-270273) and GARNICS (FP-7-247947) funded by the European Commission, and the Generalitat de Catalunya through the Robotics group (SGR2009-00155). V. Ruiz de Angulo acknowledges support from Spanish Ministry of Science and Education, under the project DPI2010-18449. C. Torras acknowledges support from the Consolider project MIPRCV (CSD2007-00018)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robust elastic 2D/3D geometric graph matching

  • Serradell, Eduard
  • Kybic, Jan
  • Moreno-Noguer, Francesc
  • Fua, Pascal
Trabajo presentado a la Medical Imaging celebrada en San Diego el 4 de febrero de 2012., We present an algorithm for geometric matching of graphs embedded in 2D or 3D space. It is applicable for registering any graph-like structures appearing in biomedical images, such as blood vessels, pulmonary bronchi, nerve fibers, or dendritic arbors. Our approach does not rely on the similarity of local appearance features, so it is suitable for multimodal registration with a large difference in appearance. Unlike earlier methods, the algorithm uses edge shape, does not require an initial pose estimate, can handle partial matches, and can cope with nonlinear deformations and topological differences. The matching consists of two steps. First, we find an affine transform that roughly aligns the graphs by exploring the set of all consistent correspondences between the nodes. This can be done at an acceptably low computational expense by using parameter uncertainties for pruning, backtracking as needed. Parameter uncertainties are updated in a Kalman-like scheme with each match. In the second step we allow for a nonlinear part of the deformation, modeled as a Gaussian Process. Short sequences of edges are grouped into superedges, which are then matched between graphs. This allows for topological differences. A maximum consistent set of superedge matches is found using a dedicated branch-and-bound solver, which is over 100 times faster than a standard linear programming approach. Geometrical and topological consistency of candidate matches is determined in a fast hierarchical manner. We demonstrate the effectiveness of our technique at registering angiography and retinal fundus images, as well as neural image stacks., Jan Kybic was supported by the Czech Science Foundation project P202/11/0111. This project was also funded in part by the ERC project MicroNano, the Spanish Ministry of Science and Innovation CICYT project PAU+ DPI2011-27510, the MIPRCV Consolider Ingenio 2010 project CSD2007-00018 and by the EU project GARNICS FP7-247947., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robust non-rigid registration of 2D and 3D graphs

  • Serradell, Eduard
  • Glowacki, Przemyslaw
  • Kybic, Jan
  • Moreno-Noguer, Francesc
  • Fua, Pascal
Trabajo presentado al CVPR celebrado en Providence (USA) del 16 al 21 de junio de 2012., We present a new approach to matching graphs embedded in R2 or R3. Unlike earlier methods, our approach does not rely on the similarity of local appearance features, does not require an initial alignment, can handle partial matches, and can cope with non-linear deformations and topological differences. To handle arbitrary non-linear deformations, we represent them as Gaussian Processes. In the absence of appearance information, we iteratively establish correspondences between graph nodes, update the structure accordingly, and use the current mapping estimate to find the most likely correspondences that will be used in the next iteration. This makes the computation tractable. We demonstrate the effectiveness of our approach first on synthetic cases and then on angiography data, retinal fundus images, and microscopy image stacks acquired at very different resolutions., This work has been partially funded by the ERC projects MicroNano and GARNICS FP7-247947, the Spanish Ministry of Economy and Competitiveness under projects PAU+DPI2011-27510 and MIPRCV Consolider Ingenio 2010 CSD2007-00018, and the Czech Science Foundation under project P202/11/0111. We thank the project ‘La Marató de TV3 082131 for providing the heart dataset., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Channel coding for joint colour and depth segmentation

  • Wallenberg, Marcus
  • Felsberg, Michael
  • Forssén, Per-Erik
  • Dellen, Babette
Trabajo presentado al 33rd DAGM Symposium celebrado en Frankfurt del 31 de agosto al 2 de septiembre de 2011., Segmentation is an important preprocessing step in many applications. Compared to colour segmentation, fusion of colour and depth greatly improves the segmentation result. Such a fusion is easy to do by stacking measurements in different value dimensions, but there are better ways. In this paper we perform fusion using the channel representation, and demonstrate how a state-of-the-art segmentation algorithm can be modified to use channel values as inputs. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using the superparamagnetic clustering algorithm. Our experiments show that depth gradients are more useful than depth values for segmentation, and that channel coding both colour and depth gradients makes tuned parameter settings generalise better to novel images., This work was supported by the EC's 7th Framework Programme (FP7/2007-2013), grant agreement 247947 (GARNICS), the Swedish Research Council through a grant for the project Embodied Visual Object Recognition, and by Link oping University. B. Dellen also acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Deformation and illumination invariant feature point descriptor

  • Moreno-Noguer, Francesc
Trabajo presentado al CVPR celebrado en Providence (USA) del 20 al 25 de junio de 2011., Recent advances in 3D shape recognition have shown that kernels based on diffusion geometry can be effectively used to describe local features of deforming surfaces. In this paper, we introduce a new framework that allows using these kernels on 2D local patches, yielding a novel feature point descriptor that is both invariant to non-rigid image deformations and illumination changes. In order to build the descriptor, 2D image patches are embedded as 3D surfaces, by multiplying the intensity level by an arbitrarily large and constant weight that favors anisotropic diffusion and retains the gradient magnitude information. Patches are then described in terms of a heat kernel signature, which is made invariant to intensity changes, rotation and scaling. The resulting feature point descriptor is proven to be significantly more discriminative than state of the art ones, even those which are specifically designed for describing non-rigid image deformations., This work has been partially funded by the Spanish Ministry of Science and Innovation under projects DPI2008-06022 and Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Design of non-anthropomorphic robotic hands for anthropomorphic tasks

  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
  • Perez-Gracia, Alba
Trabajo presentado al 35th Mechanisms and Robotics Conference celebrado en Washington del 28 al 31 de agosto de 2011., In this paper, we explore the idea of designing non-anthropomorphic, multi-fingered robotic hands for tasks that replicate the motion of the human hand. Taking as input data rigid-body trajectories for the five fingertips, we develop a method to perform dimensional synthesis for a kinematic chain with a tree structure, with three common joints and five branches. We state the forward kinematics equations of relative displacements for each serial chain expressed as dual quaternions, and solve for up to five chains simultaneously to reach a number of poses along the hand trajectory using a hybrid global numerical solver that integrates a genetic algorithm and a Levenberg-Marquardt local optimizer. Although the number of candidate solutions in this problem is very high, the use of the genetic algorithm lets us to perform an exhaustive exploration of the solution space and retain a subset of them. We then can choose some of the solutions based on the specific task to perform. Note that these designs could match the task exactly while having a finger design radically different from that of the human hand., This work is partially supported by the Spanish Ministry of Science and Innovation under projects DPI2010-18449, Consolider Ingenio 2010 CSD2007-00018, and a Ramon y Cajal Research Fellowship. Also by the EU project GARNICS FP7-247947., Peer reviewed, Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Integrating task planning and interactive learning for robots to work in human environments

  • Agostini, Alejandro
  • Torras, Carme
  • Wörgötter, Florentin
Trabajo presentado al IJCAI celebrado en Barcelona del 16 al 22 de julio de 2011., Human environments are challenging for robots, which need to be trainable by lay people and learn new behaviours rapidly without disrupting much the ongoing activity. A system that integrates AI techniques for planning and learning is here proposed to satisfy these strong demands. The approach rapidly learns planning operators from few action experiences using a competitive strategy where many alternatives of cause-effect explanations are evaluated in parallel, and the most successful ones are used to generate the operators. The success of a cause-effect explanation is evaluated by a probabilistic estimate that compensates the lack of experience, producing more confident estimations and speeding up the learning in relation to other known estimates. The system operates without task interruption by integrating in the planning-learning loop a human teacher that supports the planner in making decisions. All the mechanisms are integrated and synchronized in the robot using a general decision-making framework. The feasibility and scalability of the architecture are evaluated in two different robot platforms: a Stäubli arm, and the humanoid ARMAR III., This research is partially funded by the EU GARNICS project FP7-247947 and the Generalitat de Catalunya through the Robotics group., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Learning the semantics of object-action relations by observation

  • Aksoy, Eren Erdal
  • Abramov, Alexey
  • Dörr, Johannes
  • Ning, Kejun
  • Dellen, Babette
  • Wörgötter, Florentin
Recognizing manipulations performed by a human and the transfer and execution of this by a robot is a difficult problem. We address this in the current study by introducing a novel representation of the relations between objects at decisive time points during a manipulation. Thereby, we encode the essential changes in a visual scenery in a condensed way such that a robot can recognize and learn a manipulation without prior object knowledge. To achieve this we continuously track image segments in the video and construct a dynamic graph sequence. Topological transitions of those graphs occur whenever a spatial relation between some segments has changed in a discontinuous way and these moments are stored in a transition matrix called the semantic event chain (SEC). We demonstrate that these time points are highly descriptive for distinguishing between different manipulations. Employing simple sub-string search algorithms, SECs can be compared and type-similar manipulations can be recognized with high confidence. As the approach is generic, statistical learning can be used to find the archetypal SEC of a given manipulation class. The performance of the algorithm is demonstrated on a set of real videos showing hands manipulating various objects and performing different actions. In experiments with a robotic arm, we show that the SEC can be learned by observing human manipulations, transferred to a new scenario, and then reproduced by the machine. © SAGE Publications 2011., The research leading to these results has received funding from the European Community's Seventh Framework Programme FP7=2007-2013 - Challenge 2 - Cognitive Systems, Interaction, Robotics - under grant agreement No 247947 - GARNICS. B.D. acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal Fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Probabilistic simultaneous pose and non-rigid shape recovery

  • Moreno-Noguer, Francesc
  • Porta, Josep M.
Trabajo presentado al CVPR 2011 celebrado en Providence (USA) del 20 al 25 de junio., We present an algorithm to simultaneously recover non- rigid shape and camera poses from point correspondences between a reference shape and a sequence of input images. The key novel contribution of our approach is in bringing the tools of the probabilistic SLAM methodology from a rigid to a deformable domain. Under the assumption that the shape may be represented as a weighted sum of deformation modes, we show that the problem of estimating the modal weights along with the camera poses, may be probabilistically formulated as a maximum a posterior estimate and solved using an iterative least squares optimization. An extensive evaluation on synthetic and real data, shows that our approach has several significant advantages over current approaches, such as performing robustly under large amounts of noise and outliers, and neither requiring to track points over the whole sequence nor initializations close from the ground truth solution., This work has been partially funded by the Spanish Ministry of Science and Innovation under projects DPI2008-06022 and Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Segmenting color images into surface patches by exploiting sparse depth data

  • Dellen, Babette
  • Alenyà, Guillem
  • Foix, Sergi
  • Torras, Carme
Trabajo presentado al WACV 2011 celebrado en Kona (USA) del 5 al 7 de enero., We present a new method for segmenting color images into their composite surfaces by combining color segmentation with model-based fitting utilizing sparse depth data, acquired using time-of-flight (Swissranger, PMD CamCube) and stereo techniques. The main target of our work is the segmentation of plant structures, i.e., leaves, from color-depth images, and the extraction of color and 3D shape information for automating manipulation tasks. Since segmentation is performed in the dense color space, even sparse, incomplete, or noisy depth information can be used. This kind of data often represents a major challenge for methods operating in the 3D data space directly. To achieve our goal, we construct a three-stage segmentation hierarchy by segmenting the color image with different resolutions - assuming that ``true'' surface boundaries must appear at some point along the segmentation hierarchy. 3D surfaces are then fitted to the color-segment areas using depth data. Those segments which minimize the fitting error are selected and used to construct a new segmentation. Then, an additional region merging and a growing stage are applied to avoid over-segmentation and label previously unclustered points. Experimental results demonstrate that the method is successful in segmenting a variety of domestic objects and plants into quadratic surfaces. At the end of the procedure, the sparse depth data is completed using the extracted surface models, resulting in dense depth maps. For stereo, the resulting disparity maps are compared with ground truth and the average error is computed., This research is partially funded by the EU GARNICS project FP7-247947, the Consolider-Ingenio project CSD2007-00018, and the Catalan Research Commission under 2009SGR155. G. Alenyà and S. Foix were supported by CSIC under a Jae-Doc and Jae-Pre-Doc fellowship, respectively. B. Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous correspondence and non-rigid 3D reconstruction of the coronary tree from single X-ray images

  • Serradell, Eduard
  • Romero, Adriana
  • Leta, Rubén
  • Gatta, Carlo
  • Moreno-Noguer, Francesc
Trabajo presentado al ICCV 2011 celebrado en Barcelona del 6 al 3 de noviembre., We present a novel approach to simultaneously reconstruct the 3D structure of a non-rigid coronary tree and estimate point correspondences between an input X-ray image and a reference 3D shape. At the core of our approach lies an optimization scheme that iteratively fits a generative 3D model of increasing complexity and guides the matching process. As a result, and in contrast to existing approaches that assume rigidity or quasi-rigidity of the structure, our method is able to retrieve large non-linear deformations even when the input data is corrupted by the presence of noise and partial occlusions. We extensively evaluate our approach under synthetic and real data and demonstrate a remarkable improvement compared to state-of-the-art., This work has been partially funded by the project La Marató de TV3 082131; by the Spanish Ministry of Science and Innovation under projects DPI2008-06022, DPI2010- 17112, Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947. The work of C. Gatta is supported by a Beatriu de Pinos Fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Towards plant monitoring through next best view

  • Foix, Sergi
  • Alenyà, Guillem
  • Torras, Carme
Trabajo presentado a la 14th International Conference of the Catalan Association for Artificial Intelligence (CCIA) celebrada en Lleida del 26 al 28 de octubre de 2011., Monitoring plants using leaf feature detection is a challenging perception task because different leaves, even from the same plant, may have very different shapes, sizes and deformations. In addition, leaves may be occluded by other leaves making it hard to determine some of their characteristics. In this paper we use a Time-of-Flight (ToF) camera mounted on a robot arm to acquire the depth information needed for plant leaf detection. Under a Next Best View (NBV) paradigm, we propose a criterion to compute a new camera position that offers a better view of a target leaf. The proposed criterion exploits some typical errors of the ToF camera, which are common to other 3D sensing devices as well. This approach is also useful when more than one leaf is segmented as the same region, since moving the camera following the same NBV criterion helps to disambiguate this situation., This research is partially funded by the EU GARNICS project FP7-247947, by the Span- ish Ministry of Science and Innovation under projects DPI2008-06022 and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission through the Robotics Group. S. Foix is supported by a PhD fellowship from CSIC’s JAE program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

3D modelling of leaves from color and ToF data for robotized plant measuring

  • Alenyà, Guillem
  • Dellen, Babette
  • Torras, Carme
Trabajo presentado al ICRA 2011 celebrado en Shanghai del 9 al 13 de mayo de 2011., Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modelling for this application has strong demands, particularly in what concerns 3D information gathering and speed. This paper shows that Time-of-Flight (ToF) cameras achieve a good compromise between both demands, providing a suitable complement to color vision. A new method is proposed to segment plant images into their composite surface patches by combining hierarchical color segmentation with quadratic surface fitting using ToF depth data. Experimentation shows that the interpolated depth maps derived from the obtained surfaces fit well the original scenes. Moreover, candidate leaves to be approached by a measuring instrument are ranked, and then robot-mounted cameras move closer to them to validate their suitability to being sampled. Some ambiguities arising from leaves overlap or occlusions are cleared up in this way. The work is a proof-of-concept that dense color data combined with sparse depth as provided by a ToF camera yields a good enough 3D approximation for automated plant measuring at the high throughput imposed by the application., This research is partially funded by the EU GARNICS project FP7-247947, by the Spanish Ministry of Science and Innovation under projects DPI2008-06022 and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission. G. Alenyà was supported by the CSIC under a Jae-Doc Fellowship. B. Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Indoor and outdoor depth imaging of leaves with time-of-flight and stereo vision sensors: Analysis and comparison

  • Kazmi, Wajahat
  • Foix, Sergi
  • Alenyà, Guillem
  • Andersen, Hans Jørgen
In this article we analyze the response of Time-of-Flight (ToF) cameras (active sensors) for close range imaging under three different illumination conditions and compare the results with stereo vision (passive) sensors. ToF cameras are sensitive to ambient light and have low resolution but deliver high frame rate accurate depth data under suitable conditions. We introduce metrics for performance evaluation over a small region of interest. Based on these metrics, we analyze and compare depth imaging of leaf under indoor (room) and outdoor (shadow and sunlight) conditions by varying exposure times of the sensors. Performance of three different ToF cameras (PMD CamBoard, PMD CamCube and SwissRanger SR4000) is compared against selected stereo-correspondence algorithms (local correlation and graph cuts). PMD CamCube has better cancelation of sunlight, followed by CamBoard, while SwissRanger SR4000 performs poorly under sunlight. Stereo vision is comparatively more robust to ambient illumination and provides high resolution depth data but is constrained by texture of the object along with computational efficiency. Graph cut based stereo correspondence algorithm can better retrieve the shape of the leaves but is computationally much more expensive as compared to local correlation. Finally, we propose a method to increase the dynamic range of ToF cameras for a scene involving both shadow and sunlight exposures at the same time by taking advantage of camera flags (PMD) or confidence matrix (SwissRanger). © 2013 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS)., This work is supported by the Danish Council for Strategic Research under project ASETA (www.aseta.dk) grant no. 09-067027, the Spanish Ministry of Science and Innovation under projects PAU + DPI2011-27510, the EU project GARNICS (FP7-247947) and the Catalan Research Commission (SGR-00155). Wajahat Kazmi is funded by ASETA and Sergi Foix is supported by PhD fellowship from CSIC's JAE program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations

  • Dellen, Babette
  • Husain, Farzad
  • Torras, Carme
Presentado a la 8th International Conference on Computer Vision Theory and Applications celebrada en Barcelona del 21 al 24 de febrero de 2013., A novel framework for joint segmentation and tracking in depth videos of object surfaces is presented. Initially, the 3D colored point cloud obtained using the Kinect camera is used to segment the scene into surface patches, defined by quadratic functions. The computed segments together with their functional descriptions are then used to partition the depth image of the subsequent frame in a consistent manner with respect to the precedent frame. This way, solutions established in previous frames can be reused which improves the efficiency of the algorithm and the coherency of the segmentations along the movie. The algorithm is tested for scenes showing human and robot manipulations of objects. We demonstrate that the method can successfully segment and track the human/robot arm and object surfaces along the manipulations. The performance is evaluated quantitatively by measuring the temporal coherency of the segmentations and the segmentation covering using ground truth. The method provides a visual front-end designed for robotic applications, and can potentially be used in the context of manipulation recognition, visual servoing, and robot-grasping tasks., This research is partially funded by the EU projects GARNICS (FP7-247947) and IntellAct (FP7-269959), and the Grup consolidat 2009 SGR155. B. Dellen acknowledges support from the Spanish Ministry of Science and Innovation through a Ramon y Cajal program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Characterization of textile grasping experiments

  • Alenyà, Guillem
  • Ramisa, Arnau
  • Moreno-Noguer, Francesc
  • Torras, Carme
Presentado al International Conference on Robotics and Automation celebrado en USA del 14 al 18 de mayo de 2012., Grasping highly deformable objects, like textiles, is an emerging area of research that involves both perception and manipulation abilities. As new techniques appear, it becomes essential to design strategies to compare them. However, this is not an easy task, since the large state-space of textile objects explodes when coupled with the variability of grippers, robotic hands and robot arms performing the manipulation task. This high variability makes it very difficult to design experiments to evaluate the performance of a system in a repeatable way and compare it to others. We propose a framework to allow the comparison of different grasping methods for textile objects. Instead of measuring each component separately, we therefore propose a methodology to explicitly measure the vision-manipulation correlation by taking into account the throughput of the actions. Perceptions of deformable objects should be grouped into different clusters, and the different grasping actions available should be tested for each perception type to obtain the action-perception success ratio. This characterization potentially allows to compare very different systems in terms of specialized actions, perceptions or widely useful actions, along with the cost of performing each action. We will also show that this categorization is useful in manipulation planning of deformable objects., This work was supported by the Spanish Ministry of Science and Innovation under projects PAU+ DPI2011-27510, by the EU project INTELLACT 247947 FP7-269959 and by the Catalan Research Commission through SGR-00155. A. Ramisa worked under the JAE-DOC grant from the CSIC and the FSE., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Plant leaf imaging using time of flight camera under sunlight, shadow and room conditions

  • Kazmi, Wajahat
  • Foix, Sergi
  • Alenyà, Guillem
Presentado al ROSE 2012 celebrado en Magdeburgo (Alemania) del 16 al 18 de noviembre., In this article, we analyze the effects of ambient light on Time of Flight (ToF) depth imaging for a plant's leaf in sunlight, shadow and room conditions. ToF imaging is sensitive to ambient light and we try to find the best possible integration times (IT) for each condition. This is important in order to optimize the camera calibration. Our analysis is based on several statistical metrics estimated from the ToF data. We explain the estimation of the metrics and propose a method of predicting the deteriorating behavior of the data in each condition using camera flags. Finally, we also propose a method to improve the quality of a ToF image taken in a mixed condition having different ambient light exposures., This work is supported by the Danish Council for Strategic Research under project ASETA (www.aseta.dk) grant no. 09-067027, the Spanish Ministry of Science and Innovation under projects PAU + DPI2011-27510, the EU project GARNICS (FP7-247947) and the Catalan Research Commission (SGR-00155). W. Kazmi is funded by ASETA and S. Foix is supported by PhD fellowship from CSIC’s JAE program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robotic leaf probing via segmentation of range data into surface patches

  • Alenyà, Guillem
  • Dellen, Babette
  • Foix, Sergi
  • Torras, Carme
Presentado al International Conference on Intelligent Robots and Systems (IROS AGROBOTICS), Workshop on Agricultural Robotics: Enabling Safe, Efficient, Affordable Robots for Food Production celebrado en Portugal del 7 al 12 de octubre de 2012., We present a novel method for the robotized probing of plant leaves using Time-of-Flight (ToF) sensors. Plant images are segmented into surface patches by combining a segmentation of the infrared intensity image, provided by the ToF camera, with quadratic surface fitting using ToF depth data. Leaf models are fitted to the boundaries of the segments and used to determine probing points and to evaluate the suitability of leaves for being sampled. The robustness of the approach is evaluated by repeatedly placing an especially adapted, robot-mounted spad meter on the probing points which are extracted in an automatic manner. The number of successful chlorophyll measurements is counted, and the total time for processing the visual data and probing the plant with the robot is measured for each trial. In case of failure, the underlying causes are determined and reported, allowing a better assessment of the applicability of the method in real scenarios., This research is partially funded by the EU GARNICS project FP7-247947, by the Spanish Ministry of Science and Innovation under projects PAU+ and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission. B. Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal program. S. Foix is supported by PhD fellowship from CSIC’s JAE program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Active perception of deformable objects using 3D cameras

  • Alenyà, Guillem
  • Moreno-Noguer, Francesc
  • Ramisa, Arnau
  • Torras, Carme
Presentado al Workshop de Robótica Experimental celebrado en Sevilla del 28 al 29 de noviembre de 2011., Perception and manipulation of rigid objects has received a lot of attention, and several solutions have been proposed. In contrast, dealing with deformable objects is a relatively new and challenging task because they are more complex to model, their state is difficult to determine, and self-occlusions are common and hard to estimate. In this paper we present our progress/results in the perception of deformable objects both using conventional RGB cameras and active sensing strategies by means of depth cameras. We provide insights in two different areas of application: grasping of textiles and plant leaf modelling., This research is partially funded by the EU GARNICS project FP7-247947, by the Spanish Ministry of Science and Innovation under project DPI2008-06022 and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Learning weakly correlated cause-effects for gardening with a cognitive system

  • Agostini, Alejandro
  • Torras, Carme
  • Wörgötter, Florentin
We propose a cognitive system that combines artificial intelligence techniques for planning and learning to execute tasks involving delayed and variable correlations between the actions executed and their expected effects. The system is applied to the task of controlling the growth of plants, where the evolution of the plant attributes strongly depends on different events taking place in the temporally distant past history of the plant. The main problem to tackle is how to efficiently detect these past events. This is very challenging since the inclusion of time could make the dimensionality of the search space extremely large and the collected training instances may only provide very limited information about the relevant combinations of events. To address this problem we propose a learning method that progressively identifies those events that are more likely to produce a sequence of changes under a plant treatment. Since the number of experiences is very limited compared to the size of the event space, we use a probabilistic estimate that takes into account the lack of experience to prevent biased estimations. Planning operators are generated from most accurately predicted sequences of changes. Planning and learning are integrated in a decision-making framework that operates without task interruptions by allowing a human gardener to instruct the treatments when the knowledge acquired so far is not enough to make a decision., This research was supported by the European Community Seventh Framework ProgrammeFP7/2007–2013 – Challenge 2 – Cognitive Systems, Interaction, Robotics – under Grant agreement No 247947 – GARNICS., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Using ToF and RGBD cameras for 3D robot perception and manipulation in human environments

  • Alenyà, Guillem
  • Foix, Sergi
  • Torras, Carme
Robots, traditionally confined into factories, are nowadays moving to domestic and assistive environments, where they need to deal with complex object shapes, deformable materials, and pose uncertainties at human pace. To attain quick 3D perception, new cameras delivering registered depth and intensity images at a high frame rate hold a lot of promise, and therefore many robotics researchers are now experimenting with structured-light RGBD and Time-of-Flight (ToF) cameras. In this paper both technologies are critically compared to help researchers to evaluate their use in real robots. The focus is on 3D perception at close distances for different types of objects that may be handled by a robot in a human environment. We review three robotics applications. The analysis of several performance aspects indicates the complementarity of the two camera types, since the user-friendliness and higher resolution of RGBD cameras is counterbalanced by the capability of ToF cameras to operate outdoors and perceive details., This research is partially funded by the EU GARNICS project FP7-247947, by CSIC project MANIPlus 201350E102, by the Spanish Ministry of Science and Innovation under project PAU+DPI2011-27510, and the Catalan Research Commission under Grant SGR-155., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

ToF cameras for active vision in robotics

  • Alenyà, Guillem
  • Foix, Sergi
  • Torras, Carme
ToF cameras are now a mature technology that is widely being adopted to provide sensory input to robotic applications. Depending on the nature of the objects to be perceived and the viewing distance, we distinguish two groups of applications: those requiring to capture the whole scene and those centered on an object. It will be demonstrated that it is in this last group of applications, in which the robot has to locate and possibly manipulate an object, where the distinctive characteristics of ToF cameras can be better exploited. After presenting the physical sensor features and the calibration requirements of such cameras, we review some representative works highlighting for each one which of the distinctive ToF characteristics have been more essential. Even if at low resolution, the acquisition of 3D images at frame-rate is one of the most important features, as it enables quick background/ foreground segmentation. A common use is in combination with classical color cameras. We present three developed applications, using a mobile robot and a robotic arm, to exemplify with real images some of the stated advantages., This work was supported by the EU project GARNICS FP7-247947, by the Spanish Ministry of Science and Innovation under project PAU+ DPI2011-27510, and by the Catalan Research Commission through SGR-00155, Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Modeling leaf growth of rosette plants using infrared stereo image sequences

  • Aksoy, Eren Erdal
  • Abramov, Alexey
  • Wörgötter, Florentin
  • Scharr, Hanno
  • Fischbach, Andreas
  • Dellen, Babette
In this paper, we present a novel multi-level procedure for finding and tracking leaves of a rosette plant, in our case up to 3 weeks old tobacco plants, during early growth from infrared-image sequences. This allows measuring important plant parameters, e.g. leaf growth rates, in an automatic and non-invasive manner. The procedure consists of three main stages: preprocessing, leaf segmentation, and leaf tracking. Leaf-shape models are applied to improve leaf segmentation, and further used for measuring leaf sizes and handling occlusions. Leaves typically grow radially away from the stem, a property that is exploited in our method, reducing the dimensionality of the tracking task. We successfully tested the method on infrared image sequences showing the growth of tobacco-plant seedlings up to an age of about 30. days, which allows measuring relevant plant growth parameters such as leaf growth rate. By robustly fitting a suitably modified autocatalytic growth model to all growth curves from plants under the same treatment, average plant growth models could be derived. Future applications of the method include plant-growth monitoring for optimizing plant production in green houses or plant phenotyping for plant research., The research leading to these results has received funding from the European Community’s Seventh Framework Programme FP7/2007-2013 – Challenge 2 – Cognitive Systems, Interaction, Robotics – under grant agreement No 247947 – GARNICS. Babette Dellen acknowledges support from the Spanish Ministry for Science and Innovation through a Ramon y Cajal program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Learning algorithms for robot manipulation of clothing and plant leaves

  • Torras, Carme
Trabajo presentado al ICRA Workshop on Advances in Robot Manipulation of Clothes and Flexible Objects, celebrado el 1 de junio de 2014 en Hong Kong., Research supported by the European projects PACO-PLUS, GARNICS and IntellAct, and the Spanish project PAU+., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robot manipulation in human environments: Challenges for learning algorithms

  • Torras, Carme
Resumen del trabajo presentado al Dagstuhl Seminar 2014 celebrado en Dagstuhl (Alemania) del 17 al 21 de febrero de 2014., The European projects PACO-PLUS, GARNICS and IntellAct, the Spanish projects PAU and PAU+, and the Catalan grant SGR-155., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Growth signatures of rosette plants from time-lapse video

  • Dellen, Babette
  • Scharr, Hanno
  • Torras, Carme
Plant growth is a dynamic process, and the precise course of events during early plant development is of major interest for plant research. In this work, we investigate the growth of rosette plants by processing time-lapse videos of growing plants, where we use Nicotiana tabacum (tobacco) as a model plant. In each frame of the video sequences, potential leaves are detected using a leaf-shape model. These detections are prone to errors due to the complex shape of plants and their changing appearance in the image, depending on leaf movement, leaf growth, and illumination conditions. To cope with this problem, we employ a novel graph-based tracking algorithm which can bridge gaps in the sequence by linking leaf detections across a range of neighboring frames. We use the overlap of fitted leaf models as a pairwise similarity measure, and forbid graph edges that would link leaf detections within a single frame. We tested the method on a set of tobacco-plant growth sequences, and could track the first leaves of the plant, including partially or temporarily occluded ones, along complete sequences, demonstrating the applicability of the method to automatic plant growth analysis. All seedlings displayed approximately the same growth behavior, and a characteristic growth signature was found., The research leading to these results has received funding from the European Commission Seventh Framework Programme FP7/2007-2013 Challenge 2 Cognitive Systems, Interaction, Robotics under grant agreement No 247947 GARNICS. B. Dellen acknowledges support from the Spanish Ministry for Science and Innovation through a Ramon y Cajal program., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Service robots for citizens of the future

  • Torras, Carme
This paper summarizes the author’s presentation at the 26th Annual Conference of the Academia Europaea, Barcelona, July 2014., Erasmus Lecture 2014, Robots are no longer confined to factories; they are progressively spreading to urban, social and assistive domains. In order to become handy co-workers and helpful assistants, they must be endowed with quite different abilities from their industrial ancestors. Research on service robots aims to make them intrinsically safe to people, easy to teach by non-experts, able to manipulate not only rigid but also deformable objects, and highly adaptable to non-predefined and dynamic environments. Robots worldwide will share object and environmental models, their acquired knowledge and experiences through global databases and, together with the internet of things, will strongly change the citizens' way of life in so-called smart cities. This raises a number of social and ethical issues that are now being debated not only within the Robotics community but by society at large., The described research has been partially supported by the European projects PACO-PLUS (IST-FP6-IP-027657), GARNICS (FP7-ICT-247947) and IntellAct (FP7-ICT-269959), and the Spanish project PAU + (DPI2011-27510)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Tesis doctoral (DoctoralThesis)

Perceiving dynamic environments: from surface geometry to semantic representation

  • Husain, Farzad
Universitat Politècnica de Catalunya, Programa de Doctorat: Automàtica, Robòtica i Visió., [EN]: Perceiving human environments is becoming increasingly fundamental with the gradual adaptation of robots for domestic use. High-level tasks such as the recognition of objects and actions need to be performed far the active engagement of the robot with its surroundings. Nowadays, the environment is primarily captured using visual information in the form of color and depth images. Visual cues obtained from these images serve as a base upan which perception-related applications are developed. Far example, using appearance models far detecting objects and extracting motion infarmation far recognizing actions. However, given the complex variations of naturally occurring scenes, extracting a set of robust visual cues becomes harder here than in other ontexts. In this thesis, we develop a hierarchy of tools to improve, the different aspects of robot perception in human-centered, possibly dynamic, environments. We start with the segmentation of single images and extend it to videos. Afterwards, we develop a surface tracking approach along with the incorporation of our video segmentation method. We then investigate the higher-level tasks of semantic segmentation and recognition. Finally, we focus on recognizing actions in videos. The introduction of Kinectstyle depth sensors is relatively new and its usage in the field of robotics cannot be found befare half a decade ago. Such sensors enable the acquisition of high-resolution color and depth images at a low cost. Given this opportunity, we dedícate a bulk of our work to the exploitation of the depth infarmation obtained using such sensors, thereby pushing forward the state-of-the-art in perception problems. The thesis is conceptually grouped into two parts. In the first part, we address the low-level tasks of segmentation and tracking with depth images. In many cases, depth data gives a better disambiguation of surface boundaries of different objects in a scene when compared to their color counterpart. We exploit this information in a novel depth segmentation scheme that fits quadratic surface models on different surfaces in a competing fashion . We further extend the method to the video domain by initializing the segmentation results and surface model parameters from the previous trame for the next trame. In this way, we successfully create a video segmentation algorithm, in which the segment label belonging to each surface becomes coherent over time. We also devise a particle-filter-based tracker that uses depth data to track a surface. The tracker is made more robust by combining it with our video segmentation approach. The segmentation results serve as a useful prior for high-level tasks. In the second part we deal with such tasks which include (i) object recognition, (ii) pixelwise object class segmentation, and (iii) action recognition . We propase (i) to address object recognition by creating context-aware conditional random field models. We show the importance of the context in object recognition by modeling geometrical relations between different objects in a scene. We perform (ii) object class segmentation using a convolutional neural network. We introduce a novel distance-from-wall feature and demonstrate its effectiveness in generating better class proposals for objects that are clase to the walls. The final part of the thesis deals with (iii) action recognition. We propase a 2D convolutional neural network extended to a concatenated 3D network that learns to extrae! features from the spatio-temporal domain of raw video data. The network is trained to predict an action label for each video. In summary, several perception aspects are addressed with the utilization of depth infarmation where available. Our main contributions are (a) the introduction of a depth video segmentation scheme, (b) a graphical model far object recognition, and our proposals of the deep learning models for (e) object class segmentation and (d) action recognition., [ES]: Los sistemas de percepción en entornos humanos son cada vez más importantes para la adaptación gradual de los robots a tareas domésticas. Tareas de alto nivel, tales como el reconocimiento de objetos y acciones, son necesarias para conseguir la participación activa del robot en dichas tareas. Hoy en día el entorno del robot es capturado principalmente usando información visual en forma de imágenes de color y profundidad. Las características visuales obtenidas a partir de estas imágenes sirven como base para el desarrollo de aplicaciones relacionadas con la percepción del robot. Por ejemplo, el uso de modelos de apariencia para la detección de objetos y la extracción de información del movimiento para el reconocimiento de acciones. Sin embargo, dado que las escenas pueden contener variaciones complejas, la extracción de un conjunto de características visuales puede convertirse en una tarea muy difícil. En la presente tesis hemos desarrollado una jerarquía de herramientas para mejorar diferentes aspectos de la percepción del robot en entornos humanos, posiblemente dinámicos. Esta tesis comienza con la segmentación de imágenes individuales, que luego se extiende a vídeos. Posteriormente, diseñamos un enfoque de seguimiento de superficies que incorpora nuestro método de segmentación de vídeos. A continuación, investigamos tareas de alto nivel para la segmentación semántica y el reconocimiento. Finalmente, nos centramos en el reconocimiento de acciones en vídeos. La introducción de sensores de profundidad tipo Kinect es relativamente nueva y su uso en el campo de la robótica empezó hace tan solo media década. Tales sensores permiten la adquisición de color y profundidad de imágenes de alta resolución a bajo coste. Dada esta oportunidad, dedicamos una buena parte de nuestro trabajo a la explotación de la información de profundidad obtenida a través de dichos sensores, mejorando el estado del arte en problemas de percepción. La tesis está conceptualmente dividida en dos partes. En primer lugar, abordamos las tareas de bajo nivel de segmentación y seguimiento con imágenes de profundidad. En muchos casos, los datos, de profundidad permiteuna mejor desambiguación de los límites de las superficies de diferentes objetos de una escena en comparación con los datos de color. Explotamos esta información en un nuevo esquema de segmentación de profundidad que ajusta modelos cuadráticos de superficies de forma competitiva. Extendemos el método a vídeos de modo que la etiquetación de superficies resulte coherente en el tiempo. También proponemos un rastreador basado en un filtro de partículas que utiliza los datos de profundidad para realizar el seguimiento de una superficie. El seguimiento se hace más robusto al combinarlo con nuestro enfoque de segmentación en vídeo. Los resultados de la segmentación son usados como información a priori para tareas de alto nivel. En la segunda parte nos ocupamos de este tipo de tareas que incluyen el (i) reconocimiento de objetos, (ii) la segmentación de clases de objetos a nivel de píxeles, y (iii) el reconocimiento de acciones. Proponemos (i) abordar el reconocimiento de objetos mediante la creación de modelos de campos aleatorios condicionales sensibles al contexto. Realizamos (ii) la segmentación de la clase del objeto utilizando una red neuronal de convolución. Se introduce una nueva característica de distancia-a-paredes y demostramos su eficacia en la mejora de la clasificación de objetos que están cerca de las paredes. La parte final de la tesis se ocupa del (iii) reconocimiento de acciones. Proponemos una red neuronal de convolución 2D extendida a una red 3D concatenada, que aprende a extraer las características del dominio espacio-temporal de los datos de vídeo. La red está capacitada para predecir la etiqueta de acción para cada vídeo., This work has been sponsored by the following: A PhD Scholarship and Projects GARNICS (FP7-ICT-2009-4-247947), CINNOVA (201150E088), PAU+(DPI2011-27510), IntellAct (FP7-ICT2009-6-269959) and Textil Rob (201550E028)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Tesis doctoral (DoctoralThesis)

Task-oriented viewpoint planning for free-form objects

  • Foix, Sergi
A thesis submitted to the Universitat Politècnica de Catalunya to obtain the degree of Doctor of Philosophy. Doctoral programme: Automatic Control, Robotics and Computer Vision. This thesis was completed at: Institut de Robòtica i Informàtica Industrial, CSIC-UPC., [EN]: This thesis deals with active sensing and its use in real exploration tasks under both scene ambiguities and measurement uncertainties. While object modeling is the implicit objective of most of active sensing algorithms, in this work we have explored new strategies to deal with more generic and more complex tasks. Active sensing requires the ability of moving the perceptual system to gather new information. Our approach uses a robot manipulator with a 3D Time-of-Flight (ToF) camera attached to the end-effector. For a complex task, we have focused our attention on plant phenotyping. Plants are complex objects, with leaves that change their position and size along time. Valid viewpoints for a certain plant are hardly valid for a different one, even belonging to the same species. Some instruments, such as chlorophyll meters or disk sampling tools, require being precisely positioned over a particular location of the leaf. Therefore, their use requires the modeling of specific regions of interest of the plant, including also the free space needed for avoiding obstacles and approaching the leaf with tool. It is easy to observe that predefined camera trajectories are not valid here, and that usually with one single view it is very difficult to acquire all the required information. The overall objective of this thesis is to solve complex active sensing tasks by embedding their exploratory goal into a pre-estimated geometrical model, using information-gain as the fundamental guideline for the reward function. The main contributions can be divided in two groups: first, the evaluation of ToF cameras and their calibration to assess the uncertainty of the measurements (presented in Part I); and second, the proposal of a framework capable of embedding the task, modeled as free and occupied space, and that takes into account the modeled sensor's uncertainty to improve the action selection algorithm (presented in Part II). This thesis, has given rise to 14 publications, including 5 indexed journals, and its results have been used in the GARNICS European project. The complete framework is based on the Next-Best-View methodology and it can be summarized in the following main steps. First, an initial view of the object (e.g., a plant) is acquired. From this initial view and given a set of candidate viewpoints, the expected gain obtained by moving the robot and acquiring the next image is computed. This computation takes into account the uncertainty from all the different pixels of the sensor, the expected information based on a predefined task model, and the possible occlusions. Once the most promising view is selected, the robot moves, takes a new image, integrates this information intothe model, and evaluates again the set of remaining views. Finally, the task terminates when enough information is gathered. In our examples, this process enables the robot to perform a measurement on top of a leaf. The key ingredient is to model the complexity of the task in a layered representation of free-occupied occupancy grid maps. This allows to naturally encode the requirements of the task, to maintain and update the belief state with the measurements performed, to simulate and compute the expected gains of all potential viewpoints, and to encode the termination condition. During this work the technology of ToF cameras has incredibly evolved. Nowadays it is very popular and ToF cameras are already embedded in some consumer devices. Although the quality of the measurements has been considerably improved, it is still not uniform in the sensor. We believe, as it has been demonstrated in various experiments in this work, that a careful modeling of the sensor's uncertainty is highly beneficial and helps to design better decision systems. In our case, it enables a more realistic computation of the information gain measure, and consequently, a better selection criterion., [CA]: Aquesta tesi aborda el tema de la percepció activa i el seu ús en tasques d'exploració en entorns reals tot considerant la ambigüitat en l'escena i la incertesa del sistema de percepció. Al contrari de la majoria d'algoritmes de percepció activa, on el modelatge d'objectes sol ser l'objectiu implícit, en aquesta tesi hem explorat noves estratègies per poder tractar tasques genèriques i de major complexitat. Tot sistema de percepció activa requereix un aparell sensorial amb la capacitat de variar els seus paràmetres de forma controlada, per poder, d'aquesta manera, recopilar nova informació per resoldre una tasca determinada. En tasques d'exploració, la posició i orientació del sensor són paràmetres claus per resoldre la tasca. En el nostre estudi hem fet ús d'un robot manipulador com a sistema de posicionament i d'una càmera de profunditat de temps de vol (ToF), adherida al seu efector final, com a sistema de percepció. Com a tasca final, ens hem concentrat en l'adquisició de mesures sobre fulles dins de l'àmbit del fenotipatge de les plantes. Les plantes son objectes molt complexos, amb fulles que canvien de textura, posició i mida al llarg del temps. Això comporta diverses dificultats. Per una banda, abans de dur a terme una mesura sobre un fulla s'ha d'explorar l'entorn i trobar una regió que ho permeti. A més a més, aquells punts de vista que han estat adequats per una determinada planta difícilment ho seran per una altra, tot i sent les dues de la mateixa espècie. Per un altra banda, en el moment de la mesura, certs instruments, tals com els mesuradors de clorofil·la o les eines d'extracció de mostres, requereixen ser posicionats amb molta precisió. És necessari, doncs, disposar d'un model detallat d'aquestes regions d'interès, i que inclogui no només l'espai ocupat sinó també el lliure. Gràcies a la modelització de l'espai lliure es pot dur a terme una bona evitació d'obstacles i un bon càlcul de la trajectòria d'aproximació de l'eina a la fulla. En aquest context, és fàcil veure que, en general, amb un sol punt de vista, no n'hi haprou per adquirir tota la informació necessària per prendre una mesura, i que l'ús de trajectòries predeterminades no garanteixen l'èxit. L'objectiu general d'aquesta tesi és resoldre tasques complexes de percepció activa mitjançant la codificació del seu objectiu d'exploració en un model geomètric prèviament estimat, fent servir el guany d'informació com a guia fonamental dins de la funció de cost. Les principals contribucions d'aquesta tesi es poden dividir en dos grups: primer, l'avaluació de les càmeres ToF i el seu calibratge per poder avaluar la incertesa de les seves mesures (presentat en la Part I); i en segon lloc, la proposta d'un sistema capaç de codificar la tasca mitjançant el modelatge de l'espai lliure i ocupat, i que té en compte la incertesa del sensor per millorar la selecció de les accions (presentat en la Part II). Aquesta tesi ha donat lloc a 14 publicacions, incloent 5 en revistes indexades, i els resultats obtinguts s'han fet servir en el projecte Europeu GARNICS. La funcionalitat del sistema complet està basada en els mètodes Next-Best-View (següent-millor-vista) i es pot desglossar en els següents passos principals. En primer lloc, s'obté una vista inicial de l'objecte (p. ex., una planta). A partir d'aquesta vista inicial i d'un conjunt de vistes candidates, s'estima, per cada una d'elles, el guany d'informació resultant, tant de moure la càmera com d'obtenir una nova mesura. És rellevant dir que aquest càlcul té en compte la incertesa de cada un dels píxels del sensor, l'estimació de la informació basada en el model de la tasca preestablerta i les possibles oclusions. Un cop seleccionada la vista més prometedora, el robot es mou a la nova posició, pren una nova imatge, integra aquesta informació en el model i torna a avaluar, un altre cop, el conjunt de punts de vista restants. Per últim, la tasca acaba en el moment que es recopila suficient informació., This work has been partially supported by a JAE fellowship of the Spanish Scientific Research Council (CSIC), the Spanish Ministry of Science and Innovation, the Catalan Research Commission and the European Commission under the research projects: DPI2008-06022: PAU: Percepción y acción ante incertidumbre. DPI2011-27510: PAU+: Perception and Action in Robotics Problems with Large State Spaces. 201350E102: MANIPlus: Manipulación robotizada de objetos deformables. 2009-SGR-155: SGR ROBÒTICA: Grup de recerca consolidat - Grup de Robòtica. FP6-2004-IST-4-27657: EU PACO PLUS project. FP7-ICT-2009-4-247947: GARNICS: Gardening with a cognitive system. FP7-ICT-2009-6-269959: IntellAct: Intelligent observation and execution of Actions and manipulations., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

A cognitive architecture for automatic gardening

  • Agostini, Alejandro
  • Alenyà, Guillem
  • Fischbach, Andreas
  • Scharr, Hanno
  • Wörgötter, Florentin
  • Torras, Carme
In large industrial greenhouses, plants are usually treated following well established protocols for watering, nutrients, and shading/light. While this is practical for the automation of the process, it does not tap the full potential for optimal plant treatment. To more efficiently grow plants, specific treatments according to the plant individual needs should be applied. Experienced human gardeners are very good at treating plants individually. Unfortunately, hiring a crew of gardeners to carry out this task in large greenhouses is not cost effective. In this work we present a cognitive system that integrates artificial intelligence (AI) techniques for decision-making with robotics techniques for sensing and acting to autonomously treat plants using a real-robot platform. Artificial intelligence techniques are used to decide the amount of water and nutrients each plant needs according to the history of the plant. Robotic techniques for sensing measure plant attributes (e.g. leaves) from visual information using 3D model representations. These attributes are used by the AI system to make decisions about the treatment to apply. Acting techniques execute robot movements to supply the plants with the specified amount of water and nutrients., This research was supported by the commission of the European Communities Seventh Framework Programme FP7/2007-2013 - Challenge 2 - Cognitive Systems, Interaction, Robotics - under grant agreement No. 247947 - GARNICS., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Characterization of textile grasping experiments

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Ramisa Ayats, Arnau
  • Moreno-Noguer, Francesc
  • Torras, Carme|||0000-0002-2933-398X
Grasping highly deformable objects, like textiles, is an emerging area of research that involves both percep- tion and manipulation abilities. As new techniques appear, it becomes essential to design strategies to compare them. However, this is not an easy task, since the large state-space of textile objects explodes when coupled with the variability of grippers, robotic hands and robot arms performing the manipulation task. This high variability makes it very difficult to design experiments to evaluate the performance of a system in a repeatable way and compare it to others. We propose a framework to allow the comparison of different grasping methods for textile objects. Instead of measuring each component separately, we there- fore propose a methodology to explicitly measure the vision- manipulation correlation by taking into account the throughput of the actions. Perceptions of deformable objects should be grouped into different clusters, and the different grasping actions available should be tested for each perception type to obtain the action-perception success ratio. This characterization potentially allows to compare very different systems in terms of specialized actions , perceptions or widely useful actions , along with the cost of performing each action. We will also show that this categorization is useful in manipulation planning of deformable objects., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Using ToF and RGBD cameras for 3D robot perception and manipulation in human environments

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Torras, Carme|||0000-0002-2933-398X
Robots, traditionally confined into factories, are nowadays moving to domestic and assistive environments, where they need to deal with complex object shapes, deformable materials, and pose uncertainties at human pace. To attain quick 3D perception, new cameras delivering registered depth and intensity images at a high frame rate hold a lot of promise, and therefore many robotics researchers are now experimenting with structured-light RGBD and Time-of-Flight (ToF) cameras. In this paper both technologies are critically compared to help researchers to evaluate their use in real robots. The focus is on 3D perception at close distances for different types of objects that may be handled by a robot in a human environment. We review three robotics applications. The analysis of several performance aspects indicates the complementarity of the two camera types, since the user-friendliness and higher resolution of RGBD cameras is counterbalanced by the capability of ToF cameras to operate outdoors and perceive details., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Deformation and illumination invariant feature point descriptor

  • Moreno-Noguer, Francesc
Recent advances in 3D shape recognition have shown that kernels based on diffusion geometry can be effectively used to describe local features of deforming surfaces. In this paper, we introduce a new framework that allows using these kernels on 2D local patches, yielding a novel feature point descriptor that is both invariant to non-rigid image deformations and illumination changes. In order to build the descriptor, 2D image patches are embedded as 3D surfaces, by multiplying the intensity level by an arbitrarily large and constant weight that favors anisotropic diffusion and retains the gradient magnitude information. Patches are then described in terms of a heat kernel signature, which is made invariant to intensity changes, rotation and scaling. The resulting feature point descriptor is proven to be significantly more discriminative than state of the art ones, even those which are specifically designed for describing non-rigid image deformations.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robust elastic 2D/3D geometric graph matching

  • Serradell, Eduard
  • Kybic, J.
  • Moreno-Noguer, Francesc
  • Fua, Pascal
We present an algorithm for geometric matching of graphs embedded in 2D or 3D space. It is applicable for registering any graph-like structures appearing in biomedical images, such as blood vessels, pulmonary bronchi, nerve fibers, or dendritic arbors. Our approach does not rely on the similarity of local appearance features, so it is suitable for multimodal registration with a large difference in appearance. Unlike earlier methods, the algorithm uses edge shape, does not require an initial pose estimate, can handle partial matches, and can cope with nonlinear deformations and topological differences. The matching consists of two steps. First, we find an affine transform that roughly aligns the graphs by exploring the set of all consistent correspondences between the nodes. This can be done at an acceptably low computational expense by using parameter uncertainties for pruning, backtracking as needed. Parameter uncertainties are updated in a Kalman-like scheme with each match. In the second step we allow for a nonlinear part of the deformation, modeled as a Gaussian Process. Short sequences of edges are grouped into superedges, which are then matched between graphs. This allows for topological differences. A maximum consistent set of superedge matches is found using a dedicated branch-and-bound solver, which is over 100 times faster than a standard linear programming approach. Geometrical and topological consistency of candidate matches is determined in a fast hierarchical manner. We demonstrate the effectiveness of our technique at registering angiography and retinal fundus images, as well as neural image stacks., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Tesis doctoral (DoctoralThesis)

Perceiving dynamic environments : from surface geometry to semantic representation

  • Husain, Syed Farzad
Perceiving human environments is becoming increasingly fundamental with the gradual adaptation of robots for domestic use. High-level tasks such as the recognition of objects and actions need to be performed far the active engagement of the robot with its surroundings. Nowadays, the environment is primarily captured using visual information in the form of color and depth images. Visual cues obtained from these images serve as a base upan which perception-related applications are developed. Far example, using appearance models far detecting objects and extracting motion infarmation far recognizing actions. However, given the complex variations of naturally occurring scenes, extracting a set of robust visual cues becomes harder here than in other contexts. In this thesis, we develop a hierarchy of tools to improve the different aspects of robot perception in human-centered, possibly dynamic, environments. We start with the segmentation of single images and extend it to videos. Afterwards, we develop a surface tracking approach along with the incorporation of our video segmentation method. We then investigate the higher-level tasks of semantic segmentation and recognition. Finally, we focus on recognizing actions in videos. The introduction of Kinectstyle depth sensors is relatively new and its usage in the field of robotics cannot be found befare half a decade ago. Such sensors enable the acquisition of high-resolution color and depth images at a low cost. Given this opportunity, we dedícate a bulk of our work to the exploitation of the depth infarmation obtained using such sensors, thereby pushing forward the state-of-the-art in perception problems. The thesis is conceptually grouped into two parts. In the first part, we address the low-level tasks of segmentation and tracking with depth images. In many cases, depth data gives a better disambiguation of surface boundaries of different objects in a scene when compared to their color counterpart. We exploit this information in a novel depth segmentation scheme that fits quadratic surface models on different surfaces in a competing fashion . We further extend the method to the video domain by initializing the segmentation results and surface model parameters from the previous trame for the next trame. In this way, we successfully create a video segmentation algorithm, in which the segment label belonging to each surface becomes coherent over time. We also devise a particle-filter-based tracker that uses depth data to track a surface. The tracker is made more robust by combining it with our video segmentation approach. The segmentation results serve as a useful prior for high-level tasks. In the second part we deal with such tasks which include (i) object recognition, (ii) pixelwise object class segmentation, and (iii) action recognition . We propase (i) to address object recognition by creating context-aware conditional random field models. We show the importance of the context in object recognition by modeling geometrical relations between different objects in a scene. We perform (ii) object class segmentation using a convolutional neural network. We introduce a novel distance-from-wall feature and demonstrate its effectiveness in generating better class proposals for objects that are clase to the walls. The final part of the thesis deals with (iii) action recognition. We propase a 2D convolutional neural network extended to a concatenated 3D network that learns to extrae! features from the spatio-temporal domain of raw video data. The network is trained to predict an action label for each video. In summary, several perception aspects are addressed with the utilization of depth infarmation where available. Our main contributions are (a) the introduction of a depth video segmentation scheme, (b) a graphical model far object recognition, and our proposals of the deep learning models for (e) object class segmentation and (d) action recognition., Los sistemas de percepción en entornos humanos son cada vez más importantes para la adaptación gradual de los robots a tareas domésticas. Tareas de alto nivel, tales como el reconocimiento de objetos y acciones, son necesarias para conseguir la participación activa del robot en dichas tareas. Hoy en día el entorno del robot es capturado principalmente usando información visual en forma de imágenes de color y profundidad. Las características visuales obtenidas a partir de estas imágenes sirven como base para el desarrollo de aplicaciones relacionadas con la percepción del robot. Por ejemplo, el uso de modelos de apariencia para la detección de objetos y la extracción de información del movimiento para el reconocimiento de acciones. Sin embargo, dado que las escenas pueden contener variaciones complejas, la extracción de un conjunto de características visuales puede convertirse en una tarea muy difícil. En la presente tesis hemos desarrollado una jerarquía de herramientas para mejorar diferentes aspectos de la percepción del robot en entornos humanos, posiblemente dinámicos. Esta tesis comienza con la segmentación de imágenes individuales, que luego se extiende a vídeos. Posteriormente, diseñamos un enfoque de seguimiento de superficies que incorpora nuestro método de segmentación de vídeos. A continuación, investigamos tareas de alto nivel para la segmentación semántica y el reconocimiento. Finalmente, nos centramos en el reconocimiento de acciones en vídeos. La introducción de sensores de profundidad tipo Kinect es relativamente nueva y su uso en el campo de la robótica empezó hace tan solo media década. Tales sensores permiten la adquisición de color y profundidad de imágenes de alta resolución a bajo coste. Dada esta oportunidad, dedicamos una buena parte de nuestro trabajo a la explotación de la información de profundidad obtenida a través de dichos sensores, mejorando el estado del arte en problemas de percepción. La tesis está conceptualmente dividida en dos partes. En primer lugar, abordamos las tareas de bajo nivel de segmentación y seguimiento con imágenes de profundidad. En muchos casos, los datos de profundidad permite una mejor desambiguación de los límites de las superficies de diferentes objetos de una escena en comparación con los datos de color. Explotamos esta información en un nuevo esquema de segmentación de profundidad que ajusta modelos cuadráticos de superficies de forma competitiva. Extendemos el método a vídeos de modo que la etiquetación de superficies resulte coherente en el tiempo. También proponemos un rastreador basado en un filtro de partículas que utiliza los datos de profundidad para realizar el seguimiento de una superficie. El seguimiento se hace más robusto al combinarlo con nuestro enfoque de segmentación en vídeo. Los resultados de la segmentación son usados como información a priori para tareas de alto nivel. En la segunda parte nos ocupamos de este tipo de tareas que incluyen el (i) reconocimiento de objetos, (ii) la segmentación de clases de objetos a nivel de píxeles, y (iii) el reconocimiento de acciones. Proponemos (i) abordar el reconocimiento de objetos mediante la creación de modelos de campos aleatorios condicionales sensibles al contexto. Realizamos (ii) la segmentación de la clase del objeto utilizando una red neuronal de convolución. Se introduce una nueva característica de distancia-a-paredes y demostramos su eficacia en la mejora de la clasificación de objetos que están cerca de las paredes. La parte final de la tesis se ocupa del (iii) reconocimiento de acciones. Proponemos una red neuronal de convolución 2D extendida a una red 3D concatenada, que aprende a extraer las características del dominio espacio-temporal de los datos de vídeo. La red está capacitada para predecir la etiqueta de acción para cada vídeo.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Combining geometric and appearance priors for robust homography estimation

  • Serradell, Eduard
  • Özuysa, Mustafa
  • Lepetit, Vincent
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Abstract. The homography between pairs of images are typically computed from the correspondence of keypoints, which are established by using image descriptors. When these descriptors are not reliable, either because of repetitive patterns or large amounts of clutter, additional priors need to be considered. The Blind PnP algorithm makes use of geometric priors to guide the search for matches while computing camera pose. Inspired by this, we propose a novel approach for homography estimation that combines geometric priors with appearance priors of ambiguous descriptors. More specifically, for each point we retain its best candidates according to appearance. We then prune the set of potential matches by iteratively shrinking the regions of the image that are consistent with the geometric prior. We can then successfully compute homographies between pairs of images containing highly repetitive patterns and even under oblique viewing conditions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Lock-in time-of-flight (ToF) cameras: a survey

  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Torras, Carme|||0000-0002-2933-398X
This paper reviews the state-of-the art in the field of lock-in ToF cameras, their advantages, their limitations, the existing calibration methods, and the way they are being used, sometimes in combination with other sensors. Even though lockin ToF cameras provide neither higher resolution nor larger ambiguity-free range compared to other range map estimation systems, advantages such as registered depth and intensity data at a high frame rate, compact design, low weight and reduced power consumption have motivated their increasing usage in several research areas, such as computer graphics, machine vision and robotics., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Towards plant monitoring through next best view

  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Torras, Carme|||0000-0002-2933-398X
Monitoring plants using leaf feature detection is a challenging perception task because different leaves, even from the same plant, may have very different shapes, sizes and deformations. In addition, leaves may be occluded by other leaves making it hard to determine some of their characteristics. In this paper we use a Time-of-Flight (ToF) camera mounted on a robot arm to acquire the depth information needed for plant leaf detection. Under a Next Best View (NBV) paradigm, we propose a criterion to compute a new camera position that offers a better view of a target leaf. The proposed criterion exploits some typical errors of the ToF camera, which are common to other 3D sensing devices as well. This approach is also useful when more than one leaf is segmented as the same region, since moving the camera following the same NBV criterion helps to disambiguate this situation.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Active perception of deformable objects using 3D cameras

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Moreno-Noguer, Francesc
  • Ramisa Ayats, Arnau
  • Torras, Carme|||0000-0002-2933-398X
Perception and manipulation of rigid objects has received a lot of attention, and several solutions have been proposed. In contrast, dealing with deformable objects is a relatively new and challenging task because they are more complex to model, their state is difficult to determine, and self-occlusions are common and hard to estimate. In this paper we present our progress/results in the perception of deformable objects both using conventional RGB cameras and active sensing strategies by means of depth cameras.We provide insights in two different areas of application: grasping of textiles and plant leaf modelling.
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

ToF cameras for active vision in robotics

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Torras, Carme|||0000-0002-2933-398X
ToF cameras are now a mature technology that is widely being adopted to provide sensory input to robotic applications. Depending on the nature of the objects to be perceived and the viewing distance, we distinguish two groups of applications: those requiring to capture the whole scene and those centered on an object. It will be demonstrated that it is in this last group of applications, in which the robot has to locate and possibly manipulate an object, where the distinctive characteristics of ToF cameras can be better exploited. After presenting the physical sensor features and the calibration requirements of such cameras, we review some representative works highlighting for each one which of the distinctive ToF characteristics have been more essential. Even if at low resolution, the acquisition of 3D images at frame-rate is one of the most important features, as it enables quick background/foreground segmentation. A common use is in combination with classical color cameras. We present three developed applications, using a mobile robot and a robotic arm, to exemplify with real images some of the stated advantages., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Depth-supported real-time video segmentation with the Kinect

  • Abramov, Alexey
  • Pauwels, Karl
  • Papon, Jeremie
  • Wörgötter, Florentin
  • Dellen, Babette
We present a real-time technique for the spatiotemporal segmentation of color/depth movies. Images are segmented using a parallel Metropolis algorithm implemented on a GPU utilizing both color and depth information, acquired with the Microsoft Kinect. Segments represent the equilibrium states of a Potts model, where tracking of segments is achieved by warping obtained segment labels to the next frame using real-time optical flow, which reduces the number of iterations required for the Metropolis method to encounter the new equilibrium state. By including depth information into the framework, true objects boundaries can be found more easily, improving also the temporal coherency of the method. The algorithm has been tested for videos of medium resolutions showing human manipulations of objects. The framework provides an inexpensive visual front end for visual preprocessing of videos in industrial settings and robot labs which can potentially be used in various applications., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Probabilistic simultaneous pose and non-rigid shape recovery

  • Moreno-Noguer, Francesc
  • Porta Pleite, Josep Maria|||0000-0002-5056-1717
We present an algorithm to simultaneously recover non-rigid shape and camera poses from point correspondences between a reference shape and a sequence of input images. The key novel contribution of our approach is in bringing the tools of the probabilistic SLAM methodology from a rigid to a deformable domain. Under the assumption that the shape may be represented as a weighted sum of deformation modes, we show that the problem of estimating the modal weights along with the camera poses, may be probabilistically formulated as a maximum a posterior estimate and solved using an iterative least squares optimization. An extensive evaluation on synthetic and real data, shows that our approach has several significant advantages over current approaches, such as performing robustly under large amounts of noise and outliers, and neither requiring to track points over the whole sequence nor initializations close from the ground truth solution.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations

  • Dellen, Babette
  • Husain, Syed Farzad
  • Torras, Carme|||0000-0002-2933-398X
A novel framework for joint segmentation and tracking in depth videos of object surfaces is presented. Initially, the 3D colored point cloud obtained using the Kinect camera is used to segment the scene into surface patches, defined by quadratic functions. The computed segments together with their functional descriptions are then used to partition the depth image of the subsequent frame in a consistent manner with respect to the precedent frame. This way, solutions established in previous frames can be reused which improves the efficiency of the algorithm and the coherency of the segmentations along the movie. The algorithm is tested for scenes showing human and robot manipulations of objects. We demonstrate that the method can successfully segment and track the human/robot arm and object surfaces along the manipulations. The performance is evaluated quantitatively by measuring the temporal coherency of the segmentations and the segmentation covering using ground truth. The method provides a visual front-end designed for robotic applications, and can potentially be used in the context of manipulation recognition, visual servoing, and robot-grasping tasks, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Estudio, informe, memoria (Report)

A general strategy for interactive decision-making in robotic platforms

  • Agostini, Alejandro Gabriel
  • Torras, Carme|||0000-0002-2933-398X
  • Wörgötter, Florentin
This work presents an intergated strategy for planning and learning suitable to execute tasks with robotic platforms without any previous task specification. The approach rapidly learns planning operators from few action experiences using a competitive strategy where many alternatives of cause-effect explanations are evaluated in parallel, and the most successful ones are used to generate the operators. The system operates without task interruption by integrating in the planning-learning loop a human teacher that supports the planner in making decisions. All the mechanisms are integrated and synchronized in the robot using a general decision-making framework.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Integrating task planning and interactive learning for robots to work in human environments

  • Agostini, Alejandro Gabriel
  • Torras, Carme|||0000-0002-2933-398X
  • Wörgötter, Florentin
Human environments are challenging for robots, which need to be trainable by lay people and learn new behaviours rapidly without disrupting much the ongoing activity. A system that integrates AI techniques for planning and learning is here proposed to satisfy these strong demands. The approach rapidly learns planning operators from few action experiences using a competitive strategy where many alternatives of cause-effect explanations are evaluated in parallel, and the most successful ones are used to generate the operators. The success of a cause-effect explanation is evaluated by a probabilistic estimate that compensates the lack of experience, producing more confident estimations and speeding up the learning in relation to other known estimates. The system operates without task interruption by integrating in the planning-learning loop a human teacher that supports the planner in making decisions. All the mechanisms are integrated and synchronized in the robot using a general decision-making framework. The feasibility and scalability of the architecture are evaluated in two different robot platforms: a Stäubli arm, and the humanoid ARMAR III., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Sharpening haptic inputs for teaching a manipulation skill to a robot

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
Gaussian mixtures-based learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration (PbD). Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. In this work we have used force/torque feedback through a haptic device for teaching a teleoperated robot to empty a rigid container. Structure vibrations and container inertia appeared to considerably disrupt the sensing process, so a filtering algorithm had to be devised. Moreover, some input variables seemed much more relevant to the particular task to be learned than others, which lead us to analyze the training data in order to select those relevant features through principal component analysis and a mutual information criterion. Then, a batch version of GMM/GMR [1], [2] was implemented using different training datasets (original, pre-processed data through PCA and MI). Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Channel coding for joint colour and depth segmentation

  • Wallenberg, Markus
  • Felsberg, Michael
  • Forssen, Per-Erik
  • Dellen, Babette
Segmentation is an important preprocessing step in many applications. Compared to colour segmentation, fusion of colour and depth greatly improves the segmentation result. Such a fusion is easy to do by stacking measurements in di erent value dimensions, but there are better ways. In this paper we perform fusion using the channel representation, and demonstrate how a state-of-the-art segmentation algorithm can be modi ed to use channel values as inputs. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using the superparamagnetic clustering algorithm. Our experiments show that depth gradients are more useful than depth values for segmentation, and that channel coding both colour and depth gradients makes tuned parameter settings generalise better to novel images., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Segmenting color images into surface patches by exploiting sparse depth data

  • Dellen, Babette
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Torras, Carme|||0000-0002-2933-398X
We present a new method for segmenting color images into their composite surfaces by combining color segmentation with model-based fitting utilizing sparse depth data, acquired using time-of-flight (Swissranger, PMD CamCube) and stereo techniques. The main target of our work is the segmentation of plant structures, i.e., leaves, from color-depth images, and the extraction of color and 3D shape information for automating manipulation tasks. Since segmentation is performed in the dense color space, even sparse, incomplete, or noisy depth information can be used. This kind of data often represents a major challenge for methods operating in the 3D data space directly. To achieve our goal, we construct a three-stage segmentation hierarchy by segmenting the color image with different resolutions-assuming that “true” surface boundaries must appear at some point along the segmentation hierarchy. 3D surfaces are then fitted to the color-segment areas using depth data. Those segments which minimize the fitting error are selected and used to construct a new segmentation. Then, an additional region merging and a growing stage are applied to avoid over-segmentation and label previously unclustered points. Experimental results demonstrate that the method is successful in segmenting a variety of domestic objects and plants into quadratic surfaces. At the end of the procedure, the sparse depth data is completed using the extracted surface models, resulting in dense depth maps. For stereo, the resulting disparity maps are compared with ground truth and the average error is computed., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Indoor and outdoor depth imaging of leaves with time-of-flight and stereo vision sensors: analysis and comparison

  • Kazmi, Wajahat
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Andersen, Hans Jørgen
In this article we analyze the response of Time-of-Flight (ToF) cameras (active sensors) for close range imaging under three different illumination conditions and compare the results with stereo vision (passive) sensors. ToF cameras are sensitive to ambient light and have low resolution but deliver high frame rate accurate depth data under suitable conditions. We introduce metrics for performance evaluation over a small region of interest. Based on these metrics, we analyze and compare depth imaging of leaf under indoor (room) and outdoor (shadow and sunlight) conditions by varying exposure times of the sensors. Performance of three different ToF cameras (PMD CamBoard, PMD CamCube and SwissRanger SR4000) is compared against selected stereo-correspondence algorithms (local correlation and graph cuts). PMD CamCube has better cancelation of sunlight, followed by CamBoard, while SwissRanger SR4000 performs poorly under sunlight. Stereo vision is comparatively more robust to ambient illumination and provides high resolution depth data but is constrained by texture of the object along with computational efficiency. Graph cut based stereo correspondence algorithm can better retrieve the shape of the leaves but is computationally much more expensive as compared to local correlation. Finally, we propose a method to increase the dynamic range of ToF cameras for a scene involving both shadow and sunlight exposures at the same time by taking advantage of camera flags (PMD) or confidence matrix (SwissRanger). (C) 2013 International Society for Photogrammetly and Remote Sensing, Inc. (ISPRS) Published by Elsevier B.V. All rights reserved., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Learning weakly correlated cause-effects for gardening with a cognitive system

  • Agostini, Alejandro Gabriel
  • Torras, Carme|||0000-0002-2933-398X
  • Worgotter, Florentin
We propose a cognitive system that combines artificial intelligence techniques for planning and learning to execute tasks involving delayed and variable correlations between the actions executed and their expected effects. The system is applied to the task of controlling the growth of plants, where the evolution of the plant attributes strongly depends on different events taking place in the temporally distant past history of the plant. The main problem to tackle is how to efficiently detect these past events. This is very challenging since the inclusion of time could make the dimensionality of the search space extremely large and the collected training instances may only provide very limited information about the relevant combinations of events. To address this problem we propose a learning method that progressively identifies those events that are more likely to produce a sequence of changes under a plant treatment. Since the number of experiences is very limited compared to the size of the event space, we use a probabilistic estimate that takes into account the lack of experience to prevent biased estimations. Planning operators are generated from most accurately predicted sequences of changes. Planning and learning are integrated in a decision-making framework that operates without task interruptions by allowing a human gardener to instruct the treatments when the knowledge acquired so far is not enough to make a decision., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

General robot kinematics decomposition without intermediate markers

  • Ulbrich, Stefan
  • Ruiz de Angulo García, Vicente|||0000-0002-2067-7399
  • Asfour, Tamim
  • Torras, Carme|||0000-0002-2933-398X
  • Dillmann, Rüdiger
The calibration of serial manipulators with high numbers of degrees of freedom by means of machine learning is a complex and time-consuming task. With the help of a simple strategy, this complexity can be drastically reduced and the speed of the learning procedure can be increased: When the robot is virtually divided into shorter kinematic chains, these subchains can be learned separately and, hence, much more efficiently than the complete kinematics. Such decompositions, however, require either the possibility to capture the poses of all endeffectors of all subchains at the same time, or they are limited to robots that fulfill special constraints. In this work, an alternative decomposition is presented that does not suffer from these limitations. An offline training algorithm is provided in which the composite subchains are learned sequentially with dedicated movements. A second training scheme is provided to train composite chains simultaneously and online. Both schemes can be used together with many machine learning algorithms. In the simulations, an algorithm using Parameterized Self-Organizing Maps (PSOM) modified for online learning and Gaussian Mixture Models (GMM) were chosen to show the correctness of the approach. The experimental results show that, using a two-fold decomposition, the number of samples required to reach a given precision, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous correspondence and non-rigid 3D reconstruction of the coronary tree from single X-ray images

  • Serradell, Eduard
  • Romero, Adriana
  • Leta, Ruben
  • Gatta, Carlo
  • Moreno-Noguer, Francesc
We present a novel approach to simultaneously reconstruct the 3D structure of a non-rigid coronary tree and estimate point correspondences between an input X-ray image and a reference 3D shape. At the core of our approach lies an optimization scheme that iteratively fits a generative 3D model of increasing complexity and guides the matching process. As a result, and in contrast to existing approaches that assume rigidity or quasi-rigidity of the structure, our method is able to retrieve large non-linear deformations even when the input data is corrupted by the presence of noise and partial occlusions. We extensively evaluate our approach under synthetic and real data and demonstrate a remarkable improvement compared to state-of-the-art., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Outdoor view recognition based on landmark grouping and logistic regression

  • Todt, Eduardo
  • Torras, Carme|||0000-0002-2933-398X
Vision-based robot localization outdoors has remained more elusive than its indoors counterpart. Drastic illumination changes and the scarceness of suitable landmarks are the main difficulties. This paper attempts to surmount them by deviating from the main trend of using local features. Instead, a global descriptor called landmark-view is defined, which aggregates the most visually-salient landmarks present in each scene. Thus, landmark co-occurrence and spatial and saliency relationships between them are added to the single landmark characterization, based on saliency and color distribution. A suitable framework to compare landmark-views is developed, and it is shown how this remarkably enhances the recognition performance, compared against single landmark recognition. A view-matching model is constructed using logistic regression. Experimentation using 45 views, acquired outdoors, containing 273 landmarks, yielded good recognition results. The overall percentage of correct view classification obtained was 80.6%, indicating the adequacy of the approach., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration of force-based tasks with multiple solution trajectories

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
A learning framework with a bidirectional communication channel is proposed, where a human performs several demonstrations of a task using a haptic device (providing him/her with force-torque feedback) while a robot captures these executions using only its force-based perceptive system. Our work departs from the usual approaches to learning by demonstration in that the robot has to execute the task blindly, relying only on force-torque perceptions, and, more essential, we address goal-driven manipulation tasks with multiple solution trajectories, whereas most works tackle tasks that can be learned by just finding a generalization at the trajectory level. To cope with these multiple-solution tasks, in our framework demonstrations are represented by means of a Hidden Markov Model (HMM) and the robot reproduction of the task is performed using a modified version of Gaussian Mixture Regression that incorporates temporal information (GMRa) through the forward variable of the HMM. Also, we exploit the haptic device as a teaching and communication tool in a human-robot interaction context, as an alternative to kinesthetic-based teaching systems. Results show that the robot is able to learn a container-emptying task relying only on force-based perceptions and to achieve the goal from several non-trained initial conditions.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous pose, correspondence and non-rigid shape

  • Sánchez-Riera, Jordi
  • Ostlund, Jonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Recent works have shown that 3D shape of non-rigid surfaces can be accurately retrieved from a single image given a set of 3D-to-2D correspondences between that image and another one for which the shape is known. However, existing approaches assume that such correspondences can be readily established, which is not necessarily true when large deformations produce significant appearance changes between the input and the reference images. Furthermore, it is either assumed that the pose of the camera is known, or the estimated solution is pose-ambiguous. In this paper we relax all these assumptions and, given a set of 3D and 2D unmatched points, we present an approach to simultaneously solve their correspondences, compute the camera pose and retrieve the shape of the surface in the input image. This is achieved by introducing weak priors on the pose and shape that we model as Gaussian Mixtures. By combining them into a Kalman filter we can progressively reduce the number of 2D candidates that can be potentially matched to each 3D point, while pose and shape are refined. This lets us to perform a complete and efficient exploration of the solution space and retain the best solution.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Shared random Ferns for efficient detection of multiple categories

  • Villamizar Vergel, Michael Alejandro
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
We propose a new algorithm for detecting multiple object categories that exploits the fact that different categories may share common features but with different geometric distributions. This yields an efficient detector which, in contrast to existing approaches, considerably reduces the computation cost at runtime, where the feature computation step is traditionally the most expensive. More specifically, at the learning stage we compute common features by applying the same Random Ferns over the Histograms of Oriented Gradients on the training images. We then apply a boosting step to build discriminative weak classifiers, and learn the specific geometric distribution of the Random Ferns for each class. At runtime, only a few Random Ferns have to be densely computed over each input image, and their geometric distribution allows performing the detection. The proposed method has been validated in public datasets achieving competitive detection results, which are comparable with state-of-the-art methods that use specific features per class., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Learning the semantics of object-action relations by observation

  • Aksoy, Eren Erdal
  • Abramov, Alexey
  • Dörr, Johannes
  • Ning, Kejun
  • Dellen, Babette
  • Wörgötter, Florentin
Recognizing manipulations performed by a human and the transfer and execution of this by a robot is a difficult problem. We address this in the current study by introducing a novel representation of the relations between objects at decisive time points during a manipulation. Thereby, we encode the essential changes in a visual scenery in a condensed way such that a robot can recognize and learn a manipulation without prior object knowledge. To achieve this we continuously track image segments in the video and construct a dynamic graph sequence. Topological transitions of those graphs occur whenever a spatial relation between some segments has changed in a discontinuous way and these moments are stored in a transition matrix called the semantic event chain (SEC). We demonstrate that these time points are highly descriptive for distinguishing between different manipulations. Employing simple sub-string search algorithms, SECs can be compared and type-similar manipulations can be recognized with high confidence. As the approach is generic, statistical learning can be used to find the archetypal SEC of a given manipulation class. The performance of the algorithm is demonstrated on a set of real videos showing hands manipulating various objects and performing different actions. In experiments with a robotic arm, we show that the SEC can be learned by observing human manipulations, transferred to a new scenario, and then reproduced by the machine., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robotic leaf probing via segmentation of range data into surface patches

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Dellen, Babette
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Torras, Carme|||0000-0002-2933-398X
We present a novel method for the robotized probing of plant leaves using Time-of-Flight (ToF) sensors. Plant images are segmented into surface patches by combining a segmentation of the infrared intensity image, provided by the ToF camera, with quadratic surface fitting using ToF depth data. Leaf models are fitted to the boundaries of the segments and used to determine probing points and to evaluate the suitability of leaves for being sampled. The robustness of the approach is evaluated by repeatedly placing an especially adapted, robot-mounted spad meter on the probing points which are extracted in an automatic manner. The number of successful chlorophyll measurements is counted, and the total time for processing the visual data and probing the plant with the robot is measured for each trial. In case of failure, the underlying causes are determined and reported, allowing a better assessment of the applicability of the method in real scenarios., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Service robots for citizens of the future

  • Torras, Carme|||0000-0002-2933-398X
Robots are no longer confined to factories; they are progressively spreading to urban, social and assistive domains. In order to become handy co-workers and helpful assistants, they must be endowed with quite different abilities from their industrial ancestors. Research on service robots aims to make them intrinsically safe to people, easy to teach by non-experts, able to manipulate not only rigid but also deformable objects, and highly adaptable to non-predefined and dynamic environments. Robots worldwide will share object and environmental models, their acquired knowledge and experiences through global databases and, together with the internet of things, will strongly change the citizens’ way of life in so-called smart cities. This raises a number of social and ethical issues that are now being debated not only within the Robotics community but by society at large., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Growth signatures of rosette plants from time-lapse video

  • Dellen, Babette
  • Scharr, Hanno
  • Torras, Carme|||0000-0002-2933-398X
Plant growth is a dynamic process, and the precise course of events during early plant development is of major interest for plant research. In this work, we investigate the growth of rosette plants by processing time-lapse videos of growing plants, where we use Nicotiana tabacum (tobacco) as a model plant. In each frame of the video sequences, potential leaves are detected using a leaf-shape model. These detections are prone to errors due to the complex shape of plants and their changing appearance in the image, depending on leaf movement, leaf growth, and illumination conditions. To cope with this problem, we employ a novel graph-based tracking algorithm which can bridge gaps in the sequence by linking leaf detections across a range of neighboring frames. We use the overlap of fitted leaf models as a pairwise similarity measure, and forbid graph edges that would link leaf detections within a single frame. We tested the method on a set of tobacco-plant growth sequences, and could track the first leaves of the plant, including partially or temporarily occluded ones, along complete sequences, demonstrating the applicability of the method to automatic plant growth analysis. All seedlings displayed approximately the same growth behavior, and a characteristic growth signature was found., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robust non-rigid registration of 2D and 3D graphs

  • Serradell, Eduard
  • Glowacki, Przemyslaw
  • Jan, Kybic
  • Moreno-Noguer, Francesc
  • Fua, Pascal
We present a new approach to matching graphs embedded in R2 or R3. Unlike earlier methods, our approach does not rely on the similarity of local appearance features, does not require an initial alignment, can handle partial matches, and can cope with non-linear deformations and topological differences. To handle arbitrary non-linear deformations, we represent them as Gaussian Processes. In the absence of appearance information, we iteratively establish correspondences between graph nodes, update the structure accordingly, and use the current mapping estimate to find the most likely correspondences that will be used in the next iteration. This makes the computation tractable. We demonstrate the effectiveness of our approach first on synthetic cases and then on angiography data, retinal fundus images, and microscopy image stacks acquired at very different resolutions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Robotized plant probing: leaf segmentation utilizing time-of-flight data

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Dellen, Babette
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Torras, Carme|||0000-0002-2933-398X
Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modeling for this application has strong demands, particularly in what concerns three-dimensional (3-D) information gathering and speed., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Tesis doctoral (DoctoralThesis)

Task-oriented viewpoint planning for free-form objects

  • Foix Salmerón, Sergi|||0000-0001-9249-6696
This thesis deals with active sensing and its use in real exploration tasks under both scene ambiguities and measurement uncertainties. While object modeling is the implicit objective of most of active sensing algorithms, in this work we have explored new strategies to deal with more generic and more complex tasks. Active sensing requires the ability of moving the perceptual system to gather new information. Our approach uses a robot manipulator with a 3D Time-of-Flight (ToF) camera attached to the end-effector. For a complex task, we have focused our attention on plant phenotyping. Plants are complex objects, with leaves that change their position and size along time. Valid viewpoints for a certain plant are hardly valid for a different one, even belonging to the same species. Some instruments, such as chlorophyll meters or disk sampling tools, require being precisely positioned over a particular location of the leaf. Therefore, their use requires the modeling of specific regions of interest of the plant, including also the free space needed for avoiding obstacles and approaching the leaf with tool. It is easy to observe that predefined camera trajectories are not valid here, and that usually with one single view it is very difficult to acquire all the required information. The overall objective of this thesis is to solve complex active sensing tasks by embedding their exploratory goal into a pre-estimated geometrical model, using information-gain as the fundamental guideline for the reward function. The main contributions can be divided in two groups: first, the evaluation of ToF cameras and their calibration to assess the uncertainty of the measurements (presented in Part I); and second, the proposal of a framework capable of embedding the task, modeled as free and occupied space, and that takes into account the modeled sensor's uncertainty to improve the action selection algorithm (presented in Part II). This thesis has given rise to 14 publications, including 5 indexed journals, and its results have been used in the GARNICS European project. The complete framework is based on the Next-Best-View methodology and it can be summarized in the following main steps. First, an initial view of the object (e.g., a plant) is acquired. From this initial view and given a set of candidate viewpoints, the expected gain obtained by moving the robot and acquiring the next image is computed. This computation takes into account the uncertainty from all the different pixels of the sensor, the expected information based on a predefined task model, and the possible occlusions. Once the most promising view is selected, the robot moves, takes a new image, integrates this information intothe model, and evaluates again the set of remaining views. Finally, the task terminates when enough information is gathered. In our examples, this process enables the robot to perform a measurement on top of a leaf. The key ingredient is to model the complexity of the task in a layered representation of free-occupied occupancy grid maps. This allows to naturally encode the requirements of the task, to maintain and update the belief state with the measurements performed, to simulate and compute the expected gains of all potential viewpoints, and to encode the termination condition. During this work the technology of ToF cameras has incredibly evolved. Nowadays it is very popular and ToF cameras are already embedded in some consumer devices. Although the quality of the measurements has been considerably improved, it is still not uniform in the sensor. We believe, as it has been demonstrated in various experiments in this work, that a careful modeling of the sensor's uncertainty is highly beneficial and helps to design better decision systems. In our case, it enables a more realistic computation of the information gain measure, and consequently, a better selection criterion., Aquesta tesi aborda el tema de la percepció activa i el seu ús en tasques d'exploració en entorns reals tot considerant la ambigüitat en l'escena i la incertesa del sistema de percepció. Al contrari de la majoria d'algoritmes de percepció activa, on el modelatge d'objectes sol ser l'objectiu implícit, en aquesta tesi hem explorat noves estratègies per poder tractar tasques genèriques i de major complexitat. Tot sistema de percepció activa requereix un aparell sensorial amb la capacitat de variar els seus paràmetres de forma controlada, per poder, d'aquesta manera, recopilar nova informació per resoldre una tasca determinada. En tasques d'exploració, la posició i orientació del sensor són paràmetres claus per resoldre la tasca. En el nostre estudi hem fet ús d'un robot manipulador com a sistema de posicionament i d'una càmera de profunditat de temps de vol (ToF), adherida al seu efector final, com a sistema de percepció. Com a tasca final, ens hem concentrat en l'adquisició de mesures sobre fulles dins de l'àmbit del fenotipatge de les plantes. Les plantes son objectes molt complexos, amb fulles que canvien de textura, posició i mida al llarg del temps. Això comporta diverses dificultats. Per una banda, abans de dur a terme una mesura sobre un fulla s'ha d'explorar l'entorn i trobar una regió que ho permeti. A més a més, aquells punts de vista que han estat adequats per una determinada planta difícilment ho seran per una altra, tot i sent les dues de la mateixa espècie. Per un altra banda, en el moment de la mesura, certs instruments, tals com els mesuradors de clorofil·la o les eines d'extracció de mostres, requereixen ser posicionats amb molta precisió. És necessari, doncs, disposar d'un model detallat d'aquestes regions d'interès, i que inclogui no només l'espai ocupat sinó també el lliure. Gràcies a la modelització de l'espai lliure es pot dur a terme una bona evitació d'obstacles i un bon càlcul de la trajectòria d'aproximació de l'eina a la fulla. En aquest context, és fàcil veure que, en general, amb un sol punt de vista no n'hi ha prou per adquirir tota la informació necessària per prendre una mesura, i que l'ús de trajectòries predeterminades no garanteixen l'èxit. L'objectiu general d'aquesta tesi és resoldre tasques complexes de percepció activa mitjançant la codificació del seu objectiu d'exploració en un model geomètric prèviament estimat, fent servir el guany d'informació com a guia fonamental dins de la funció de cost. Les principals contribucions d'aquesta tesi es poden dividir en dos grups: primer, l'avaluació de les càmeres ToF i el seu calibratge per poder avaluar la incertesa de les seves mesures (presentat en la Part I); i en segon lloc, la proposta d'un sistema capaç de codificar la tasca mitjançant el modelatge de l'espai lliure i ocupat, i que té en compte la incertesa del sensor per millorar la selecció de les accions (presentat en la Part II). Aquesta tesi ha donat lloc a 14 publicacions, incloent 5 en revistes indexades, i els resultats obtinguts s'han fet servir en el projecte Europeu GARNICS. La funcionalitat del sistema complet està basada en els mètodes Next-Best-View (següent-millor-vista) i es pot desglossar en els següents passos principals. En primer lloc, s'obté una vista inicial de l'objecte (p. ex., una planta). A partir d'aquesta vista inicial i d'un conjunt de vistes candidates, s'estima, per cada una d'elles, el guany d'informació resultant, tant de moure la càmera com d'obtenir una nova mesura. És rellevant dir que aquest càlcul té en compte la incertesa de cada un dels píxels del sensor, l'estimació de la informació basada en el model de la tasca preestablerta i les possibles oclusions. Un cop seleccionada la vista més prometedora, el robot es mou a la nova posició, pren una nova imatge, integra aquesta informació en el model i torna a avaluar, un altre cop, el conjunt de punts de vista restants. Per últim, la tasca acaba en el moment que es recopila suficient informació.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

3D modelling of leaves from color and ToF data for robotized plant measuring

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Dellen, Babette
  • Torras, Carme|||0000-0002-2933-398X
Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modelling for this application has strong demands, particularly in what concerns 3D information gathering and speed. This paper shows that Time-of-Flight (ToF) cameras achieve a good compromise between both demands, providing a suitable complement to color vision. A new method is proposed to segment plant images into their composite surface patches by combining hierarchical color segmentation with quadratic surface fitting using ToF depth data. Experimentation shows that the interpolated depth maps derived from the obtained surfaces fit well the original scenes. Moreover, candidate leaves to be approached by a measuring instrument are ranked, and then robot-mounted cameras move closer to them to validate their suitability to being sampled. Some ambiguities arising from leaves overlap or occlusions are cleared up in this way. The work is a proof-of-concept that dense color data combined with sparse depth as provided by a ToF camera yields a good enough 3D approximation for automated plant measuring at the high throughput imposed by the application., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration in the force domain

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
Researchers are becoming aware of the importance of other information sources besides visual data in robot learning by demonstration (LbD). Forcebased perceptions are shown to convey very relevant information – missed by visual and position sensors – for learning specific tasks. In this paper, we review some recent works using forces as input data in LbD and Human-Robot interaction (HRI) scenarios, and propose a complete learning framework for teaching force-based manipulation skills to a robot through a haptic device. We suggest to use haptic interfaces not only as a demonstration tool but also as a communication channel between the human and the robot, getting the teacher more involved in the teaching process by experiencing the force signals sensed by the robot. Within the proposed framework, we provide solutions for treating force signals, extracting relevant information about the task, encoding the training data and generalizing to perform successfully under unknown conditions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Learning force-based robot skills from haptic demonstration

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
Locally weighted as well as Gaussian mixtures learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration. Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. For the first time, force/torque feedback through a haptic device has been used for teaching a teleoperated robot to empty a rigid container. The memory-based LWPLS and the non-memory-based LWPR algorithms [1,2,3], as well as both the batch and the incremental versions of GMM/GMR [4,5] were implemented, their comparison leading to very similar results, with the same pattern as regards to both the involved robot joints and the different initial experimental conditions. Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages, where the taught motion will be refined by autonomous robot rehearsal through reinforcement learning.
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Stochastic exploration of ambiguities for nonrigid shape recovery

  • Moreno-Noguer, Francesc
  • Fua, Pascal
Recovering the 3D shape of deformable surfaces from single images is known to be a highly ambiguous problem because many different shapes may have very similar projections. This is commonly addressed by restricting the set of possible shapes to linear combinations of deformation modes and by imposing additional geometric constraints. Unfortunately, because image measurements are noisy, such constraints do not always guarantee that the correct shape will be recovered. To overcome this limitation, we introduce a stochastic sampling approach to efficiently explore the set of solutions of an objective function based on point correspondences. This allows to propose a small set of ambiguous candidate 3D shapes and then use additional image information to choose the best one. As a proof of concept, we use either motion or shading cues to this end and show that we can handle a complex objective function without having to solve a difficult non-linear minimization problem. The advantages of our method are demonstrated on a variety of problems including both real and synthetic data., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Plant leaf imaging using time of flight camera under sunlight, shadow and room conditions

  • Kazmi, Wajahat
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
In this article, we analyze the effects of ambient light on Time of Flight (ToF) depth imaging for a plant’s leaf in sunlight, shadow and room conditions. ToF imaging is sensitive to ambient light and we try to find the best possible integration times (IT) for each condition. This is important in order to optimize camera calibration. Our analysis is based on several statistical metrics estimated from the ToF data. We explain the estimation of the metrics and propose a method of predicting the deteriorating behavior of the data in each condition using camera flags. Finally, we also propose a method to improve the quality of a ToF image taken in a mixed condition having different ambient light exposures.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Design of non-anthropomorphic robotic hands for anthropomorphic tasks

  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
  • Pérez Gracia, Alba|||0000-0002-6023-7857
In this paper, we explore the idea of designing non- anthropomorphic multi-fingered robotic hands for tasks tha t replicate the motion of the human hand. Taking as input data a finite set of rigid-body positions for the five fingertips, we de- velop a method to perform dimensional synthesis for a kinema tic chain with a tree structure, with five branches that share thr ee common joints. We state the forward kinematics equations of relative dis- placements for each serial chain expressed as dual quaterni ons, and solve for up to five chains simultaneously to reach a numbe r of positions along the hand trajectory. This is done using a h y- brid global numerical solver that integrates a genetic algo rithm and a Levenberg-Marquardt local optimizer. Although the number of candidate solutions in this problem is very high, the use of the genetic algorithm allows us to per form an exhaustive exploration of the solution space to obtain a s et of solutions. We can then choose some of the solutions based on t he specific task to perform. Note that these designs match the ta sk exactly while generally having a finger design radically dif ferent from that of the human hand., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Service robots for citizens of the future

  • Torras, Carme
Robots are no longer confined to factories; they are progressively spreading to urban, social and assistive domains. In order to become handy co-workers and helpful assistants, they must be endowed with quite different abilities from their industrial ancestors. Research on service robots aims to make them intrinsically safe to people, easy to teach by non-experts, able to manipulate not only rigid but also deformable objects, and highly adaptable to non-predefined and dynamic environments. Robots worldwide will share object and environmental models, their acquired knowledge and experiences through global databases and, together with the internet of things, will strongly change the citizens’ way of life in so-called smart cities. This raises a number of social and ethical issues that are now being debated not only within the Robotics community but by society at large., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Plant leaf imaging using time of flight camera under sunlight, shadow and room conditions

  • Kazmi, Wajahat
  • Foix Salmerón, Sergi
  • Alenyà Ribas, Guillem
In this article, we analyze the effects of ambient light on Time of Flight (ToF) depth imaging for a plant’s leaf in sunlight, shadow and room conditions. ToF imaging is sensitive to ambient light and we try to find the best possible integration times (IT) for each condition. This is important in order to optimize camera calibration. Our analysis is based on several statistical metrics estimated from the ToF data. We explain the estimation of the metrics and propose a method of predicting the deteriorating behavior of the data in each condition using camera flags. Finally, we also propose a method to improve the quality of a ToF image taken in a mixed condition having different ambient light exposures., Preprint
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Learning force-based robot skills from haptic demonstration

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Locally weighted as well as Gaussian mixtures learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration. Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. For the first time, force/torque feedback through a haptic device has been used for teaching a teleoperated robot to empty a rigid container. The memory-based LWPLS and the non-memory-based LWPR algorithms [1,2,3], as well as both the batch and the incremental versions of GMM/GMR [4,5] were implemented, their comparison leading to very similar results, with the same pattern as regards to both the involved robot joints and the different initial experimental conditions. Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages, where the taught motion will be refined by autonomous robot rehearsal through reinforcement learning., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Shared random Ferns for efficient detection of multiple categories

  • Villamizar Vergel, Michael Alejandro
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan
  • Sanfeliu Cortés, Alberto
We propose a new algorithm for detecting multiple object categories that exploits the fact that different categories may share common features but with different geometric distributions. This yields an efficient detector which, in contrast to existing approaches, considerably reduces the computation cost at runtime, where the feature computation step is traditionally the most expensive. More specifically, at the learning stage we compute common features by applying the same Random Ferns over the Histograms of Oriented Gradients on the training images. We then apply a boosting step to build discriminative weak classifiers, and learn the specific geometric distribution of the Random Ferns for each class. At runtime, only a few Random Ferns have to be densely computed over each input image, and their geometric distribution allows performing the detection. The proposed method has been validated in public datasets achieving competitive detection results, which are comparable with state-of-the-art methods that use specific features per class., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Robotic leaf probing via segmentation of range data into surface patches

  • Alenyà Ribas, Guillem
  • Dellen, Babette
  • Foix Salmerón, Sergi
  • Torras, Carme
We present a novel method for the robotized probing of plant leaves using Time-of-Flight (ToF) sensors. Plant images are segmented into surface patches by combining a segmentation of the infrared intensity image, provided by the ToF camera, with quadratic surface fitting using ToF depth data. Leaf models are fitted to the boundaries of the segments and used to determine probing points and to evaluate the suitability of leaves for being sampled. The robustness of the approach is evaluated by repeatedly placing an especially adapted, robot-mounted spad meter on the probing points which are extracted in an automatic manner. The number of successful chlorophyll measurements is counted, and the total time for processing the visual data and probing the plant with the robot is measured for each trial. In case of failure, the underlying causes are determined and reported, allowing a better assessment of the applicability of the method in real scenarios., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

General robot kinematics decomposition without intermediate markers

  • Ulbrich, Stefan
  • Ruiz de Angulo García, Vicente
  • Asfour, Tamim
  • Torras, Carme
  • Dillmann, Rüdiger
The calibration of serial manipulators with high numbers of degrees of freedom by means of machine learning is a complex and time-consuming task. With the help of a simple strategy, this complexity can be drastically reduced and the speed of the learning procedure can be increased: When the robot is virtually divided into shorter kinematic chains, these subchains can be learned separately and, hence, much more efficiently than the complete kinematics. Such decompositions, however, require either the possibility to capture the poses of all endeffectors of all subchains at the same time, or they are limited to robots that fulfill special constraints. In this work, an alternative decomposition is presented that does not suffer from these limitations. An offline training algorithm is provided in which the composite subchains are learned sequentially with dedicated movements. A second training scheme is provided to train composite chains simultaneously and online. Both schemes can be used together with many machine learning algorithms. In the simulations, an algorithm using Parameterized Self-Organizing Maps (PSOM) modified for online learning and Gaussian Mixture Models (GMM) were chosen to show the correctness of the approach. The experimental results show that, using a two-fold decomposition, the number of samples required to reach a given precision, Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Learning the semantics of object-action relations by observation

  • Aksoy, Eren Erdal
  • Abramov, Alexey
  • Dörr, Johannes
  • Ning, Kejun
  • Dellen, Babette
  • Wörgötter, Florentin
Recognizing manipulations performed by a human and the transfer and execution of this by a robot is a difficult problem. We address this in the current study by introducing a novel representation of the relations between objects at decisive time points during a manipulation. Thereby, we encode the essential changes in a visual scenery in a condensed way such that a robot can recognize and learn a manipulation without prior object knowledge. To achieve this we continuously track image segments in the video and construct a dynamic graph sequence. Topological transitions of those graphs occur whenever a spatial relation between some segments has changed in a discontinuous way and these moments are stored in a transition matrix called the semantic event chain (SEC). We demonstrate that these time points are highly descriptive for distinguishing between different manipulations. Employing simple sub-string search algorithms, SECs can be compared and type-similar manipulations can be recognized with high confidence. As the approach is generic, statistical learning can be used to find the archetypal SEC of a given manipulation class. The performance of the algorithm is demonstrated on a set of real videos showing hands manipulating various objects and performing different actions. In experiments with a robotic arm, we show that the SEC can be learned by observing human manipulations, transferred to a new scenario, and then reproduced by the machine., Peer Reviewed, Preprint
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations

  • Dellen, Babette
  • Husain, Syed Farzad
  • Torras, Carme
A novel framework for joint segmentation and tracking in depth videos of object surfaces is presented. Initially, the 3D colored point cloud obtained using the Kinect camera is used to segment the scene into surface patches, defined by quadratic functions. The computed segments together with their functional descriptions are then used to partition the depth image of the subsequent frame in a consistent manner with respect to the precedent frame. This way, solutions established in previous frames can be reused which improves the efficiency of the algorithm and the coherency of the segmentations along the movie. The algorithm is tested for scenes showing human and robot manipulations of objects. We demonstrate that the method can successfully segment and track the human/robot arm and object surfaces along the manipulations. The performance is evaluated quantitatively by measuring the temporal coherency of the segmentations and the segmentation covering using ground truth. The method provides a visual front-end designed for robotic applications, and can potentially be used in the context of manipulation recognition, visual servoing, and robot-grasping tasks, Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Sharpening haptic inputs for teaching a manipulation skill to a robot

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Gaussian mixtures-based learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration (PbD). Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. In this work we have used force/torque feedback through a haptic device for teaching a teleoperated robot to empty a rigid container. Structure vibrations and container inertia appeared to considerably disrupt the sensing process, so a filtering algorithm had to be devised. Moreover, some input variables seemed much more relevant to the particular task to be learned than others, which lead us to analyze the training data in order to select those relevant features through principal component analysis and a mutual information criterion. Then, a batch version of GMM/GMR [1], [2] was implemented using different training datasets (original, pre-processed data through PCA and MI). Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Robust elastic 2D/3D geometric graph matching

  • Serradell, Eduard
  • Kybic, J.
  • Moreno-Noguer, Francesc
  • Fua, Pascal
We present an algorithm for geometric matching of graphs embedded in 2D or 3D space. It is applicable for registering any graph-like structures appearing in biomedical images, such as blood vessels, pulmonary bronchi, nerve fibers, or dendritic arbors. Our approach does not rely on the similarity of local appearance features, so it is suitable for multimodal registration with a large difference in appearance. Unlike earlier methods, the algorithm uses edge shape, does not require an initial pose estimate, can handle partial matches, and can cope with nonlinear deformations and topological differences. The matching consists of two steps. First, we find an affine transform that roughly aligns the graphs by exploring the set of all consistent correspondences between the nodes. This can be done at an acceptably low computational expense by using parameter uncertainties for pruning, backtracking as needed. Parameter uncertainties are updated in a Kalman-like scheme with each match. In the second step we allow for a nonlinear part of the deformation, modeled as a Gaussian Process. Short sequences of edges are grouped into superedges, which are then matched between graphs. This allows for topological differences. A maximum consistent set of superedge matches is found using a dedicated branch-and-bound solver, which is over 100 times faster than a standard linear programming approach. Geometrical and topological consistency of candidate matches is determined in a fast hierarchical manner. We demonstrate the effectiveness of our technique at registering angiography and retinal fundus images, as well as neural image stacks., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Characterization of textile grasping experiments

  • Alenyà Ribas, Guillem
  • Ramisa Ayats, Arnau
  • Moreno-Noguer, Francesc
  • Torras, Carme
Grasping highly deformable objects, like textiles, is an emerging area of research that involves both percep- tion and manipulation abilities. As new techniques appear, it becomes essential to design strategies to compare them. However, this is not an easy task, since the large state-space of textile objects explodes when coupled with the variability of grippers, robotic hands and robot arms performing the manipulation task. This high variability makes it very difficult to design experiments to evaluate the performance of a system in a repeatable way and compare it to others. We propose a framework to allow the comparison of different grasping methods for textile objects. Instead of measuring each component separately, we there- fore propose a methodology to explicitly measure the vision- manipulation correlation by taking into account the throughput of the actions. Perceptions of deformable objects should be grouped into different clusters, and the different grasping actions available should be tested for each perception type to obtain the action-perception success ratio. This characterization potentially allows to compare very different systems in terms of specialized actions , perceptions or widely useful actions , along with the cost of performing each action. We will also show that this categorization is useful in manipulation planning of deformable objects., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Design of non-anthropomorphic robotic hands for anthropomorphic tasks

  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
  • Pérez Gracia, Alba
In this paper, we explore the idea of designing non- anthropomorphic multi-fingered robotic hands for tasks tha t replicate the motion of the human hand. Taking as input data a finite set of rigid-body positions for the five fingertips, we de- velop a method to perform dimensional synthesis for a kinema tic chain with a tree structure, with five branches that share thr ee common joints. We state the forward kinematics equations of relative dis- placements for each serial chain expressed as dual quaterni ons, and solve for up to five chains simultaneously to reach a numbe r of positions along the hand trajectory. This is done using a h y- brid global numerical solver that integrates a genetic algo rithm and a Levenberg-Marquardt local optimizer. Although the number of candidate solutions in this problem is very high, the use of the genetic algorithm allows us to per form an exhaustive exploration of the solution space to obtain a s et of solutions. We can then choose some of the solutions based on t he specific task to perform. Note that these designs match the ta sk exactly while generally having a finger design radically dif ferent from that of the human hand., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Towards plant monitoring through next best view

  • Foix Salmerón, Sergi
  • Alenyà Ribas, Guillem
  • Torras, Carme
Monitoring plants using leaf feature detection is a challenging perception task because different leaves, even from the same plant, may have very different shapes, sizes and deformations. In addition, leaves may be occluded by other leaves making it hard to determine some of their characteristics. In this paper we use a Time-of-Flight (ToF) camera mounted on a robot arm to acquire the depth information needed for plant leaf detection. Under a Next Best View (NBV) paradigm, we propose a criterion to compute a new camera position that offers a better view of a target leaf. The proposed criterion exploits some typical errors of the ToF camera, which are common to other 3D sensing devices as well. This approach is also useful when more than one leaf is segmented as the same region, since moving the camera following the same NBV criterion helps to disambiguate this situation., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Simultaneous pose, correspondence and non-rigid shape

  • Sánchez-Riera, Jordi
  • Ostlund, Jonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Recent works have shown that 3D shape of non-rigid surfaces can be accurately retrieved from a single image given a set of 3D-to-2D correspondences between that image and another one for which the shape is known. However, existing approaches assume that such correspondences can be readily established, which is not necessarily true when large deformations produce significant appearance changes between the input and the reference images. Furthermore, it is either assumed that the pose of the camera is known, or the estimated solution is pose-ambiguous. In this paper we relax all these assumptions and, given a set of 3D and 2D unmatched points, we present an approach to simultaneously solve their correspondences, compute the camera pose and retrieve the shape of the surface in the input image. This is achieved by introducing weak priors on the pose and shape that we model as Gaussian Mixtures. By combining them into a Kalman filter we can progressively reduce the number of 2D candidates that can be potentially matched to each 3D point, while pose and shape are refined. This lets us to perform a complete and efficient exploration of the solution space and retain the best solution., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Channel coding for joint colour and depth segmentation

  • Wallenberg, Markus
  • Felsberg, Michael
  • Forssen, Per-Erik
  • Dellen, Babette
Segmentation is an important preprocessing step in many applications. Compared to colour segmentation, fusion of colour and depth greatly improves the segmentation result. Such a fusion is easy to do by stacking measurements in di erent value dimensions, but there are better ways. In this paper we perform fusion using the channel representation, and demonstrate how a state-of-the-art segmentation algorithm can be modi ed to use channel values as inputs. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using the superparamagnetic clustering algorithm. Our experiments show that depth gradients are more useful than depth values for segmentation, and that channel coding both colour and depth gradients makes tuned parameter settings generalise better to novel images., Peer Reviewed, Postprint (updated version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Segmenting color images into surface patches by exploiting sparse depth data

  • Dellen, Babette
  • Alenyà Ribas, Guillem
  • Foix Salmerón, Sergi
  • Torras, Carme
We present a new method for segmenting color images into their composite surfaces by combining color segmentation with model-based fitting utilizing sparse depth data, acquired using time-of-flight (Swissranger, PMD CamCube) and stereo techniques. The main target of our work is the segmentation of plant structures, i.e., leaves, from color-depth images, and the extraction of color and 3D shape information for automating manipulation tasks. Since segmentation is performed in the dense color space, even sparse, incomplete, or noisy depth information can be used. This kind of data often represents a major challenge for methods operating in the 3D data space directly. To achieve our goal, we construct a three-stage segmentation hierarchy by segmenting the color image with different resolutions-assuming that “true” surface boundaries must appear at some point along the segmentation hierarchy. 3D surfaces are then fitted to the color-segment areas using depth data. Those segments which minimize the fitting error are selected and used to construct a new segmentation. Then, an additional region merging and a growing stage are applied to avoid over-segmentation and label previously unclustered points. Experimental results demonstrate that the method is successful in segmenting a variety of domestic objects and plants into quadratic surfaces. At the end of the procedure, the sparse depth data is completed using the extracted surface models, resulting in dense depth maps. For stereo, the resulting disparity maps are compared with ground truth and the average error is computed., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Robotized plant probing: leaf segmentation utilizing time-of-flight data

  • Alenyà Ribas, Guillem
  • Dellen, Babette
  • Foix Salmerón, Sergi
  • Torras, Carme
Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modeling for this application has strong demands, particularly in what concerns three-dimensional (3-D) information gathering and speed., Peer Reviewed, Preprint
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

3D modelling of leaves from color and ToF data for robotized plant measuring

  • Alenyà Ribas, Guillem
  • Dellen, Babette
  • Torras, Carme
Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modelling for this application has strong demands, particularly in what concerns 3D information gathering and speed. This paper shows that Time-of-Flight (ToF) cameras achieve a good compromise between both demands, providing a suitable complement to color vision. A new method is proposed to segment plant images into their composite surface patches by combining hierarchical color segmentation with quadratic surface fitting using ToF depth data. Experimentation shows that the interpolated depth maps derived from the obtained surfaces fit well the original scenes. Moreover, candidate leaves to be approached by a measuring instrument are ranked, and then robot-mounted cameras move closer to them to validate their suitability to being sampled. Some ambiguities arising from leaves overlap or occlusions are cleared up in this way. The work is a proof-of-concept that dense color data combined with sparse depth as provided by a ToF camera yields a good enough 3D approximation for automated plant measuring at the high throughput imposed by the application., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Combining geometric and appearance priors for robust homography estimation

  • Serradell, Eduard
  • Özuysa, Mustafa
  • Lepetit, Vincent
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Abstract. The homography between pairs of images are typically computed from the correspondence of keypoints, which are established by using image descriptors. When these descriptors are not reliable, either because of repetitive patterns or large amounts of clutter, additional priors need to be considered. The Blind PnP algorithm makes use of geometric priors to guide the search for matches while computing camera pose. Inspired by this, we propose a novel approach for homography estimation that combines geometric priors with appearance priors of ambiguous descriptors. More specifically, for each point we retain its best candidates according to appearance. We then prune the set of potential matches by iteratively shrinking the regions of the image that are consistent with the geometric prior. We can then successfully compute homographies between pairs of images containing highly repetitive patterns and even under oblique viewing conditions., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Deformation and illumination invariant feature point descriptor

  • Moreno-Noguer, Francesc
Recent advances in 3D shape recognition have shown that kernels based on diffusion geometry can be effectively used to describe local features of deforming surfaces. In this paper, we introduce a new framework that allows using these kernels on 2D local patches, yielding a novel feature point descriptor that is both invariant to non-rigid image deformations and illumination changes. In order to build the descriptor, 2D image patches are embedded as 3D surfaces, by multiplying the intensity level by an arbitrarily large and constant weight that favors anisotropic diffusion and retains the gradient magnitude information. Patches are then described in terms of a heat kernel signature, which is made invariant to intensity changes, rotation and scaling. The resulting feature point descriptor is proven to be significantly more discriminative than state of the art ones, even those which are specifically designed for describing non-rigid image deformations., Postprint (author’s final draft)