Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 75
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous pose, correspondence and non-rigid shape

  • Sánchez-Riera, Jordi
  • Ostlund, Jonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Trabajo presentado al CVPR 2010 celebrada en San Francisco (EE.UU.) del 13 al 18 de junio., Recent works have shown that 3D shape of non-rigid surfaces can be accurately retrieved from a single image given a set of 3D-to-2D correspondences between that image and another one for which the shape is known. However, existing approaches assume that such correspondences can be readily established, which is not necessarily true when large deformations produce significant appearance changes between the input and the reference images. Furthermore, it is either assumed that the pose of the camera is known, or the estimated solution is pose-ambiguous. In this paper we relax all these assumptions and, given a set of 3D and 2D unmatched points, we present an approach to simultaneously solve their correspondences, compute the camera pose and retrieve the shape of the surface in the input image. This is achieved by introducing weak priors on the pose and shape that we model as Gaussian Mixtures. By combining them into a Kalman filter we can progressively reduce the number of 2D candidates that can be potentially matched to each 3D point, while pose and shape are refined. This lets us to perform a complete and efficient exploration of the solution space and retain the best solution., This work was supported by projects: 'CONSOLIDER-INGENIO 2010 Multimodal interaction in pattern recognition and computer vision' (V-00069), 'Percepción y acción ante incertidumbre' (4803), 'Priors para la detección 3D en tiempo real de objetos rígidos y deformables' (4805), 'Gardening with a cognitive system' (4920). This work has been partially funded by the Spanish Ministry of Science and Innovation under projects 200850I055, DPI2008-06022, and Consolider Ingenio 2010 CSD2007-00018, by EU GARNICS project FP7-247947, and by the Swiss National Science Foundation., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Shared random Ferns for efficient detection of multiple categories

  • Villamizar, Michael
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan
  • Sanfeliu, Alberto
Trabajo presentado al 20th ICPR celebrado en Estambul (Turquía) del 23 al 26 de agosto de 2010., We propose a new algorithm for detecting multiple object categories that exploits the fact that different categories may share common features but with different geometric distributions. This yields an efficient detector which, in contrast to existing approaches, considerably reduces the computation cost at runtime, where the feature computation step is traditionally the most expensive. More specifically, at the learning stage we compute common features by applying the same Random Ferns over the Histograms of Oriented Gradients on the training images. We then apply a boosting step to build discriminative weak classifiers, and learn the specific geometric distribution of the Random Ferns for each class. At runtime, only a few Random Ferns have to be densely computed over each input image, and their geometric distribution allows performing the detection. The proposed method has been validated in public datasets achieving competitive detection results, which are comparable with state-of-the-art methods that use specific features per class., This work was supported by projects: 'CONSOLIDER-INGENIO 2010 Multimodal interaction in pattern recognition and computer vision' (V-00069), 'Robotica ubicua para entornos urbanos' (J-01225), 'Percepción y acción ante incertidumbre' (4803), 'Gardening with a cognitive system' (4920). This work has been partially funded by the Spanish Ministry of Science and Innovation under projects UbROB DPI2007-61452, PAU DPI2008-06022, and MIPRCV Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947. The first au- thor is funded by the Technical University of Catalonia (UPC)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Learning force-based robot skills from haptic demonstration

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Trabajo presentado a la International Conference of the Catalan Association for Artificial Intelligence (CCIA) celebrada en Espluga de Francolí (Tarragona) del 20 al 22 de octubre de 2010., Locally weighted as well as Gaussian mixtures learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration. Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. For the first time, force/torque feedback through a haptic device has been used for teaching a teleoperated robot to empty a rigid container. The memory-based LWPLS and the non-memory-based LWPR algorithms, as well as both the batch and the incremental versions of GMM/GMR were implemented, their comparison leading to very similar results, with the same pattern as regards to both the involved robot joints and the different initial experimental conditions. Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages, where the taught motion will be refined by autonomous robot rehearsal through reinforcement learning., This work has been partially supported by the European projects “PACO-PLUS” (IST-4-27657) and “GARNICS: Gardening with a Cognitive System” ( FP7-247947), the Spanish project “Multimodal Interaction in Pattern Recognition and Computer Vision (MIPRCV)” (Consolider Ingenio 2010 project CSD2007-00018) and the consolidated Robotics group of the Generalitat de Catalunya. L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Combining Geometric and Appearance Priors for Robust Homography Estimation

  • Serradell, Eduard
  • Özuysal, Mustafa
  • Lepetit, Vincent
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Trabajo presentado al 11th European Conference on Computer Vision celebrado en Creta del 5 al 11 de septiembre de 2010., The homography between pairs of images are typically computed from the correspondence of keypoints, which are established by using image descriptors. When these descriptors are not reliable, either because of repetitive patterns or large amounts of clutter, additional priors need to be considered. The Blind PnP algorithm makes use of geometric priors to guide the search for matches while computing camera pose. Inspired by this, we propose a novel approach for homography estimation that combines geometric priors with appearance priors of ambiguous descriptors. More specifically, for each point we retain its best candidates according to appearance. We then prune the set of potential matches by iteratively shrinking the regions of the image that are consistent with the geometric prior. We can then successfully compute homographies between pairs of images containing highly repetitive patterns and even under oblique viewing conditions., This work has been partially funded by the Spanish Ministry of Science and Innovation under CICYT projects PAU DPI2008-06022 and UbROB DPI2007-61452; by MIPRCV Consolider Ingenio 2010 CSD2007-00018; by the EU project GARNICS FP7-247947 and by the Swiss National Science Foundation., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Sharpening haptic inputs for teaching a manipulation skill to a robot

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
8 páginas.-- Comunicación presentada al 1st International Conference on Applied Bionics and Biomechanics celebrado en Venecia (Italia) en Octubre de 2010., Gaussian mixtures-based learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration (PbD). Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. In this work we have used force/torque feedback through a haptic device for teaching a teleoperated robot to empty a rigid container. Structure vibrations and container inertia appeared to considerably disrupt the sensing process, so a filtering algorithm had to be devised. Moreover, some input variables seemed much more relevant to the particular task to be learned than others, which lead us to analyze the training data in order to select those relevant features through principal component analysis and a mutual information criterion. Then, a batch version of GMM/GMR [1], [2] was implemented using different training datasets (original, pre-processed data through PCA and MI). Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages., This work has been partially supported by the European projects PACO-PLUS (IST-4-27657) and GARNICS (FP7-247947), the Spanish project Multimodal Interaction in Pattern Recognition and Computer Vision (MIPRCV) (Consolider Ingenio 2010 project CSD2007-00018) and the Robotics group of the Generalitat de Catalunya. L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration in the force domain

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
6 páginas, 6 figuras.-- Presentado al Workshop on Agents Learning Interactively from Human Teachers(ALIHT) celebrado en Barcelona del 16 al 22 de julio de 2011., Researchers are becoming aware of the importance of other information sources besides visual data in robot learning by demonstration (LbD). Force-based perceptions are shown to convey very relevant information – missed by visual and position sensors – for learning specific tasks. In this paper, we review some recent works using forces as input data in LbD and Human-Robot interaction (HRI) scenarios, and propose a complete learning framework for teaching force-based manipulation skills to a robot through a haptic device. We suggest to use haptic interfaces not only as a demonstration tool but also as a communication channel between the human and the robot, getting the teacher more involved in the teaching process by experiencing the force signals sensed by the robot. Within the proposed framework, we provide solutions for treating force signals, extracting relevant information about the task, encoding the training data and generalizing to perform successfully under unknown conditions., This research is partially sponsored by the European projects GARNICS (FP7-247947) and IntellAct (FP7-269959), the Spanish projects MIPRCV (Consolider Ingenio 2010 project CSD2007-00018) and PAU (DPI2008-06022). L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration of force-based tasks with multiple solution trajectories

  • Rozo, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Presentado al ICAR 2011 celebrado en Tallin (Estonia) del 20 al 23 de junio de 2011., A learning framework with a bidirectional communication channel is proposed, where a human performs several demonstrations of a task using a haptic device (providing him/her with force-torque feedback) while a robot captures these executions using only its force-based perceptive system. Our work departs from the usual approaches to learning by demonstration in that the robot has to execute the task blindly, relying only on force-torque perceptions, and, more essential, we address goal-driven manipulation tasks with multiple solution trajectories, whereas most works tackle tasks that can be learned by just finding a generalization at the trajectory level. To cope with these multiple-solution tasks, in our framework demonstrations are represented by means of a Hidden Markov Model (HMM) and the robot reproduction of the task is performed using a modified version of Gaussian Mixture Regression that incorporates temporal information (GMRa) through the forward variable of the HMM. Also, we exploit the haptic device as a teaching and communication tool in a human-robot interaction context, as an alternative to kinesthetic-based teaching systems. Results show that the robot is able to learn a container-emptying task relying only on force-based perceptions and to achieve the goal from several non-trained initial conditions., This research is partially sponsored by the European projects GARNICS (FP7-247947) and IntellAct (FP7-269959), the Spanish projects Multimodal Interaction in Pattern Recognition and Computer Vision (MIPRCV) (Consolider Ingenio 2010 project CSD2007-00018) and PAU (DPI2008-06022). L. Rozo was supported by the CSIC under a JAE-PREDOC scholarship, Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Depth-supported real-time video segmentation with the Kinect

  • Alexey, Abramov
  • Pauwels, Karl
  • Papon, Jeremie
  • Wörgötter, Florentin
  • Dellen, Babette
Trabajo presentado al WACV celebrado en Breckenridge (USA) del 9 al 11 de enero de 2012., We present a real-time technique for the spatiotemporal segmentation of color/depth movies. Images are segmented using a parallel Metropolis algorithm implemented on a GPU utilizing both color and depth information, acquired with the Microsoft Kinect. Segments represent the equilibrium states of a Potts model, where tracking of segments is achieved by warping obtained segment labels to the next frame using real-time optical flow, which reduces the number of iterations required for the Metropolis method to encounter the new equilibrium state. By including depth information into the framework, true objects boundaries can be found more easily, improving also the temporal coherency of the method. The algorithm has been tested for videos of medium resolutions showing human manipulations of objects. The framework provides an inexpensive visual front end for visual preprocessing of videos in industrial settings and robot labs which can potentially be used in various applications., This research has received funding by the EU GARNICS project FP7-247947 and the EU IntellAct project FP7-269959. B.Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship. K. Pauwels acknowledges support from CEI BioTIC GENIL (CEB09-0010) of the MICINN CEI program., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Depth-supported real-time video segmentation with the Kinect

  • Alexey, Abramov
  • Pauwels, Karl
  • Papon, Jeremie
  • Wörgötter, Florentin
  • Dellen, Babette
This research has received funding by the EU GARNICS project FP7-247947 and the EU IntellAct project FP7-269959. B.Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship. K. Pauwels acknowledges support from CEI BioTIC GENIL (CEB09-0010) of the MICINN CEI program., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robust elastic 2D/3D geometric graph matching

  • Serradell, Eduard
  • Kybic, Jan
  • Moreno-Noguer, Francesc
  • Fua, Pascal
Trabajo presentado a la Medical Imaging celebrada en San Diego el 4 de febrero de 2012., We present an algorithm for geometric matching of graphs embedded in 2D or 3D space. It is applicable for registering any graph-like structures appearing in biomedical images, such as blood vessels, pulmonary bronchi, nerve fibers, or dendritic arbors. Our approach does not rely on the similarity of local appearance features, so it is suitable for multimodal registration with a large difference in appearance. Unlike earlier methods, the algorithm uses edge shape, does not require an initial pose estimate, can handle partial matches, and can cope with nonlinear deformations and topological differences. The matching consists of two steps. First, we find an affine transform that roughly aligns the graphs by exploring the set of all consistent correspondences between the nodes. This can be done at an acceptably low computational expense by using parameter uncertainties for pruning, backtracking as needed. Parameter uncertainties are updated in a Kalman-like scheme with each match. In the second step we allow for a nonlinear part of the deformation, modeled as a Gaussian Process. Short sequences of edges are grouped into superedges, which are then matched between graphs. This allows for topological differences. A maximum consistent set of superedge matches is found using a dedicated branch-and-bound solver, which is over 100 times faster than a standard linear programming approach. Geometrical and topological consistency of candidate matches is determined in a fast hierarchical manner. We demonstrate the effectiveness of our technique at registering angiography and retinal fundus images, as well as neural image stacks., Jan Kybic was supported by the Czech Science Foundation project P202/11/0111. This project was also funded in part by the ERC project MicroNano, the Spanish Ministry of Science and Innovation CICYT project PAU+ DPI2011-27510, the MIPRCV Consolider Ingenio 2010 project CSD2007-00018 and by the EU project GARNICS FP7-247947., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robust non-rigid registration of 2D and 3D graphs

  • Serradell, Eduard
  • Glowacki, Przemyslaw
  • Kybic, Jan
  • Moreno-Noguer, Francesc
  • Fua, Pascal
Trabajo presentado al CVPR celebrado en Providence (USA) del 16 al 21 de junio de 2012., We present a new approach to matching graphs embedded in R2 or R3. Unlike earlier methods, our approach does not rely on the similarity of local appearance features, does not require an initial alignment, can handle partial matches, and can cope with non-linear deformations and topological differences. To handle arbitrary non-linear deformations, we represent them as Gaussian Processes. In the absence of appearance information, we iteratively establish correspondences between graph nodes, update the structure accordingly, and use the current mapping estimate to find the most likely correspondences that will be used in the next iteration. This makes the computation tractable. We demonstrate the effectiveness of our approach first on synthetic cases and then on angiography data, retinal fundus images, and microscopy image stacks acquired at very different resolutions., This work has been partially funded by the ERC projects MicroNano and GARNICS FP7-247947, the Spanish Ministry of Economy and Competitiveness under projects PAU+DPI2011-27510 and MIPRCV Consolider Ingenio 2010 CSD2007-00018, and the Czech Science Foundation under project P202/11/0111. We thank the project ‘La Marató de TV3 082131 for providing the heart dataset., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Channel coding for joint colour and depth segmentation

  • Wallenberg, Marcus
  • Felsberg, Michael
  • Forssén, Per-Erik
  • Dellen, Babette
Trabajo presentado al 33rd DAGM Symposium celebrado en Frankfurt del 31 de agosto al 2 de septiembre de 2011., Segmentation is an important preprocessing step in many applications. Compared to colour segmentation, fusion of colour and depth greatly improves the segmentation result. Such a fusion is easy to do by stacking measurements in different value dimensions, but there are better ways. In this paper we perform fusion using the channel representation, and demonstrate how a state-of-the-art segmentation algorithm can be modified to use channel values as inputs. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using the superparamagnetic clustering algorithm. Our experiments show that depth gradients are more useful than depth values for segmentation, and that channel coding both colour and depth gradients makes tuned parameter settings generalise better to novel images., This work was supported by the EC's 7th Framework Programme (FP7/2007-2013), grant agreement 247947 (GARNICS), the Swedish Research Council through a grant for the project Embodied Visual Object Recognition, and by Link oping University. B. Dellen also acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Deformation and illumination invariant feature point descriptor

  • Moreno-Noguer, Francesc
Trabajo presentado al CVPR celebrado en Providence (USA) del 20 al 25 de junio de 2011., Recent advances in 3D shape recognition have shown that kernels based on diffusion geometry can be effectively used to describe local features of deforming surfaces. In this paper, we introduce a new framework that allows using these kernels on 2D local patches, yielding a novel feature point descriptor that is both invariant to non-rigid image deformations and illumination changes. In order to build the descriptor, 2D image patches are embedded as 3D surfaces, by multiplying the intensity level by an arbitrarily large and constant weight that favors anisotropic diffusion and retains the gradient magnitude information. Patches are then described in terms of a heat kernel signature, which is made invariant to intensity changes, rotation and scaling. The resulting feature point descriptor is proven to be significantly more discriminative than state of the art ones, even those which are specifically designed for describing non-rigid image deformations., This work has been partially funded by the Spanish Ministry of Science and Innovation under projects DPI2008-06022 and Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Design of non-anthropomorphic robotic hands for anthropomorphic tasks

  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
  • Perez-Gracia, Alba
Trabajo presentado al 35th Mechanisms and Robotics Conference celebrado en Washington del 28 al 31 de agosto de 2011., In this paper, we explore the idea of designing non-anthropomorphic, multi-fingered robotic hands for tasks that replicate the motion of the human hand. Taking as input data rigid-body trajectories for the five fingertips, we develop a method to perform dimensional synthesis for a kinematic chain with a tree structure, with three common joints and five branches. We state the forward kinematics equations of relative displacements for each serial chain expressed as dual quaternions, and solve for up to five chains simultaneously to reach a number of poses along the hand trajectory using a hybrid global numerical solver that integrates a genetic algorithm and a Levenberg-Marquardt local optimizer. Although the number of candidate solutions in this problem is very high, the use of the genetic algorithm lets us to perform an exhaustive exploration of the solution space and retain a subset of them. We then can choose some of the solutions based on the specific task to perform. Note that these designs could match the task exactly while having a finger design radically different from that of the human hand., This work is partially supported by the Spanish Ministry of Science and Innovation under projects DPI2010-18449, Consolider Ingenio 2010 CSD2007-00018, and a Ramon y Cajal Research Fellowship. Also by the EU project GARNICS FP7-247947., Peer reviewed, Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Integrating task planning and interactive learning for robots to work in human environments

  • Agostini, Alejandro
  • Torras, Carme
  • Wörgötter, Florentin
Trabajo presentado al IJCAI celebrado en Barcelona del 16 al 22 de julio de 2011., Human environments are challenging for robots, which need to be trainable by lay people and learn new behaviours rapidly without disrupting much the ongoing activity. A system that integrates AI techniques for planning and learning is here proposed to satisfy these strong demands. The approach rapidly learns planning operators from few action experiences using a competitive strategy where many alternatives of cause-effect explanations are evaluated in parallel, and the most successful ones are used to generate the operators. The success of a cause-effect explanation is evaluated by a probabilistic estimate that compensates the lack of experience, producing more confident estimations and speeding up the learning in relation to other known estimates. The system operates without task interruption by integrating in the planning-learning loop a human teacher that supports the planner in making decisions. All the mechanisms are integrated and synchronized in the robot using a general decision-making framework. The feasibility and scalability of the architecture are evaluated in two different robot platforms: a Stäubli arm, and the humanoid ARMAR III., This research is partially funded by the EU GARNICS project FP7-247947 and the Generalitat de Catalunya through the Robotics group., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Probabilistic simultaneous pose and non-rigid shape recovery

  • Moreno-Noguer, Francesc
  • Porta, Josep M.
Trabajo presentado al CVPR 2011 celebrado en Providence (USA) del 20 al 25 de junio., We present an algorithm to simultaneously recover non- rigid shape and camera poses from point correspondences between a reference shape and a sequence of input images. The key novel contribution of our approach is in bringing the tools of the probabilistic SLAM methodology from a rigid to a deformable domain. Under the assumption that the shape may be represented as a weighted sum of deformation modes, we show that the problem of estimating the modal weights along with the camera poses, may be probabilistically formulated as a maximum a posterior estimate and solved using an iterative least squares optimization. An extensive evaluation on synthetic and real data, shows that our approach has several significant advantages over current approaches, such as performing robustly under large amounts of noise and outliers, and neither requiring to track points over the whole sequence nor initializations close from the ground truth solution., This work has been partially funded by the Spanish Ministry of Science and Innovation under projects DPI2008-06022 and Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Segmenting color images into surface patches by exploiting sparse depth data

  • Dellen, Babette
  • Alenyà, Guillem
  • Foix, Sergi
  • Torras, Carme
Trabajo presentado al WACV 2011 celebrado en Kona (USA) del 5 al 7 de enero., We present a new method for segmenting color images into their composite surfaces by combining color segmentation with model-based fitting utilizing sparse depth data, acquired using time-of-flight (Swissranger, PMD CamCube) and stereo techniques. The main target of our work is the segmentation of plant structures, i.e., leaves, from color-depth images, and the extraction of color and 3D shape information for automating manipulation tasks. Since segmentation is performed in the dense color space, even sparse, incomplete, or noisy depth information can be used. This kind of data often represents a major challenge for methods operating in the 3D data space directly. To achieve our goal, we construct a three-stage segmentation hierarchy by segmenting the color image with different resolutions - assuming that ``true'' surface boundaries must appear at some point along the segmentation hierarchy. 3D surfaces are then fitted to the color-segment areas using depth data. Those segments which minimize the fitting error are selected and used to construct a new segmentation. Then, an additional region merging and a growing stage are applied to avoid over-segmentation and label previously unclustered points. Experimental results demonstrate that the method is successful in segmenting a variety of domestic objects and plants into quadratic surfaces. At the end of the procedure, the sparse depth data is completed using the extracted surface models, resulting in dense depth maps. For stereo, the resulting disparity maps are compared with ground truth and the average error is computed., This research is partially funded by the EU GARNICS project FP7-247947, the Consolider-Ingenio project CSD2007-00018, and the Catalan Research Commission under 2009SGR155. G. Alenyà and S. Foix were supported by CSIC under a Jae-Doc and Jae-Pre-Doc fellowship, respectively. B. Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous correspondence and non-rigid 3D reconstruction of the coronary tree from single X-ray images

  • Serradell, Eduard
  • Romero, Adriana
  • Leta, Rubén
  • Gatta, Carlo
  • Moreno-Noguer, Francesc
Trabajo presentado al ICCV 2011 celebrado en Barcelona del 6 al 3 de noviembre., We present a novel approach to simultaneously reconstruct the 3D structure of a non-rigid coronary tree and estimate point correspondences between an input X-ray image and a reference 3D shape. At the core of our approach lies an optimization scheme that iteratively fits a generative 3D model of increasing complexity and guides the matching process. As a result, and in contrast to existing approaches that assume rigidity or quasi-rigidity of the structure, our method is able to retrieve large non-linear deformations even when the input data is corrupted by the presence of noise and partial occlusions. We extensively evaluate our approach under synthetic and real data and demonstrate a remarkable improvement compared to state-of-the-art., This work has been partially funded by the project La Marató de TV3 082131; by the Spanish Ministry of Science and Innovation under projects DPI2008-06022, DPI2010- 17112, Consolider Ingenio 2010 CSD2007-00018; and by the EU project GARNICS FP7-247947. The work of C. Gatta is supported by a Beatriu de Pinos Fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Towards plant monitoring through next best view

  • Foix, Sergi
  • Alenyà, Guillem
  • Torras, Carme
Trabajo presentado a la 14th International Conference of the Catalan Association for Artificial Intelligence (CCIA) celebrada en Lleida del 26 al 28 de octubre de 2011., Monitoring plants using leaf feature detection is a challenging perception task because different leaves, even from the same plant, may have very different shapes, sizes and deformations. In addition, leaves may be occluded by other leaves making it hard to determine some of their characteristics. In this paper we use a Time-of-Flight (ToF) camera mounted on a robot arm to acquire the depth information needed for plant leaf detection. Under a Next Best View (NBV) paradigm, we propose a criterion to compute a new camera position that offers a better view of a target leaf. The proposed criterion exploits some typical errors of the ToF camera, which are common to other 3D sensing devices as well. This approach is also useful when more than one leaf is segmented as the same region, since moving the camera following the same NBV criterion helps to disambiguate this situation., This research is partially funded by the EU GARNICS project FP7-247947, by the Span- ish Ministry of Science and Innovation under projects DPI2008-06022 and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission through the Robotics Group. S. Foix is supported by a PhD fellowship from CSIC’s JAE program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

3D modelling of leaves from color and ToF data for robotized plant measuring

  • Alenyà, Guillem
  • Dellen, Babette
  • Torras, Carme
Trabajo presentado al ICRA 2011 celebrado en Shanghai del 9 al 13 de mayo de 2011., Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modelling for this application has strong demands, particularly in what concerns 3D information gathering and speed. This paper shows that Time-of-Flight (ToF) cameras achieve a good compromise between both demands, providing a suitable complement to color vision. A new method is proposed to segment plant images into their composite surface patches by combining hierarchical color segmentation with quadratic surface fitting using ToF depth data. Experimentation shows that the interpolated depth maps derived from the obtained surfaces fit well the original scenes. Moreover, candidate leaves to be approached by a measuring instrument are ranked, and then robot-mounted cameras move closer to them to validate their suitability to being sampled. Some ambiguities arising from leaves overlap or occlusions are cleared up in this way. The work is a proof-of-concept that dense color data combined with sparse depth as provided by a ToF camera yields a good enough 3D approximation for automated plant measuring at the high throughput imposed by the application., This research is partially funded by the EU GARNICS project FP7-247947, by the Spanish Ministry of Science and Innovation under projects DPI2008-06022 and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission. G. Alenyà was supported by the CSIC under a Jae-Doc Fellowship. B. Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations

  • Dellen, Babette
  • Husain, Farzad
  • Torras, Carme
Presentado a la 8th International Conference on Computer Vision Theory and Applications celebrada en Barcelona del 21 al 24 de febrero de 2013., A novel framework for joint segmentation and tracking in depth videos of object surfaces is presented. Initially, the 3D colored point cloud obtained using the Kinect camera is used to segment the scene into surface patches, defined by quadratic functions. The computed segments together with their functional descriptions are then used to partition the depth image of the subsequent frame in a consistent manner with respect to the precedent frame. This way, solutions established in previous frames can be reused which improves the efficiency of the algorithm and the coherency of the segmentations along the movie. The algorithm is tested for scenes showing human and robot manipulations of objects. We demonstrate that the method can successfully segment and track the human/robot arm and object surfaces along the manipulations. The performance is evaluated quantitatively by measuring the temporal coherency of the segmentations and the segmentation covering using ground truth. The method provides a visual front-end designed for robotic applications, and can potentially be used in the context of manipulation recognition, visual servoing, and robot-grasping tasks., This research is partially funded by the EU projects GARNICS (FP7-247947) and IntellAct (FP7-269959), and the Grup consolidat 2009 SGR155. B. Dellen acknowledges support from the Spanish Ministry of Science and Innovation through a Ramon y Cajal program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Characterization of textile grasping experiments

  • Alenyà, Guillem
  • Ramisa, Arnau
  • Moreno-Noguer, Francesc
  • Torras, Carme
Presentado al International Conference on Robotics and Automation celebrado en USA del 14 al 18 de mayo de 2012., Grasping highly deformable objects, like textiles, is an emerging area of research that involves both perception and manipulation abilities. As new techniques appear, it becomes essential to design strategies to compare them. However, this is not an easy task, since the large state-space of textile objects explodes when coupled with the variability of grippers, robotic hands and robot arms performing the manipulation task. This high variability makes it very difficult to design experiments to evaluate the performance of a system in a repeatable way and compare it to others. We propose a framework to allow the comparison of different grasping methods for textile objects. Instead of measuring each component separately, we therefore propose a methodology to explicitly measure the vision-manipulation correlation by taking into account the throughput of the actions. Perceptions of deformable objects should be grouped into different clusters, and the different grasping actions available should be tested for each perception type to obtain the action-perception success ratio. This characterization potentially allows to compare very different systems in terms of specialized actions, perceptions or widely useful actions, along with the cost of performing each action. We will also show that this categorization is useful in manipulation planning of deformable objects., This work was supported by the Spanish Ministry of Science and Innovation under projects PAU+ DPI2011-27510, by the EU project INTELLACT 247947 FP7-269959 and by the Catalan Research Commission through SGR-00155. A. Ramisa worked under the JAE-DOC grant from the CSIC and the FSE., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Plant leaf imaging using time of flight camera under sunlight, shadow and room conditions

  • Kazmi, Wajahat
  • Foix, Sergi
  • Alenyà, Guillem
Presentado al ROSE 2012 celebrado en Magdeburgo (Alemania) del 16 al 18 de noviembre., In this article, we analyze the effects of ambient light on Time of Flight (ToF) depth imaging for a plant's leaf in sunlight, shadow and room conditions. ToF imaging is sensitive to ambient light and we try to find the best possible integration times (IT) for each condition. This is important in order to optimize the camera calibration. Our analysis is based on several statistical metrics estimated from the ToF data. We explain the estimation of the metrics and propose a method of predicting the deteriorating behavior of the data in each condition using camera flags. Finally, we also propose a method to improve the quality of a ToF image taken in a mixed condition having different ambient light exposures., This work is supported by the Danish Council for Strategic Research under project ASETA (www.aseta.dk) grant no. 09-067027, the Spanish Ministry of Science and Innovation under projects PAU + DPI2011-27510, the EU project GARNICS (FP7-247947) and the Catalan Research Commission (SGR-00155). W. Kazmi is funded by ASETA and S. Foix is supported by PhD fellowship from CSIC’s JAE program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robotic leaf probing via segmentation of range data into surface patches

  • Alenyà, Guillem
  • Dellen, Babette
  • Foix, Sergi
  • Torras, Carme
Presentado al International Conference on Intelligent Robots and Systems (IROS AGROBOTICS), Workshop on Agricultural Robotics: Enabling Safe, Efficient, Affordable Robots for Food Production celebrado en Portugal del 7 al 12 de octubre de 2012., We present a novel method for the robotized probing of plant leaves using Time-of-Flight (ToF) sensors. Plant images are segmented into surface patches by combining a segmentation of the infrared intensity image, provided by the ToF camera, with quadratic surface fitting using ToF depth data. Leaf models are fitted to the boundaries of the segments and used to determine probing points and to evaluate the suitability of leaves for being sampled. The robustness of the approach is evaluated by repeatedly placing an especially adapted, robot-mounted spad meter on the probing points which are extracted in an automatic manner. The number of successful chlorophyll measurements is counted, and the total time for processing the visual data and probing the plant with the robot is measured for each trial. In case of failure, the underlying causes are determined and reported, allowing a better assessment of the applicability of the method in real scenarios., This research is partially funded by the EU GARNICS project FP7-247947, by the Spanish Ministry of Science and Innovation under projects PAU+ and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission. B. Dellen acknowledges support from the Spanish Ministry for Science and Innovation via a Ramon y Cajal program. S. Foix is supported by PhD fellowship from CSIC’s JAE program., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Active perception of deformable objects using 3D cameras

  • Alenyà, Guillem
  • Moreno-Noguer, Francesc
  • Ramisa, Arnau
  • Torras, Carme
Presentado al Workshop de Robótica Experimental celebrado en Sevilla del 28 al 29 de noviembre de 2011., Perception and manipulation of rigid objects has received a lot of attention, and several solutions have been proposed. In contrast, dealing with deformable objects is a relatively new and challenging task because they are more complex to model, their state is difficult to determine, and self-occlusions are common and hard to estimate. In this paper we present our progress/results in the perception of deformable objects both using conventional RGB cameras and active sensing strategies by means of depth cameras. We provide insights in two different areas of application: grasping of textiles and plant leaf modelling., This research is partially funded by the EU GARNICS project FP7-247947, by the Spanish Ministry of Science and Innovation under project DPI2008-06022 and MIPRCV Consolider Ingenio CSD2007-00018, and the Catalan Research Commission., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Learning algorithms for robot manipulation of clothing and plant leaves

  • Torras, Carme
Trabajo presentado al ICRA Workshop on Advances in Robot Manipulation of Clothes and Flexible Objects, celebrado el 1 de junio de 2014 en Hong Kong., Research supported by the European projects PACO-PLUS, GARNICS and IntellAct, and the Spanish project PAU+., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

Robot manipulation in human environments: Challenges for learning algorithms

  • Torras, Carme
Resumen del trabajo presentado al Dagstuhl Seminar 2014 celebrado en Dagstuhl (Alemania) del 17 al 21 de febrero de 2014., The European projects PACO-PLUS, GARNICS and IntellAct, the Spanish projects PAU and PAU+, and the Catalan grant SGR-155., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Characterization of textile grasping experiments

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Ramisa Ayats, Arnau
  • Moreno-Noguer, Francesc
  • Torras, Carme|||0000-0002-2933-398X
Grasping highly deformable objects, like textiles, is an emerging area of research that involves both percep- tion and manipulation abilities. As new techniques appear, it becomes essential to design strategies to compare them. However, this is not an easy task, since the large state-space of textile objects explodes when coupled with the variability of grippers, robotic hands and robot arms performing the manipulation task. This high variability makes it very difficult to design experiments to evaluate the performance of a system in a repeatable way and compare it to others. We propose a framework to allow the comparison of different grasping methods for textile objects. Instead of measuring each component separately, we there- fore propose a methodology to explicitly measure the vision- manipulation correlation by taking into account the throughput of the actions. Perceptions of deformable objects should be grouped into different clusters, and the different grasping actions available should be tested for each perception type to obtain the action-perception success ratio. This characterization potentially allows to compare very different systems in terms of specialized actions , perceptions or widely useful actions , along with the cost of performing each action. We will also show that this categorization is useful in manipulation planning of deformable objects., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Deformation and illumination invariant feature point descriptor

  • Moreno-Noguer, Francesc
Recent advances in 3D shape recognition have shown that kernels based on diffusion geometry can be effectively used to describe local features of deforming surfaces. In this paper, we introduce a new framework that allows using these kernels on 2D local patches, yielding a novel feature point descriptor that is both invariant to non-rigid image deformations and illumination changes. In order to build the descriptor, 2D image patches are embedded as 3D surfaces, by multiplying the intensity level by an arbitrarily large and constant weight that favors anisotropic diffusion and retains the gradient magnitude information. Patches are then described in terms of a heat kernel signature, which is made invariant to intensity changes, rotation and scaling. The resulting feature point descriptor is proven to be significantly more discriminative than state of the art ones, even those which are specifically designed for describing non-rigid image deformations.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robust elastic 2D/3D geometric graph matching

  • Serradell, Eduard
  • Kybic, J.
  • Moreno-Noguer, Francesc
  • Fua, Pascal
We present an algorithm for geometric matching of graphs embedded in 2D or 3D space. It is applicable for registering any graph-like structures appearing in biomedical images, such as blood vessels, pulmonary bronchi, nerve fibers, or dendritic arbors. Our approach does not rely on the similarity of local appearance features, so it is suitable for multimodal registration with a large difference in appearance. Unlike earlier methods, the algorithm uses edge shape, does not require an initial pose estimate, can handle partial matches, and can cope with nonlinear deformations and topological differences. The matching consists of two steps. First, we find an affine transform that roughly aligns the graphs by exploring the set of all consistent correspondences between the nodes. This can be done at an acceptably low computational expense by using parameter uncertainties for pruning, backtracking as needed. Parameter uncertainties are updated in a Kalman-like scheme with each match. In the second step we allow for a nonlinear part of the deformation, modeled as a Gaussian Process. Short sequences of edges are grouped into superedges, which are then matched between graphs. This allows for topological differences. A maximum consistent set of superedge matches is found using a dedicated branch-and-bound solver, which is over 100 times faster than a standard linear programming approach. Geometrical and topological consistency of candidate matches is determined in a fast hierarchical manner. We demonstrate the effectiveness of our technique at registering angiography and retinal fundus images, as well as neural image stacks., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Combining geometric and appearance priors for robust homography estimation

  • Serradell, Eduard
  • Özuysa, Mustafa
  • Lepetit, Vincent
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Abstract. The homography between pairs of images are typically computed from the correspondence of keypoints, which are established by using image descriptors. When these descriptors are not reliable, either because of repetitive patterns or large amounts of clutter, additional priors need to be considered. The Blind PnP algorithm makes use of geometric priors to guide the search for matches while computing camera pose. Inspired by this, we propose a novel approach for homography estimation that combines geometric priors with appearance priors of ambiguous descriptors. More specifically, for each point we retain its best candidates according to appearance. We then prune the set of potential matches by iteratively shrinking the regions of the image that are consistent with the geometric prior. We can then successfully compute homographies between pairs of images containing highly repetitive patterns and even under oblique viewing conditions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Towards plant monitoring through next best view

  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Torras, Carme|||0000-0002-2933-398X
Monitoring plants using leaf feature detection is a challenging perception task because different leaves, even from the same plant, may have very different shapes, sizes and deformations. In addition, leaves may be occluded by other leaves making it hard to determine some of their characteristics. In this paper we use a Time-of-Flight (ToF) camera mounted on a robot arm to acquire the depth information needed for plant leaf detection. Under a Next Best View (NBV) paradigm, we propose a criterion to compute a new camera position that offers a better view of a target leaf. The proposed criterion exploits some typical errors of the ToF camera, which are common to other 3D sensing devices as well. This approach is also useful when more than one leaf is segmented as the same region, since moving the camera following the same NBV criterion helps to disambiguate this situation.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Active perception of deformable objects using 3D cameras

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Moreno-Noguer, Francesc
  • Ramisa Ayats, Arnau
  • Torras, Carme|||0000-0002-2933-398X
Perception and manipulation of rigid objects has received a lot of attention, and several solutions have been proposed. In contrast, dealing with deformable objects is a relatively new and challenging task because they are more complex to model, their state is difficult to determine, and self-occlusions are common and hard to estimate. In this paper we present our progress/results in the perception of deformable objects both using conventional RGB cameras and active sensing strategies by means of depth cameras.We provide insights in two different areas of application: grasping of textiles and plant leaf modelling.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Depth-supported real-time video segmentation with the Kinect

  • Abramov, Alexey
  • Pauwels, Karl
  • Papon, Jeremie
  • Wörgötter, Florentin
  • Dellen, Babette
We present a real-time technique for the spatiotemporal segmentation of color/depth movies. Images are segmented using a parallel Metropolis algorithm implemented on a GPU utilizing both color and depth information, acquired with the Microsoft Kinect. Segments represent the equilibrium states of a Potts model, where tracking of segments is achieved by warping obtained segment labels to the next frame using real-time optical flow, which reduces the number of iterations required for the Metropolis method to encounter the new equilibrium state. By including depth information into the framework, true objects boundaries can be found more easily, improving also the temporal coherency of the method. The algorithm has been tested for videos of medium resolutions showing human manipulations of objects. The framework provides an inexpensive visual front end for visual preprocessing of videos in industrial settings and robot labs which can potentially be used in various applications., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Probabilistic simultaneous pose and non-rigid shape recovery

  • Moreno-Noguer, Francesc
  • Porta Pleite, Josep Maria|||0000-0002-5056-1717
We present an algorithm to simultaneously recover non-rigid shape and camera poses from point correspondences between a reference shape and a sequence of input images. The key novel contribution of our approach is in bringing the tools of the probabilistic SLAM methodology from a rigid to a deformable domain. Under the assumption that the shape may be represented as a weighted sum of deformation modes, we show that the problem of estimating the modal weights along with the camera poses, may be probabilistically formulated as a maximum a posterior estimate and solved using an iterative least squares optimization. An extensive evaluation on synthetic and real data, shows that our approach has several significant advantages over current approaches, such as performing robustly under large amounts of noise and outliers, and neither requiring to track points over the whole sequence nor initializations close from the ground truth solution.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations

  • Dellen, Babette
  • Husain, Syed Farzad
  • Torras, Carme|||0000-0002-2933-398X
A novel framework for joint segmentation and tracking in depth videos of object surfaces is presented. Initially, the 3D colored point cloud obtained using the Kinect camera is used to segment the scene into surface patches, defined by quadratic functions. The computed segments together with their functional descriptions are then used to partition the depth image of the subsequent frame in a consistent manner with respect to the precedent frame. This way, solutions established in previous frames can be reused which improves the efficiency of the algorithm and the coherency of the segmentations along the movie. The algorithm is tested for scenes showing human and robot manipulations of objects. We demonstrate that the method can successfully segment and track the human/robot arm and object surfaces along the manipulations. The performance is evaluated quantitatively by measuring the temporal coherency of the segmentations and the segmentation covering using ground truth. The method provides a visual front-end designed for robotic applications, and can potentially be used in the context of manipulation recognition, visual servoing, and robot-grasping tasks, Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Integrating task planning and interactive learning for robots to work in human environments

  • Agostini, Alejandro Gabriel
  • Torras, Carme|||0000-0002-2933-398X
  • Wörgötter, Florentin
Human environments are challenging for robots, which need to be trainable by lay people and learn new behaviours rapidly without disrupting much the ongoing activity. A system that integrates AI techniques for planning and learning is here proposed to satisfy these strong demands. The approach rapidly learns planning operators from few action experiences using a competitive strategy where many alternatives of cause-effect explanations are evaluated in parallel, and the most successful ones are used to generate the operators. The success of a cause-effect explanation is evaluated by a probabilistic estimate that compensates the lack of experience, producing more confident estimations and speeding up the learning in relation to other known estimates. The system operates without task interruption by integrating in the planning-learning loop a human teacher that supports the planner in making decisions. All the mechanisms are integrated and synchronized in the robot using a general decision-making framework. The feasibility and scalability of the architecture are evaluated in two different robot platforms: a Stäubli arm, and the humanoid ARMAR III., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Sharpening haptic inputs for teaching a manipulation skill to a robot

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
Gaussian mixtures-based learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration (PbD). Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. In this work we have used force/torque feedback through a haptic device for teaching a teleoperated robot to empty a rigid container. Structure vibrations and container inertia appeared to considerably disrupt the sensing process, so a filtering algorithm had to be devised. Moreover, some input variables seemed much more relevant to the particular task to be learned than others, which lead us to analyze the training data in order to select those relevant features through principal component analysis and a mutual information criterion. Then, a batch version of GMM/GMR [1], [2] was implemented using different training datasets (original, pre-processed data through PCA and MI). Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Channel coding for joint colour and depth segmentation

  • Wallenberg, Markus
  • Felsberg, Michael
  • Forssen, Per-Erik
  • Dellen, Babette
Segmentation is an important preprocessing step in many applications. Compared to colour segmentation, fusion of colour and depth greatly improves the segmentation result. Such a fusion is easy to do by stacking measurements in di erent value dimensions, but there are better ways. In this paper we perform fusion using the channel representation, and demonstrate how a state-of-the-art segmentation algorithm can be modi ed to use channel values as inputs. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using the superparamagnetic clustering algorithm. Our experiments show that depth gradients are more useful than depth values for segmentation, and that channel coding both colour and depth gradients makes tuned parameter settings generalise better to novel images., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Segmenting color images into surface patches by exploiting sparse depth data

  • Dellen, Babette
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Torras, Carme|||0000-0002-2933-398X
We present a new method for segmenting color images into their composite surfaces by combining color segmentation with model-based fitting utilizing sparse depth data, acquired using time-of-flight (Swissranger, PMD CamCube) and stereo techniques. The main target of our work is the segmentation of plant structures, i.e., leaves, from color-depth images, and the extraction of color and 3D shape information for automating manipulation tasks. Since segmentation is performed in the dense color space, even sparse, incomplete, or noisy depth information can be used. This kind of data often represents a major challenge for methods operating in the 3D data space directly. To achieve our goal, we construct a three-stage segmentation hierarchy by segmenting the color image with different resolutions-assuming that “true” surface boundaries must appear at some point along the segmentation hierarchy. 3D surfaces are then fitted to the color-segment areas using depth data. Those segments which minimize the fitting error are selected and used to construct a new segmentation. Then, an additional region merging and a growing stage are applied to avoid over-segmentation and label previously unclustered points. Experimental results demonstrate that the method is successful in segmenting a variety of domestic objects and plants into quadratic surfaces. At the end of the procedure, the sparse depth data is completed using the extracted surface models, resulting in dense depth maps. For stereo, the resulting disparity maps are compared with ground truth and the average error is computed., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous correspondence and non-rigid 3D reconstruction of the coronary tree from single X-ray images

  • Serradell, Eduard
  • Romero, Adriana
  • Leta, Ruben
  • Gatta, Carlo
  • Moreno-Noguer, Francesc
We present a novel approach to simultaneously reconstruct the 3D structure of a non-rigid coronary tree and estimate point correspondences between an input X-ray image and a reference 3D shape. At the core of our approach lies an optimization scheme that iteratively fits a generative 3D model of increasing complexity and guides the matching process. As a result, and in contrast to existing approaches that assume rigidity or quasi-rigidity of the structure, our method is able to retrieve large non-linear deformations even when the input data is corrupted by the presence of noise and partial occlusions. We extensively evaluate our approach under synthetic and real data and demonstrate a remarkable improvement compared to state-of-the-art., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration of force-based tasks with multiple solution trajectories

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
A learning framework with a bidirectional communication channel is proposed, where a human performs several demonstrations of a task using a haptic device (providing him/her with force-torque feedback) while a robot captures these executions using only its force-based perceptive system. Our work departs from the usual approaches to learning by demonstration in that the robot has to execute the task blindly, relying only on force-torque perceptions, and, more essential, we address goal-driven manipulation tasks with multiple solution trajectories, whereas most works tackle tasks that can be learned by just finding a generalization at the trajectory level. To cope with these multiple-solution tasks, in our framework demonstrations are represented by means of a Hidden Markov Model (HMM) and the robot reproduction of the task is performed using a modified version of Gaussian Mixture Regression that incorporates temporal information (GMRa) through the forward variable of the HMM. Also, we exploit the haptic device as a teaching and communication tool in a human-robot interaction context, as an alternative to kinesthetic-based teaching systems. Results show that the robot is able to learn a container-emptying task relying only on force-based perceptions and to achieve the goal from several non-trained initial conditions.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Simultaneous pose, correspondence and non-rigid shape

  • Sánchez-Riera, Jordi
  • Ostlund, Jonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Recent works have shown that 3D shape of non-rigid surfaces can be accurately retrieved from a single image given a set of 3D-to-2D correspondences between that image and another one for which the shape is known. However, existing approaches assume that such correspondences can be readily established, which is not necessarily true when large deformations produce significant appearance changes between the input and the reference images. Furthermore, it is either assumed that the pose of the camera is known, or the estimated solution is pose-ambiguous. In this paper we relax all these assumptions and, given a set of 3D and 2D unmatched points, we present an approach to simultaneously solve their correspondences, compute the camera pose and retrieve the shape of the surface in the input image. This is achieved by introducing weak priors on the pose and shape that we model as Gaussian Mixtures. By combining them into a Kalman filter we can progressively reduce the number of 2D candidates that can be potentially matched to each 3D point, while pose and shape are refined. This lets us to perform a complete and efficient exploration of the solution space and retain the best solution.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Shared random Ferns for efficient detection of multiple categories

  • Villamizar Vergel, Michael Alejandro
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan|||0000-0002-6354-8941
  • Sanfeliu Cortés, Alberto|||0000-0003-3868-9678
We propose a new algorithm for detecting multiple object categories that exploits the fact that different categories may share common features but with different geometric distributions. This yields an efficient detector which, in contrast to existing approaches, considerably reduces the computation cost at runtime, where the feature computation step is traditionally the most expensive. More specifically, at the learning stage we compute common features by applying the same Random Ferns over the Histograms of Oriented Gradients on the training images. We then apply a boosting step to build discriminative weak classifiers, and learn the specific geometric distribution of the Random Ferns for each class. At runtime, only a few Random Ferns have to be densely computed over each input image, and their geometric distribution allows performing the detection. The proposed method has been validated in public datasets achieving competitive detection results, which are comparable with state-of-the-art methods that use specific features per class., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robotic leaf probing via segmentation of range data into surface patches

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Dellen, Babette
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Torras, Carme|||0000-0002-2933-398X
We present a novel method for the robotized probing of plant leaves using Time-of-Flight (ToF) sensors. Plant images are segmented into surface patches by combining a segmentation of the infrared intensity image, provided by the ToF camera, with quadratic surface fitting using ToF depth data. Leaf models are fitted to the boundaries of the segments and used to determine probing points and to evaluate the suitability of leaves for being sampled. The robustness of the approach is evaluated by repeatedly placing an especially adapted, robot-mounted spad meter on the probing points which are extracted in an automatic manner. The number of successful chlorophyll measurements is counted, and the total time for processing the visual data and probing the plant with the robot is measured for each trial. In case of failure, the underlying causes are determined and reported, allowing a better assessment of the applicability of the method in real scenarios., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robust non-rigid registration of 2D and 3D graphs

  • Serradell, Eduard
  • Glowacki, Przemyslaw
  • Jan, Kybic
  • Moreno-Noguer, Francesc
  • Fua, Pascal
We present a new approach to matching graphs embedded in R2 or R3. Unlike earlier methods, our approach does not rely on the similarity of local appearance features, does not require an initial alignment, can handle partial matches, and can cope with non-linear deformations and topological differences. To handle arbitrary non-linear deformations, we represent them as Gaussian Processes. In the absence of appearance information, we iteratively establish correspondences between graph nodes, update the structure accordingly, and use the current mapping estimate to find the most likely correspondences that will be used in the next iteration. This makes the computation tractable. We demonstrate the effectiveness of our approach first on synthetic cases and then on angiography data, retinal fundus images, and microscopy image stacks acquired at very different resolutions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

3D modelling of leaves from color and ToF data for robotized plant measuring

  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
  • Dellen, Babette
  • Torras, Carme|||0000-0002-2933-398X
Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modelling for this application has strong demands, particularly in what concerns 3D information gathering and speed. This paper shows that Time-of-Flight (ToF) cameras achieve a good compromise between both demands, providing a suitable complement to color vision. A new method is proposed to segment plant images into their composite surface patches by combining hierarchical color segmentation with quadratic surface fitting using ToF depth data. Experimentation shows that the interpolated depth maps derived from the obtained surfaces fit well the original scenes. Moreover, candidate leaves to be approached by a measuring instrument are ranked, and then robot-mounted cameras move closer to them to validate their suitability to being sampled. Some ambiguities arising from leaves overlap or occlusions are cleared up in this way. The work is a proof-of-concept that dense color data combined with sparse depth as provided by a ToF camera yields a good enough 3D approximation for automated plant measuring at the high throughput imposed by the application., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration in the force domain

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
Researchers are becoming aware of the importance of other information sources besides visual data in robot learning by demonstration (LbD). Forcebased perceptions are shown to convey very relevant information – missed by visual and position sensors – for learning specific tasks. In this paper, we review some recent works using forces as input data in LbD and Human-Robot interaction (HRI) scenarios, and propose a complete learning framework for teaching force-based manipulation skills to a robot through a haptic device. We suggest to use haptic interfaces not only as a demonstration tool but also as a communication channel between the human and the robot, getting the teacher more involved in the teaching process by experiencing the force signals sensed by the robot. Within the proposed framework, we provide solutions for treating force signals, extracting relevant information about the task, encoding the training data and generalizing to perform successfully under unknown conditions., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Learning force-based robot skills from haptic demonstration

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo|||0000-0003-3627-4938
  • Torras, Carme|||0000-0002-2933-398X
Locally weighted as well as Gaussian mixtures learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration. Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. For the first time, force/torque feedback through a haptic device has been used for teaching a teleoperated robot to empty a rigid container. The memory-based LWPLS and the non-memory-based LWPR algorithms [1,2,3], as well as both the batch and the incremental versions of GMM/GMR [4,5] were implemented, their comparison leading to very similar results, with the same pattern as regards to both the involved robot joints and the different initial experimental conditions. Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages, where the taught motion will be refined by autonomous robot rehearsal through reinforcement learning.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Plant leaf imaging using time of flight camera under sunlight, shadow and room conditions

  • Kazmi, Wajahat
  • Foix Salmerón, Sergi|||0000-0001-9249-6696
  • Alenyà Ribas, Guillem|||0000-0002-6018-154X
In this article, we analyze the effects of ambient light on Time of Flight (ToF) depth imaging for a plant’s leaf in sunlight, shadow and room conditions. ToF imaging is sensitive to ambient light and we try to find the best possible integration times (IT) for each condition. This is important in order to optimize camera calibration. Our analysis is based on several statistical metrics estimated from the ToF data. We explain the estimation of the metrics and propose a method of predicting the deteriorating behavior of the data in each condition using camera flags. Finally, we also propose a method to improve the quality of a ToF image taken in a mixed condition having different ambient light exposures.
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Design of non-anthropomorphic robotic hands for anthropomorphic tasks

  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
  • Pérez Gracia, Alba|||0000-0002-6023-7857
In this paper, we explore the idea of designing non- anthropomorphic multi-fingered robotic hands for tasks tha t replicate the motion of the human hand. Taking as input data a finite set of rigid-body positions for the five fingertips, we de- velop a method to perform dimensional synthesis for a kinema tic chain with a tree structure, with five branches that share thr ee common joints. We state the forward kinematics equations of relative dis- placements for each serial chain expressed as dual quaterni ons, and solve for up to five chains simultaneously to reach a numbe r of positions along the hand trajectory. This is done using a h y- brid global numerical solver that integrates a genetic algo rithm and a Levenberg-Marquardt local optimizer. Although the number of candidate solutions in this problem is very high, the use of the genetic algorithm allows us to per form an exhaustive exploration of the solution space to obtain a s et of solutions. We can then choose some of the solutions based on t he specific task to perform. Note that these designs match the ta sk exactly while generally having a finger design radically dif ferent from that of the human hand., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Plant leaf imaging using time of flight camera under sunlight, shadow and room conditions

  • Kazmi, Wajahat
  • Foix Salmerón, Sergi
  • Alenyà Ribas, Guillem
In this article, we analyze the effects of ambient light on Time of Flight (ToF) depth imaging for a plant’s leaf in sunlight, shadow and room conditions. ToF imaging is sensitive to ambient light and we try to find the best possible integration times (IT) for each condition. This is important in order to optimize camera calibration. Our analysis is based on several statistical metrics estimated from the ToF data. We explain the estimation of the metrics and propose a method of predicting the deteriorating behavior of the data in each condition using camera flags. Finally, we also propose a method to improve the quality of a ToF image taken in a mixed condition having different ambient light exposures., Preprint
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Learning force-based robot skills from haptic demonstration

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Locally weighted as well as Gaussian mixtures learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration. Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. For the first time, force/torque feedback through a haptic device has been used for teaching a teleoperated robot to empty a rigid container. The memory-based LWPLS and the non-memory-based LWPR algorithms [1,2,3], as well as both the batch and the incremental versions of GMM/GMR [4,5] were implemented, their comparison leading to very similar results, with the same pattern as regards to both the involved robot joints and the different initial experimental conditions. Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages, where the taught motion will be refined by autonomous robot rehearsal through reinforcement learning., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Shared random Ferns for efficient detection of multiple categories

  • Villamizar Vergel, Michael Alejandro
  • Moreno-Noguer, Francesc
  • Andrade-Cetto, Juan
  • Sanfeliu Cortés, Alberto
We propose a new algorithm for detecting multiple object categories that exploits the fact that different categories may share common features but with different geometric distributions. This yields an efficient detector which, in contrast to existing approaches, considerably reduces the computation cost at runtime, where the feature computation step is traditionally the most expensive. More specifically, at the learning stage we compute common features by applying the same Random Ferns over the Histograms of Oriented Gradients on the training images. We then apply a boosting step to build discriminative weak classifiers, and learn the specific geometric distribution of the Random Ferns for each class. At runtime, only a few Random Ferns have to be densely computed over each input image, and their geometric distribution allows performing the detection. The proposed method has been validated in public datasets achieving competitive detection results, which are comparable with state-of-the-art methods that use specific features per class., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Robotic leaf probing via segmentation of range data into surface patches

  • Alenyà Ribas, Guillem
  • Dellen, Babette
  • Foix Salmerón, Sergi
  • Torras, Carme
We present a novel method for the robotized probing of plant leaves using Time-of-Flight (ToF) sensors. Plant images are segmented into surface patches by combining a segmentation of the infrared intensity image, provided by the ToF camera, with quadratic surface fitting using ToF depth data. Leaf models are fitted to the boundaries of the segments and used to determine probing points and to evaluate the suitability of leaves for being sampled. The robustness of the approach is evaluated by repeatedly placing an especially adapted, robot-mounted spad meter on the probing points which are extracted in an automatic manner. The number of successful chlorophyll measurements is counted, and the total time for processing the visual data and probing the plant with the robot is measured for each trial. In case of failure, the underlying causes are determined and reported, allowing a better assessment of the applicability of the method in real scenarios., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations

  • Dellen, Babette
  • Husain, Syed Farzad
  • Torras, Carme
A novel framework for joint segmentation and tracking in depth videos of object surfaces is presented. Initially, the 3D colored point cloud obtained using the Kinect camera is used to segment the scene into surface patches, defined by quadratic functions. The computed segments together with their functional descriptions are then used to partition the depth image of the subsequent frame in a consistent manner with respect to the precedent frame. This way, solutions established in previous frames can be reused which improves the efficiency of the algorithm and the coherency of the segmentations along the movie. The algorithm is tested for scenes showing human and robot manipulations of objects. We demonstrate that the method can successfully segment and track the human/robot arm and object surfaces along the manipulations. The performance is evaluated quantitatively by measuring the temporal coherency of the segmentations and the segmentation covering using ground truth. The method provides a visual front-end designed for robotic applications, and can potentially be used in the context of manipulation recognition, visual servoing, and robot-grasping tasks, Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Sharpening haptic inputs for teaching a manipulation skill to a robot

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Gaussian mixtures-based learning algorithms are suitable strategies for trajectory learning and skill acquisition, in the context of programming by demonstration (PbD). Input streams other than visual information, as used in most applications up to date, reveal themselves as quite useful in trajectory learning experiments where visual sources are not available. In this work we have used force/torque feedback through a haptic device for teaching a teleoperated robot to empty a rigid container. Structure vibrations and container inertia appeared to considerably disrupt the sensing process, so a filtering algorithm had to be devised. Moreover, some input variables seemed much more relevant to the particular task to be learned than others, which lead us to analyze the training data in order to select those relevant features through principal component analysis and a mutual information criterion. Then, a batch version of GMM/GMR [1], [2] was implemented using different training datasets (original, pre-processed data through PCA and MI). Tests where the teacher was instructed to follow a strategy compared to others where he was not lead to useful conclusions that permit devising the new research stages., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Robust elastic 2D/3D geometric graph matching

  • Serradell, Eduard
  • Kybic, J.
  • Moreno-Noguer, Francesc
  • Fua, Pascal
We present an algorithm for geometric matching of graphs embedded in 2D or 3D space. It is applicable for registering any graph-like structures appearing in biomedical images, such as blood vessels, pulmonary bronchi, nerve fibers, or dendritic arbors. Our approach does not rely on the similarity of local appearance features, so it is suitable for multimodal registration with a large difference in appearance. Unlike earlier methods, the algorithm uses edge shape, does not require an initial pose estimate, can handle partial matches, and can cope with nonlinear deformations and topological differences. The matching consists of two steps. First, we find an affine transform that roughly aligns the graphs by exploring the set of all consistent correspondences between the nodes. This can be done at an acceptably low computational expense by using parameter uncertainties for pruning, backtracking as needed. Parameter uncertainties are updated in a Kalman-like scheme with each match. In the second step we allow for a nonlinear part of the deformation, modeled as a Gaussian Process. Short sequences of edges are grouped into superedges, which are then matched between graphs. This allows for topological differences. A maximum consistent set of superedge matches is found using a dedicated branch-and-bound solver, which is over 100 times faster than a standard linear programming approach. Geometrical and topological consistency of candidate matches is determined in a fast hierarchical manner. We demonstrate the effectiveness of our technique at registering angiography and retinal fundus images, as well as neural image stacks., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Characterization of textile grasping experiments

  • Alenyà Ribas, Guillem
  • Ramisa Ayats, Arnau
  • Moreno-Noguer, Francesc
  • Torras, Carme
Grasping highly deformable objects, like textiles, is an emerging area of research that involves both percep- tion and manipulation abilities. As new techniques appear, it becomes essential to design strategies to compare them. However, this is not an easy task, since the large state-space of textile objects explodes when coupled with the variability of grippers, robotic hands and robot arms performing the manipulation task. This high variability makes it very difficult to design experiments to evaluate the performance of a system in a repeatable way and compare it to others. We propose a framework to allow the comparison of different grasping methods for textile objects. Instead of measuring each component separately, we there- fore propose a methodology to explicitly measure the vision- manipulation correlation by taking into account the throughput of the actions. Perceptions of deformable objects should be grouped into different clusters, and the different grasping actions available should be tested for each perception type to obtain the action-perception success ratio. This characterization potentially allows to compare very different systems in terms of specialized actions , perceptions or widely useful actions , along with the cost of performing each action. We will also show that this categorization is useful in manipulation planning of deformable objects., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Design of non-anthropomorphic robotic hands for anthropomorphic tasks

  • Simó Serra, Edgar
  • Moreno-Noguer, Francesc
  • Pérez Gracia, Alba
In this paper, we explore the idea of designing non- anthropomorphic multi-fingered robotic hands for tasks tha t replicate the motion of the human hand. Taking as input data a finite set of rigid-body positions for the five fingertips, we de- velop a method to perform dimensional synthesis for a kinema tic chain with a tree structure, with five branches that share thr ee common joints. We state the forward kinematics equations of relative dis- placements for each serial chain expressed as dual quaterni ons, and solve for up to five chains simultaneously to reach a numbe r of positions along the hand trajectory. This is done using a h y- brid global numerical solver that integrates a genetic algo rithm and a Levenberg-Marquardt local optimizer. Although the number of candidate solutions in this problem is very high, the use of the genetic algorithm allows us to per form an exhaustive exploration of the solution space to obtain a s et of solutions. We can then choose some of the solutions based on t he specific task to perform. Note that these designs match the ta sk exactly while generally having a finger design radically dif ferent from that of the human hand., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Towards plant monitoring through next best view

  • Foix Salmerón, Sergi
  • Alenyà Ribas, Guillem
  • Torras, Carme
Monitoring plants using leaf feature detection is a challenging perception task because different leaves, even from the same plant, may have very different shapes, sizes and deformations. In addition, leaves may be occluded by other leaves making it hard to determine some of their characteristics. In this paper we use a Time-of-Flight (ToF) camera mounted on a robot arm to acquire the depth information needed for plant leaf detection. Under a Next Best View (NBV) paradigm, we propose a criterion to compute a new camera position that offers a better view of a target leaf. The proposed criterion exploits some typical errors of the ToF camera, which are common to other 3D sensing devices as well. This approach is also useful when more than one leaf is segmented as the same region, since moving the camera following the same NBV criterion helps to disambiguate this situation., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Simultaneous pose, correspondence and non-rigid shape

  • Sánchez-Riera, Jordi
  • Ostlund, Jonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Recent works have shown that 3D shape of non-rigid surfaces can be accurately retrieved from a single image given a set of 3D-to-2D correspondences between that image and another one for which the shape is known. However, existing approaches assume that such correspondences can be readily established, which is not necessarily true when large deformations produce significant appearance changes between the input and the reference images. Furthermore, it is either assumed that the pose of the camera is known, or the estimated solution is pose-ambiguous. In this paper we relax all these assumptions and, given a set of 3D and 2D unmatched points, we present an approach to simultaneously solve their correspondences, compute the camera pose and retrieve the shape of the surface in the input image. This is achieved by introducing weak priors on the pose and shape that we model as Gaussian Mixtures. By combining them into a Kalman filter we can progressively reduce the number of 2D candidates that can be potentially matched to each 3D point, while pose and shape are refined. This lets us to perform a complete and efficient exploration of the solution space and retain the best solution., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Channel coding for joint colour and depth segmentation

  • Wallenberg, Markus
  • Felsberg, Michael
  • Forssen, Per-Erik
  • Dellen, Babette
Segmentation is an important preprocessing step in many applications. Compared to colour segmentation, fusion of colour and depth greatly improves the segmentation result. Such a fusion is easy to do by stacking measurements in di erent value dimensions, but there are better ways. In this paper we perform fusion using the channel representation, and demonstrate how a state-of-the-art segmentation algorithm can be modi ed to use channel values as inputs. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using the superparamagnetic clustering algorithm. Our experiments show that depth gradients are more useful than depth values for segmentation, and that channel coding both colour and depth gradients makes tuned parameter settings generalise better to novel images., Peer Reviewed, Postprint (updated version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Segmenting color images into surface patches by exploiting sparse depth data

  • Dellen, Babette
  • Alenyà Ribas, Guillem
  • Foix Salmerón, Sergi
  • Torras, Carme
We present a new method for segmenting color images into their composite surfaces by combining color segmentation with model-based fitting utilizing sparse depth data, acquired using time-of-flight (Swissranger, PMD CamCube) and stereo techniques. The main target of our work is the segmentation of plant structures, i.e., leaves, from color-depth images, and the extraction of color and 3D shape information for automating manipulation tasks. Since segmentation is performed in the dense color space, even sparse, incomplete, or noisy depth information can be used. This kind of data often represents a major challenge for methods operating in the 3D data space directly. To achieve our goal, we construct a three-stage segmentation hierarchy by segmenting the color image with different resolutions-assuming that “true” surface boundaries must appear at some point along the segmentation hierarchy. 3D surfaces are then fitted to the color-segment areas using depth data. Those segments which minimize the fitting error are selected and used to construct a new segmentation. Then, an additional region merging and a growing stage are applied to avoid over-segmentation and label previously unclustered points. Experimental results demonstrate that the method is successful in segmenting a variety of domestic objects and plants into quadratic surfaces. At the end of the procedure, the sparse depth data is completed using the extracted surface models, resulting in dense depth maps. For stereo, the resulting disparity maps are compared with ground truth and the average error is computed., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

3D modelling of leaves from color and ToF data for robotized plant measuring

  • Alenyà Ribas, Guillem
  • Dellen, Babette
  • Torras, Carme
Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modelling for this application has strong demands, particularly in what concerns 3D information gathering and speed. This paper shows that Time-of-Flight (ToF) cameras achieve a good compromise between both demands, providing a suitable complement to color vision. A new method is proposed to segment plant images into their composite surface patches by combining hierarchical color segmentation with quadratic surface fitting using ToF depth data. Experimentation shows that the interpolated depth maps derived from the obtained surfaces fit well the original scenes. Moreover, candidate leaves to be approached by a measuring instrument are ranked, and then robot-mounted cameras move closer to them to validate their suitability to being sampled. Some ambiguities arising from leaves overlap or occlusions are cleared up in this way. The work is a proof-of-concept that dense color data combined with sparse depth as provided by a ToF camera yields a good enough 3D approximation for automated plant measuring at the high throughput imposed by the application., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Combining geometric and appearance priors for robust homography estimation

  • Serradell, Eduard
  • Özuysa, Mustafa
  • Lepetit, Vincent
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Abstract. The homography between pairs of images are typically computed from the correspondence of keypoints, which are established by using image descriptors. When these descriptors are not reliable, either because of repetitive patterns or large amounts of clutter, additional priors need to be considered. The Blind PnP algorithm makes use of geometric priors to guide the search for matches while computing camera pose. Inspired by this, we propose a novel approach for homography estimation that combines geometric priors with appearance priors of ambiguous descriptors. More specifically, for each point we retain its best candidates according to appearance. We then prune the set of potential matches by iteratively shrinking the regions of the image that are consistent with the geometric prior. We can then successfully compute homographies between pairs of images containing highly repetitive patterns and even under oblique viewing conditions., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Deformation and illumination invariant feature point descriptor

  • Moreno-Noguer, Francesc
Recent advances in 3D shape recognition have shown that kernels based on diffusion geometry can be effectively used to describe local features of deforming surfaces. In this paper, we introduce a new framework that allows using these kernels on 2D local patches, yielding a novel feature point descriptor that is both invariant to non-rigid image deformations and illumination changes. In order to build the descriptor, 2D image patches are embedded as 3D surfaces, by multiplying the intensity level by an arbitrarily large and constant weight that favors anisotropic diffusion and retains the gradient magnitude information. Patches are then described in terms of a heat kernel signature, which is made invariant to intensity changes, rotation and scaling. The resulting feature point descriptor is proven to be significantly more discriminative than state of the art ones, even those which are specifically designed for describing non-rigid image deformations., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Robust non-rigid registration of 2D and 3D graphs

  • Serradell, Eduard
  • Glowacki, Przemyslaw
  • Jan, Kybic
  • Moreno-Noguer, Francesc
  • Fua, Pascal
We present a new approach to matching graphs embedded in R2 or R3. Unlike earlier methods, our approach does not rely on the similarity of local appearance features, does not require an initial alignment, can handle partial matches, and can cope with non-linear deformations and topological differences. To handle arbitrary non-linear deformations, we represent them as Gaussian Processes. In the absence of appearance information, we iteratively establish correspondences between graph nodes, update the structure accordingly, and use the current mapping estimate to find the most likely correspondences that will be used in the next iteration. This makes the computation tractable. We demonstrate the effectiveness of our approach first on synthetic cases and then on angiography data, retinal fundus images, and microscopy image stacks acquired at very different resolutions., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Integrating task planning and interactive learning for robots to work in human environments

  • Agostini, Alejandro Gabriel
  • Torras, Carme
  • Wörgötter, Florentin
Human environments are challenging for robots, which need to be trainable by lay people and learn new behaviours rapidly without disrupting much the ongoing activity. A system that integrates AI techniques for planning and learning is here proposed to satisfy these strong demands. The approach rapidly learns planning operators from few action experiences using a competitive strategy where many alternatives of cause-effect explanations are evaluated in parallel, and the most successful ones are used to generate the operators. The success of a cause-effect explanation is evaluated by a probabilistic estimate that compensates the lack of experience, producing more confident estimations and speeding up the learning in relation to other known estimates. The system operates without task interruption by integrating in the planning-learning loop a human teacher that supports the planner in making decisions. All the mechanisms are integrated and synchronized in the robot using a general decision-making framework. The feasibility and scalability of the architecture are evaluated in two different robot platforms: a Stäubli arm, and the humanoid ARMAR III., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration of force-based tasks with multiple solution trajectories

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
A learning framework with a bidirectional communication channel is proposed, where a human performs several demonstrations of a task using a haptic device (providing him/her with force-torque feedback) while a robot captures these executions using only its force-based perceptive system. Our work departs from the usual approaches to learning by demonstration in that the robot has to execute the task blindly, relying only on force-torque perceptions, and, more essential, we address goal-driven manipulation tasks with multiple solution trajectories, whereas most works tackle tasks that can be learned by just finding a generalization at the trajectory level. To cope with these multiple-solution tasks, in our framework demonstrations are represented by means of a Hidden Markov Model (HMM) and the robot reproduction of the task is performed using a modified version of Gaussian Mixture Regression that incorporates temporal information (GMRa) through the forward variable of the HMM. Also, we exploit the haptic device as a teaching and communication tool in a human-robot interaction context, as an alternative to kinesthetic-based teaching systems. Results show that the robot is able to learn a container-emptying task relying only on force-based perceptions and to achieve the goal from several non-trained initial conditions., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Robot learning from demonstration in the force domain

  • Rozo Castañeda, Leonel
  • Jiménez Schlegl, Pablo
  • Torras, Carme
Researchers are becoming aware of the importance of other information sources besides visual data in robot learning by demonstration (LbD). Forcebased perceptions are shown to convey very relevant information – missed by visual and position sensors – for learning specific tasks. In this paper, we review some recent works using forces as input data in LbD and Human-Robot interaction (HRI) scenarios, and propose a complete learning framework for teaching force-based manipulation skills to a robot through a haptic device. We suggest to use haptic interfaces not only as a demonstration tool but also as a communication channel between the human and the robot, getting the teacher more involved in the teaching process by experiencing the force signals sensed by the robot. Within the proposed framework, we provide solutions for treating force signals, extracting relevant information about the task, encoding the training data and generalizing to perform successfully under unknown conditions., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Active perception of deformable objects using 3D cameras

  • Alenyà Ribas, Guillem
  • Moreno-Noguer, Francesc
  • Ramisa Ayats, Arnau
  • Torras, Carme
Perception and manipulation of rigid objects has received a lot of attention, and several solutions have been proposed. In contrast, dealing with deformable objects is a relatively new and challenging task because they are more complex to model, their state is difficult to determine, and self-occlusions are common and hard to estimate. In this paper we present our progress/results in the perception of deformable objects both using conventional RGB cameras and active sensing strategies by means of depth cameras.We provide insights in two different areas of application: grasping of textiles and plant leaf modelling., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Simultaneous correspondence and non-rigid 3D reconstruction of the coronary tree from single X-ray images

  • Serradell, Eduard
  • Romero, Adriana
  • Leta, Ruben
  • Gatta, Carlo
  • Moreno-Noguer, Francesc
We present a novel approach to simultaneously reconstruct the 3D structure of a non-rigid coronary tree and estimate point correspondences between an input X-ray image and a reference 3D shape. At the core of our approach lies an optimization scheme that iteratively fits a generative 3D model of increasing complexity and guides the matching process. As a result, and in contrast to existing approaches that assume rigidity or quasi-rigidity of the structure, our method is able to retrieve large non-linear deformations even when the input data is corrupted by the presence of noise and partial occlusions. We extensively evaluate our approach under synthetic and real data and demonstrate a remarkable improvement compared to state-of-the-art., Peer Reviewed, Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Probabilistic simultaneous pose and non-rigid shape recovery

  • Moreno-Noguer, Francesc
  • Porta Pleite, Josep Maria
We present an algorithm to simultaneously recover non-rigid shape and camera poses from point correspondences between a reference shape and a sequence of input images. The key novel contribution of our approach is in bringing the tools of the probabilistic SLAM methodology from a rigid to a deformable domain. Under the assumption that the shape may be represented as a weighted sum of deformation modes, we show that the problem of estimating the modal weights along with the camera poses, may be probabilistically formulated as a maximum a posterior estimate and solved using an iterative least squares optimization. An extensive evaluation on synthetic and real data, shows that our approach has several significant advantages over current approaches, such as performing robustly under large amounts of noise and outliers, and neither requiring to track points over the whole sequence nor initializations close from the ground truth solution., Postprint (author’s final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Depth-supported real-time video segmentation with the Kinect

  • Abramov, Alexey
  • Pauwels, Karl
  • Papon, Jeremie
  • Wörgötter, Florentin
  • Dellen, Babette
We present a real-time technique for the spatiotemporal segmentation of color/depth movies. Images are segmented using a parallel Metropolis algorithm implemented on a GPU utilizing both color and depth information, acquired with the Microsoft Kinect. Segments represent the equilibrium states of a Potts model, where tracking of segments is achieved by warping obtained segment labels to the next frame using real-time optical flow, which reduces the number of iterations required for the Metropolis method to encounter the new equilibrium state. By including depth information into the framework, true objects boundaries can be found more easily, improving also the temporal coherency of the method. The algorithm has been tested for videos of medium resolutions showing human manipulations of objects. The framework provides an inexpensive visual front end for visual preprocessing of videos in industrial settings and robot labs which can potentially be used in various applications., Peer Reviewed, Postprint (published version)