Pasar al contenido principal

ROBINSTRUCT: INSTRUYENDO ROBOTS USANDO HABILIDADES COMUNICATIVAS NATURALES

TIN2014-58178-R

•
Nombre agencia financiadora Ministerio de Economía y Competitividad
Acrónimo agencia financiadora MINECO
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos I+D+i Retos de Investigación
Año convocatoria 2014
Plan Estatal 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Unidad de gestión Secretaría de Estado de Educación, Formación Profesional y Universidades
Centro beneficiario AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS (CSIC)
Identificador persistente http://dx.doi.org/10.13039/501100003329

Resultados relacionados

Resultados totales (Incluyendo duplicados): 28
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Mode-shape interpretation: Re-thinking modal space for recovering deformable shapes

  • Agudo Martínez, Antonio
  • Montiel, J. M. M.
  • Calvo, Begoña
  • Moreno-Noguer, Francesc
Trabajo presentado al WACV 2016: IEEE Winter Conference on Applications of Computer Vision, celebrado en Lake Placid, NY(US) del 7 al 9 de marzo de 2016., This paper describes an on-line approach for estimating non-rigid shape and camera pose from monocular video sequences. We assume an initial estimate of the shape at rest to be given and represented by a triangulated mesh, which is encoded by a matrix of the distances between every pair of vertexes. By applying spectral analysis on this matrix, we are then able to compute a low-dimensional shape basis, that in contrast to standard approaches, has a very direct physical interpretation and requires a much smaller number of modes to span a large variety of deformations, either for inextensible or extensible configurations. Based on this low-rank model, we then sequentially retrieve both camera motion and non-rigid shape in each image, optimizing the model parameters with bundle adjustment over a sliding window of image frames. Since the number of these parameters is small, specially when considering physical priors, our approach may potentially achieve real-time performance. Experimental results on real videos for different scenarios demonstrate remarkable robustness to artifacts such as missing and noisy observations., This work has been partially supported by the Spanish MCI under projects RobInstruct TIN2014-58178-R, SVMap DIP2012-32168 and Keratocono DPI2014-54981-R; by the ERA-net CHISTERA projects VISEN PCIN-2013-047 and I-DRESS PCIN-2015-147; and by a scholarship FPU12/04886 of the Spanish MECD., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Learning relational dynamics of stochastic domains for planning

  • Martínez, David
  • Alenyà, Guillem
  • Torras, Carme
  • Ribeiro, Tony
  • Inoue, Katsumi
Trabajo presentado al 26th International Conference on Automated Planning and Scheduling (ICAPS) celebrado en Londres (UK) del 12 al 17 de junio de 2016., Probabilistic planners are very flexible tools that can provide good solutions for difficult tasks. However, they rely on a model of the domain, which may be costly to either hand code or automatically learn for complex tasks. We propose a new learning approach that (a) requires only a set of state transitions to learn the model; (b) can cope with uncertainty in the effects; (c) uses a relational representation to generalize over different objects; and (d) in addition to action effects, it can also learn exogenous effects that are not related to any action, e.g., moving objects, endogenous growth and natural development. The proposed learning approach combines a multi-valued variant of inductive logic programming for the generation of candidate models, with an optimization method to select the best set of planning operators to model a problem. Finally, experimental validation is provided that shows improvements over previous work., This work has been supported by the MINECO project RobInstruct TIN2014-58178-R and the ERA-Net CHISTERA project I-DRESS PCIN-2015-147. D. Martínez is also supported by the Spanish Ministry of Education, Culture and Sport via a FPU doctoral grant (FPU12-04173)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

High-frequency MAV state estimation using low-cost inertial and optical flow measurement units

  • Santamaria-Navarro, Àngel
  • Solà, Joan
  • Andrade-Cetto, Juan
Trabajo presentado a la International Conference on Intelligent Robots and Systems celebrada en Hamburgo (Alemania) del 28 de septiembre al 2 de octubre de 2015., This paper develops a simple and low-cost method for 3D, high-rate vehicle state estimation, specially designed for free-flying Micro Aerial Vehicles (MAVs). We fuse observations from inertial measurement units and the recently appeared low-cost optical flow smart cameras. These smart cameras integrate a sonar altimeter, a triaxial gyrometer and an optical flow sensor, and directly provide metric ego-motion information in the form of body velocities and altitude. Compared to state-of-the-art visual-inertial odometry methods, we are able to drastically reduce the computational load in the main processor unit, and obtain an accurate estimation of the vehicle state at a high update rate of 100Hz. We thus extend the current use of these smart cameras from hovering purposes to odometry estimation. In order to propose a simple algorithmic solution, we investigate the performances of two Kalman filters, in the extended and error-state flavors, alongside a large number of algorithm variations, using simulations and real experiments with precise ground-truth. We observe that the marginal performance gain attained with these algorithm improvements does not pay for the effort of implementing them. We conclude that a classical EKF in its simplest form is sufficient for providing motion estimates that coherently exploit the available measurements., This work has been partially funded by the EU project ARCAS FP7-287617. and by the Spanish Ministry of Economy and Competitiveness projects PAU+ DPI-2011-27510 and ROBINSTRUCT TIN2014-58178-R., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Learning shape, motion and elastic models in force space

  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Trabajo presentado a la International Conference on Computer Vision celebrada en Santiago (España) del 7 al 13 de diciembre de 2015., In this paper, we address the problem of simultaneously recovering the 3D shape and pose of a deformable and potentially elastic object from 2D motion. This is a highly ambiguous problem typically tackled by using low-rank shape and trajectory constraints. We show that formulating the problem in terms of a low-rank force space that induces the deformation, allows for a better physical interpretation of the resulting priors and a more accurate representation of the actual object's behavior. However, this comes at the price of, besides force and pose, having to estimate the elastic model of the object. For this, we use an Expectation Maximization strategy, where each of these parameters are successively learned within partial M-steps, while robustly dealing with missing observations. We thoroughly validate the approach on both mocap and real sequences, showing more accurate 3D reconstructions than state-of-the-art, and additionally providing an estimate of the full elastic model with no a priori information., This work was partly funded by the MINECO projects DIP2012-32168 and TIN2014-58178-R; by the ERA-net CHISTERA project VISEN PCIN-2013-047; and by a scholarship FPU12/04886 of the Spanish MECD., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Semantic tuples for evaluation of image sentence generation

  • Ellebracht, Lily D.
  • Ramisa, Arnau
  • Swaroop Madhyastha, Pranava
  • Cordero-Rama, Jose
  • Moreno-Noguer, Francesc
  • Quattoni, Ariadna
Trabajo presentado al Workshop on Vision and Language (VL’15), celebrado en Lisboa (Portugal) el 18 de semptiebre de 2015., The automatic generation of image captions has received considerable attention. The problem of evaluating caption generation systems, though, has not been that much explored. We propose a novel evaluation approach based on comparing the underlying visual semantics of the candidate and ground-truth captions. With this goal in mind we have defined a semantic representation for visually descriptive language and have augmented a subset of the Flickr-8K dataset with semantic annotations. Our evaluation metric (BAST) can be used not only to compare systems but also to do error analysis and get a better understanding of the type of mistakes a system does. To compute BAST we need to predict the semantic representation for the automatically generated captions. We use the Flickr-ST dataset to train classifiers that predict STs so that evaluation can be fully automated., This work was partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R and by the ERA-net CHISTERA project VISEN PCIN-2013-047., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Combining geometric, textual and visual features for predicting prepositions in image descriptions

  • Ramisa, Arnau
  • Wang, Josiah
  • Lu, Ying
  • Dellandrea, Emmanuel
  • Moreno-Noguer, Francesc
  • Gaizauskas, Robert
Trabajo presentado a la Conference on Empirical Methods in Natural Language Processing celebrada en Lisboa (Portugal) del 17 al 21 de septiembre de 2015., We investigate the role that geometric, textual and visual features play in the task of predicting a preposition that links two visual entities depicted in an image. The task is an important part of the subsequent process of generating image descriptions. We explore the prediction of prepositions for a pair of entities, both in the case when the labels of such entities are known and unknown. In all situations we found clear evidence that all three features contribute to the prediction task., This work was funded by the ERA-Net CHISTERA D2K VisualSense project (Spanish MINECO PCIN-2013-047, UK EPSRC EP/K019082/1 and French ANR Grant ANR-12-CHRI-0002-04) and the Spanish MINECO RobInstruct project TIN2014-58178-R. Ying Lu was also supported by the China Scholarship Council., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Discriminative learning of deep convolutional feature point descriptors

  • Simo-Serra, Edgar
  • Trulls, Eduard
  • Ferraz, Luis
  • Kokkinos, Iasonas
  • Fua, Pascal
  • Moreno-Noguer, Francesc
Trabajo presentado a la International Conference on Computer Vision celebrada en Santiago (España) del 7 al 13 de diciembre de 2015., Deep learning has revolutionalized image-level tasks such as classification, but patch-level tasks, such as correspondence, still rely on hand-crafted features, e.g. SIFT. In this paper we use Convolutional Neural Networks (CNNs) to learn discriminant patch representations and in particular train a Siamese network with pairs of (non-)corresponding patches. We deal with the large number of potential pairs with the combination of a stochastic sampling of the training set and an aggressive mining strategy biased towards patches that are hard to classify. By using the L2 distance during both training and testing we develop 128-D descriptors whose euclidean distances reflect patch similarity, and which can be used as a drop-in replacement for any task involving SIFT. We demonstrate consistent performance gains over the state of the art, and generalize well against scaling and rotation, perspective transformation, non-rigid deformation, and illumination changes. Our descriptors are efficient to compute and amenable to modern GPUs, and are publicly available., This work was partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project ViSen PCIN-2013-047, by EU projects AEROARMS H2020-ICT-2014-1-644271, ISUPPORT H2020-ICT-2014-1-643666 and MOBOT FP7-ICT-2011-600796, and by the ERC project MicroNano., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Matchability prediction for full-search template matching algorithms

  • Peñate-Sánchez, Adrián
  • Porzi, Lorenzo
  • Moreno-Noguer, Francesc
Trabajo presentado a la International Conference on 3D Vision, celebrada en Lyon (Francia) del 19 al 22 de octubre de 2015., While recent approaches have shown that it is possible to do template matching by exhaustively scanning the parameter space, the resulting algorithms are still quite demanding. In this paper we alleviate the computational load of these algorithms by proposing an efficient approach for predicting the match ability of a template, before it is actually performed. This avoids large amounts of unnecessary computations. We learn the match ability of templates by using dense convolutional neural network descriptors that do not require ad-hoc criteria to characterize a template. By using deep learning descriptions of patches we are able to predict match ability over the whole image quite reliably. We will also show how no specific training data is required to solve problems like panorama stitching in which you usually require data from the scene in question. Due to the highly parallelizable nature of this tasks we offer an efficient technique with a negligible computational cost at test time., This work has been partially funded by Spanish Ministry of Economy and Competitiveness under project RobInstruct TIN2014-58178-R and ERA-Net Chistera project ViSen PCIN-2013-047; and by the EU project AEROARMS H2020-ICT-2014-1-644271., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

BASS: Boundary-aware superpixel segmentation

  • Rubio Romano, Antonio
  • Yu, LongLong
  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
Trabajo presentado a la 23rd International Conference on Pattern Recognition, celebrada en Cancún (México) del 5 al 8 de diciembre de 2016., We propose a new superpixel algorithm based on exploiting the boundary information of an image, as objects in images can generally be described by their boundaries. Our proposed approach initially estimates the boundaries and uses them to place superpixel seeds in the areas in which they are more dense. Afterwards, we minimize an energy function in order to expand the seeds into full superpixels. In addition to standard terms such as color consistency and compactness, we propose using the geodesic distance which concentrates small superpixels in regions of the image with more information, while letting larger superpixels cover more homogeneous regions. By both improving the initialization using the boundaries and coherency of the superpixels with geodesic distances, we are able to maintain the coherency of the image structure with fewer superpixels than other approaches. We show the resulting algorithm to yield smaller Variation of Information metrics in seven different datasets while maintaining Undersegmentation Error values similar to the state-of-the-art methods., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project I-DRESS PCIN-2015-147 and by the EU project AEROARMS H2020-ICT-2014-1-644271. A. Rubio is supported by the industrial doctorate grant 2015-DI-010 of the AGAUR., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Recovering pose and 3D deformable shape from multi-instance image ensembles

  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Trabajo presentado a la 13th Asian Conference on Computer Vision (ACCV), celebrada en Taipei (Taiwan) del 20 al 24 de noviembre de 2016., In recent years, there has been a growing interest on tackling the Non-Rigid Structure from Motion problem (NRSfM), where the shape of a deformable object and the pose of a moving camera are simultaneously estimated from a monocular video sequence. Existing solutions are limited to single objects and continuous, smoothly changing sequences. In this paper we extend NRSfM to a multi-instance domain, in which the images do not need to have temporal consistency, allowing for instance, to jointly reconstruct the face of multiple persons from an unordered list of images. For this purpose, we present a new formulation of the problem based on a dual low-rank shape representation, that simultaneously captures the between- and within-individual deformations. The parameters of this model are learned using a variant of the probabilistic linear discriminant analysis that requires consecutive batches of expectation and maximization steps. The resulting approach estimates 3D deformable shape and pose of multiple instances from only 2D point observations on a collection images, without requiring pre-trained 3D data, and is shown to be robust to noisy measurements and missing points. We provide quantitative and qualitative evaluation on both synthetic and real data, and show consistent benefits compared to current state of the art., This work has been partially supported by the Spanish Ministry of Science and Innovation under project RobInstruct TIN2014-58178-R; by the ERA-net CHISTERA projects VISEN PCIN-2013-047 and I-DRESS PCIN-2015-147., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Nonlinear model predictive control for aerial manipulation

  • Lunni, Dario
  • Santamaria-Navarro, Àngel
  • Rossi, Roberto
  • Rocco, Paolo
  • Bascetta, Luca
  • Andrade-Cetto, Juan
Trabajo presentado a la International Conference on Unmanned Aircraft Systems (ICUAS), celebrada en Miami (US) del 13 al 16 de junio de 2017., This paper presents a nonlinear model predictive controller to follow desired 3D trajectories with the end effector of an unmanned aerial manipulator (i.e., a multirotor with a serial arm attached). To the knowledge of the authors, this is the first time that such controller runs online and on board a limited computational unit to drive a kinematically augmented aerial vehicle. Besides the trajectory following target, we explore the possibility of accomplishing other tasks during flight by taking advantage of the system redundancy. We define several tasks designed for aerial manipulators and show in simulation case studies how they can be achieved by either a weighting strategy, within a main optimization process, or a hierarchical approach consisting on nested optimizations. Moreover, experiments are presented to demonstrate the performance of such controller in a real robot., Their work was partially funded by the EU project AEROARMS H2020-ICT-2014-1-644271, and by the Spanish Ministry of Economy and Competitiveness project ROBINSTRUCT TIN2014-58178-R., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

PL-SLAM: Real-time monocular visual SLAM with points and lines

  • Pumarola, Albert
  • Vakhitov, Alexander
  • Agudo Martínez, Antonio
  • Sanfeliu, Alberto
  • Moreno-Noguer, Francesc
Trabajo presentado a la IEEE International Conference on Robotics and Automation (ICRA), celebrada en Singapur del 29 de mayo al 3 de junio de 2017., Low textured scenes are well known to be one of the main Achilles heels of geometric computer vision algorithms relying on point correspondences, and in particular for visual SLAM. Yet, there are many environments in which, despite being low textured, one can still reliably estimate line-based geometric primitives, for instance in city and indoor scenes, or in the so-called ``Manhattan worlds'', where structured edges are predominant. In this paper we propose a solution to handle these situations. Specifically, we build upon ORB-SLAM, presumably the current state-of-the-art solution both in terms of accuracy as efficiency, and extend its formulation to simultaneously handle both point and line correspondences. We propose a solution that can even work when most of the points are vanished out from the input images, and, interestingly it can be initialized from solely the detection of line correspondences in three consecutive frames. We thoroughly evaluate our approach and the new initialization strategy on the TUM RGB-D benchmark and demonstrate that the use of lines does not only improve the performance of the original ORB-SLAM solution in poorly textured frames, but also systematically improves it in sequence frames combining points and lines, without compromising the efficiency., This work has been partially supported by the EU project AEROARMS H2020-ICT-2014-1-644271, by the MINECO projects RobInstruct TIN2014-58178-R and Rob-Int-Coop DPI2013-42458-P, by the ERA-Net Chistera project IDRESS PCIN-2015-147 and by the Russian MES grant RFMEFI61516X0003., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Global model with local interpretation for dynamic shape reconstruction

  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Trabajo presentado a la IEEE Winter Conference on Applications of Computer Vision (WACV), celebrada en Santa Rosa, California (US) del 24 al 31 de marzo de 2017., The most standard approach to resolve the inherent ambiguities of the non-rigid structure from motion problem is using low-rank models that approximate deforming shapes by a linear combination of rigid basis. These models are typically global, i.e., each shape basis contributes equally to all points of the surface. While this approach has been shown effective to represent smooth deformations, its performance degrades for surfaces composed of various regions, each following a different deformation rule. Piecewise methods attempt to capture this type of behavior by locally modeling surface patches, although they subsequently require enforcing global constraints to assemble back the patches. In this paper we propose an approach that combines the best of global and local models: it locally considers low-rank models but, by construction, does not need to impose global constraints to guarantee local patch continuity. We achieve this by a simple expectation maximization strategy that besides learning global shape bases, it locally adapts their contribution to each specific surface region. Furthermore, as a side contribution, in order to split the surface into different local patches, we propose a novel physically-based mesh segmentation approach that obeys an energy criterion. The complete framework is evaluated in both synthetic and real datasets, and shows an improved performance to competing methods., This work has been partially supported by the Spanish Ministry of Science and Innovation under project RobInstruct TIN2014-58178-R, by the ERA-net CHISTERA project I-DRESS PCIN-2015-147 and by the EU project LOGIMATIC H2020-Galileo-2015-1-687534., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Depth-aware convolutional neural networks for accurate 3D pose estimation in RGB-D images

  • Porzi, Lorenzo
  • Peñate-Sánchez, Adrián
  • Ricci, Elisa
  • Moreno-Noguer, Francesc
Trabajo presentado a la IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), celebrada en Vancouver (Canada) del 24 al 28 de septiembre de 2017., Most recent approaches to 3D pose estimation from RGB-D images address the problem in a two-stage pipeline. First, they learn a classifier-typically a random forest-to predict the position of each input pixel on the object surface. These estimates are then used to define an energy function that is minimized w.r.t. the object pose. In this paper, we focus on the first stage of the problem and propose a novel classifier based on a depth-aware Convolutional Neural Network. This classifier is able to learn a scale-adaptive regression model that yields very accurate pixel-level predictions, allowing to finally estimate the pose using a simple RANSAC-based scheme, with no need to optimize complex ad hoc energy functions. Our experiments on publicly available datasets show that our approach achieves remarkable improvements over state-of-the-art methods., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project I-DRESS PCIN-2015-147, by the EU project AEROARMS H2020-ICT-2014-1-644271, by the EU project SECOND HANDS H2020-ICT-2014-1-643950 and by the Spanish State Research Agency through the Mara de Maeztu Seal of Excellence to IRI MDM-2016-0656., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

3D human pose estimation from a single image via distance matrix regression

  • Moreno-Noguer, Francesc
Trabajo presentado a la 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), celebrada en Honolulu, Hawaii (US) del 21 al 26 de julio de 2016., This paper addresses the problem of 3D human pose estimation from a single image. We follow a standard two-step pipeline by first detecting the 2D position of the N body joints, and then using these observations to infer 3D pose. For the first step, we use a recent CNN-based detector. For the second step, most existing approaches perform 2N-to-3N regression of the Cartesian joint coordinates. We show that more precise pose estimates can be obtained by representing both the 2D and 3D human poses using $Ntimes N$ distance matrices, and formulating the problem as a 2D-to-3D distance matrix regression. For learning such a regressor we leverage on simple Neural Network architectures, which by construction, enforce positivity and symmetry of the predicted matrices. The approach has also the advantage to naturally handle missing observations and allowing to hypothesize the position of non-observed joints. Quantitative results on Humaneva and Human3.6M datasets demonstrate consistent performance gains over state-of-the-art. Qualitative evaluation on the images in-the-wild of the LSP dataset, using the regressor learned on Human3.6M, reveals very promising generalization results., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R and by the ERANet Chistera project I-DRESS PCIN-2015-147., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

DUST: Dual union of spatio-temporal subspaces for monocular multiple object 3D reconstruction

  • Agudo Martínez, Antonio
  • Moreno-Noguer, Francesc
Trabajo presentado a la 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), celebrada en Honolulu, Hawaii (US) del 21 al 26 de julio de 2016., We present an approach to reconstruct the 3D shape of multiple deforming objects from incomplete 2D trajectories acquired by a single camera. Additionally, we simultaneously provide spatial segmentation (i.e., we identify each of the objects in every frame) and temporal clustering (i.e., we split the sequence into primitive actions). This advances existing work, which only tackled the problem for one single object and non-occluded tracks. In order to handle several objects at a time from partial observations, we model point trajectories as a union of spatial and temporal subspaces, and optimize the parameters of both modalities, the non-observed point tracks and the 3D shape via augmented Lagrange multipliers. The algorithm is fully unsupervised and results in a formulation which does not need initialization. We thoroughly validate the method on challenging scenarios with several human subjects performing different activities which involve complex motions and close interaction. We show our approach achieves state-of-the-art 3D reconstruction results, while it also provides spatial and temporal segmentation., This work has been partially supported by the Spanish Ministry of Science and Innovation under project RobInstruct TIN2014-58178-R, by the ERA-net CHISTERA project I-DRESS PCIN-2015-147 and by the EU project LOGIMATIC H2020-Galileo-2015-1-687534., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Multi-modal joint embedding for fashion product retrieval

  • Rubio Romano, Antonio
  • Yu, LongLong
  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
Trabajo presentado a la IEEE International Conference on Image Processing (ICIP), celebrada en Beijing (China) del 17 al 20 de septiembre de 2017., Finding a product in the fashion world can be a daunting task. Everyday, e-commerce sites are updating with thousands of images and their associated metadata (textual information), deepening the problem, akin to finding a needle in a haystack. In this paper, we leverage both the images and textual meta-data and propose a joint multi-modal embedding that maps both the text and images into a common latent space. Distances in the latent space correspond to similarity between products, allowing us to effectively perform retrieval in this latent space, which is both efficient and accurate. We train this embedding using large-scale real world e-commerce data by both minimizing the similarity between related products and using auxiliary classification networks to that encourage the embedding to have semantic meaning. We compare against existing approaches and show significant improvements in retrieval tasks on a large-scale e-commerce dataset. We also provide an analysis of the different metadata., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R, by the ERA-Net Chistera project I-DRESS PCIN-2015-147 and by the EU project iAEROARMS H2020-ICT-2014-1-644271. A.Rubio is supported by the industrial doctorate grant 2015-DI-010 of the AGAUR., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Joint coarse-and-fine reasoning for deep optical flow

  • Vaquero, Victor
  • Ros, German
  • Moreno-Noguer, Francesc
  • López, Antonio M.
  • Sanfeliu, Alberto
Trabajo presentado a la IEEE International Conference on Image Processing (ICIP), celebrada en Beijing (China) del 17 al 20 de septiembre de 2017., We propose a novel representation for dense pixel-wise estimation tasks using CNNs that boosts accuracy and reduces training time, by explicitly exploiting joint coarse-and-fine reasoning. The coarse reasoning is performed over a discrete classification space to obtain a general rough solution, while the fine details of the solution are obtained over a continuous regression space. In our approach both components are jointly estimated, which proved to be beneficial for improving estimation accuracy. Additionally, we propose a new network architecture, which combines coarse and fine components by treating the fine estimation as a refinement built on top of the coarse solution, and therefore adding details to the general prediction. We apply our approach to the challenging problem of optical flow estimation and empirically validate it against state-of-the-art CNN-based solutions trained from scratch and tested on large optical flow datasets., This work was partially supported by European AEROARMS project (H2020-ICT-2014-1-644271) and CICYT projects ColRobTransp (DPI2016-78957-R), ROBINSTRUCT (TIN2014-58178-R)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Robot-aided cloth classification using depth information and CNNs

  • Gabás Nova, Antoni
  • Corona Puyane, Enric
  • Alenyà, Guillem
  • Torras, Carme
Trabajo presentado a la 9th International Conference on Articulated Motion and Deformable Objects (AMDO), celebrada en Palma de Mallorca (España) del 13 al 15 de julio de 2016., We present a system to deal with the problem of classifying garments from a pile of clothes. This system uses a robot arm to extract a garment and show it to a depth camera. Using only depth images of a partial view of the garment as input, a deep convolutional neural network has been trained to classify different types of garments. The robot can rotate the garment along the vertical axis in order to provide different views of the garment to enlarge the prediction confidence and avoid confusions. In addition to obtaining very high classification scores, compared to previous approaches to cloth classification that match the sensed data against a database, our system provides a fast and occlusion-robust solution to the problem., This work was partially supported by the EU CHISTERA I-DRESS project PCIN-2015-147, by the Spanish Ministry of Economy and Competitiveness under project Robinstruct TIN2014-58178-R, and by the CSIC project TextilRob 201550E028., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Structured prediction with output embeddings for semantic image annotation

  • Quattoni, Ariadna
  • Ramisa, Arnau
  • Swaroop Madhyastha, Pranava
  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
Trabajo presentado a la conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT), celebrada en San Diego, California (US) del 12 al 17 de junio de 2016., We address the task of annotating images with semantic tuples. Solving this problem requires an algorithm able to deal with hundreds of classes for each argument of the tuple. In such contexts, data sparsity becomes a key challenge. We propose handling this sparsity by incorporating feature representations of both the inputs (images) and outputs (argument classes) into a factorized log-linear model., This work was partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R and by the ERA-net CHISTERA projects VISEN PCIN- 2013-047 and I-DRESS PCIN-2015-147., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Accurate and linear time pose estimation from points and lines

  • Vakhitov, Alexander
  • Funke, Jan
  • Moreno-Noguer, Francesc
Póster presentado a la 14th European Conference on Computer Vision (ECCV), celebrada en Amsterdam (Netherlands) del 11 al 14 de octubre de 2016., The Perspective-n-Point (PnP) problem seeks to estimate the pose of a calibrated camera from n 3D-to-2D point correspondences. There are situations, though, where PnP solutions are prone to fail because feature point correspondences cannot be reliably estimated (e.g. scenes with repetitive patterns or with low texture). In such scenarios, one can still exploit alternative geometric entities, such as lines, yielding the so-called Perspective-n-Line (PnL) algorithms. Unfortunately, existing PnL solutions are not as accurate and efficient as their point-based counterparts. In this paper we propose a novel approach to introduce 3D-to-2D line correspondences into a PnP formulation, allowing to simultaneously process points and lines. For this purpose we introduce an algebraic line error that can be formulated as linear constraints on the line endpoints, even when these are not directly observable. These constraints can then be naturally integrated within the linear formulations of two state-of-the-art point-based algorithms, the OPnP and the EPnP, allowing them to indistinctly handle points, lines, or a combination of them. Exhaustive experiments show that the proposed formulation brings remarkable boost in performance compared to only point or only line based solutions, with a negligible computational overhead compared to the original OPnP and EPnP., This work is partly funded by the Russian MES grant RFMEFI61516X0003; by the Spanish MINECO project RobInstruct TIN2014-58178-R and by the ERA-Net Chistera project I-DRESS PCIN-2015-147., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

User evaluation of an interactive learning framework for single-arm and dual-arm robots

  • Jevtić, Aleksandar
  • Colomé, Adrià
  • Torras, Carme
Trabajo presentado a la International Conference on Social Robotics (ICSR), celebrada en Kansas (USA) del 1 al 3 de noviembre de 2016., Social robots are expected to adapt to their users and, like their human counterparts, learn from the interaction. In our previous work, we proposed an interactive learning framework that enables a user to intervene and modify a segment of the robot arm trajectory. The framework uses gesture teleoperation and reinforcement learning to learn new motions. In the current work, we compared the user experience with the proposed framework implemented on the single-arm and dual-arm Barrett¿s 7-DOF WAM robots equipped with a Microsoft Kinect camera for user tracking and gesture recognition. User performance and workload were measured in a series of trials with two groups of 6 participants using two robot settings in different order for counterbalancing. The experimental results showed that, for the same task, users required less time and produced shorter robot trajectories with the single-arm robot than with the dual-arm robot. The results also showed that the users who performed the task with the single-arm robot first experienced considerably less workload in performing the task with the dual-arm robot while achieving a higher task success rate in a shorter time., This work was supported by the Beatriu de Pinós fellowship, reference num.: 2013 BP-B 00239, jointly funded by the Government of Catalunya, Spain and the European Commission FP7 COFUND programme. The work was partially supported by the EU CHIST-ERA I-DRESS project, reference num. PCIN-2015-147, and the national Spanish project RobInstruct, reference num. TIN2014-58178-R., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Personalization framework for adaptive robotic feeding assistance

  • Canal, Gerard
  • Alenyà, Guillem
  • Alenyà, Guillem
  • Torras, Carme
Trabajo presentado a la International Conference on Social Robotics (ICSR), celebrada en Kansas (USA) del 1 al 3 de noviembre de 2016., The deployment of robots at home must involve robots with pre-defined skills and the capability of personalizing their behavior by non-expert users. A framework to tackle this personalization is presented and applied to an automatic feeding task. The personalization involves the caregiver providing several examples of feeding using Learning-by-Demostration, and a ProMP formalism to compute an overall trajectory and the variance along the path. Experiments show the validity of the approach in generating different feeding motions to adapt to user¿s preferences, automatically extracting the relevant task parameters. The importance of the nature of the demonstrations is also assessed, and two training strategies are compared., This work has been supported by the MINECO project RobInstruct TIN2014-58178-R and the ERA-Net CHIST-ERA project I-DRESS PCIN-2015-147. Gerard Canal is also supported by the Ministry of Economy and Knowledge of the Government of Catalonia via a FI-DGR 2016 fellowship., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Low resolution lidar-based multi object tracking for driving applications

  • Pino, Iván del
  • Vaquero, Victor
  • Masini, Beatrice
  • Solà, Joan
  • Moreno-Noguer, Francesc
  • Sanfeliu, Alberto
  • Andrade-Cetto, Juan
Trabajo presentado a la Third Iberian Robotics Conference, celebrada en Sevilla del 22 al 24 de noviembre de 2017., Vehicle detection and tracking in real scenarios are key com- ponents to develop assisted and autonomous driving systems. Lidar sen- sors are specially suitable for this task, as they bring robustness to harsh weather conditions while providing accurate spatial information. How- ever, the resolution provided by point cloud data is very scarce in com- parison to camera images. In this work we explore the possibilities of Deep Learning (DL) methodologies applied to low resolution 3D lidar sensors such as the Velodyne VLP-16 (PUCK), in the context of vehicle detection and tracking. For this purpose we developed a lidar-based sys- tem that uses a Convolutional Neural Network (CNN), to perform point- wise vehicle detection using PUCK data, and Multi-Hypothesis Extended Kalman Filters (MH-EKF), to estimate the actual position and veloci- ties of the detected vehicles. Comparative studies between the proposed lower resolution (VLP-16) tracking system and a high-end system, using Velodyne HDL-64, were carried out on the Kitti Tracking Benchmark dataset. Moreover, to analyze the influence of the CNN-based vehicle detection approach, comparisons were also performed with respect to the geometric-only detector. The results demonstrate that the proposed low resolution Deep Learning architecture is able to successfully accom- plish the vehicle detection task, outperforming the geometric baseline approach. Moreover, it has been observed that our system achieves a similar tracking performance to the high-end HDL-64 sensor at close range. On the other hand, at long range, detection is limited to half the distance of the higher-end sensor., This work has been supported by the Spanish Ministry of Economy and Competitiveness projects ROBINSTRUCT (TIN2014-58178-R) and COLROBTRANSP (DPI2016-78957-R), by the Spanish Ministry of Education FPU grant (FPU15/04446), the Spanish State Research Agency through the María de Maeztu Seal of Excellence to IRI (MDM-2016-0656) and by the EU H2020 project LOGIMATIC (H2020-Galileo-2015-1-687534)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Deconvolutional networks for point-cloud vehicle detection and tracking in driving scenarios

  • Vaquero, Victor
  • Pino, Iván del
  • Moreno-Noguer, Francesc
  • Solà, Joan
  • Sanfeliu, Alberto
  • Andrade-Cetto, Juan
Trabajo presentado a la European Conference on Mobile Robots (ECMR), celebrada en Paris (Francia) del 6 al 8 de septiembre de 2017., Vehicle detection and tracking is a core ingredient for developing autonomous driving applications in urban scenarios. Recent image-based Deep Learning (DL) techniques are obtaining breakthrough results in these perceptive tasks. However, DL research has not yet advanced much towards processing 3D point clouds from lidar range-finders. These sensors are very common in autonomous vehicles since, despite not providing as semantically rich information as images, their performance is more robust under harsh weather conditions than vision sensors. In this paper we present a full vehicle detection and tracking system that works with 3D lidar information only. Our detection step uses a Convolutional Neural Network (CNN) that receives as input a featured representation of the 3D information provided by a Velodyne HDL-64 sensor and returns a per-point classification of whether it belongs to a vehicle or not. The classified point cloud is then geometrically processed to generate observations for a multi-object tracking system implemented via a number of Multi-Hypothesis Extended Kalman Filters (MH-EKF) that estimate the position and velocity of the surrounding vehicles. The system is thoroughly evaluated on the KITTI tracking dataset, and we show the performance boost provided by our CNN-based vehicle detector over a standard geometric approach. Our lidar-based approach uses about a 4% of the data needed for an image-based detector with similarly competitive results., This work has been supported by the Spanish Ministry of Economy and Competitiveness projects ROBINSTRUCT (TIN2014-58178-R) and COLROBTRANSP (DPI2016-78957-R), by the Spanish Ministry of Education FPU grant (FPU15/04446), the Spanish State Research Agency through the María de Maeztu Seal of Excellence to IRI MDM-2016-0656) and by the EU H2020 project LOGIMATIC (H2020-Galileo-015-1-687534)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Factor descent optimization for sparsification in graph SLAM

  • Vallvé, Joan
  • Solà, Joan
  • Andrade-Cetto, Juan
Trabajo presentado a la European Conference on Mobile Robots (ECMR), celebrada en Paris (Francia) del 6 al 8 de septiembre de 2017., In the context of graph-based simultaneous localization and mapping, node pruning consists in removing a subset of nodes from the graph, while keeping the graph¿s information content as close as possible to the original. One often tackles this problem locally by isolating the Markov blanket sub-graph of a node, marginalizing this node and sparsifying the dense result. It means computing an approximation with a new set of factors. For a given approximation topology, the factors¿ mean and covariance that best approximate the original distribution can be obtained through minimization of the Kullback-Liebler divergence. For simple topologies such as Chow-Liu trees, there is a closed form for the optimal solution. However, a tree is oftentimes too sparse to explain some graphs. More complex topologies require nonlinear iterative optimization. In the present paper we propose Factor Descent, a new iterative optimization method to sparsify the dense result of node marginalization, which works by iterating factor by factor. We also provide a thorough comparison of our approach with state-of-the-art methods in real world datasets with regards to the obtained solution and convergence rates., This work has been supported by the Spanish Ministry of Economy and Competitiveness under Project ROBINSTRUCT (TIN2014-58178-R), by the EU H2020 Project LOGIMATIC (H2020-Galileo-2015-1-687534) and by the María de Maeztu Seal of Excellence to IRI MDM-2016-0656., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

3D CNNs on distance matrices for human action recognition

  • Hernández Ruiz, Alejandro José
  • Porzi, Lorenzo
  • Rota Buló, Samuel
  • Moreno-Noguer, Francesc
Trabajo presentado a la ACM on Multimedia Conference, celebrada en California (USA) del 23 al 27 de octubre de 2017., In this paper we are interested in recognizing human actions from sequences of 3D skeleton data. For this purpose we combine a 3D Convolutional Neural Network with body representations based on Euclidean Distance Matrices (EDMs), which have been recently shown to be very effective to capture the geometric structure of the human pose. One inherent limitation of the EDMs, however, is that they are defined up to a permutation of the skeleton joints, i.e., randomly shuffling the ordering of the joints yields many different representations. In oder to address this issue we introduce a novel architecture that simultaneously, and in an end-to-end manner, learns an optimal transformation of the joints, while optimizing the rest of parameters of the convolutional network. The proposed approach achieves state-of-the-art results on 3 benchmarks, including the recent NTU RGB-D dataset, for which we improve on previous LSTM-based methods by more than 10 percentage points, also surpassing other CNN-based methods while using almost 1000 times fewer parameters., This work has been supported by the Spanish Ministry of Economy and Competitiveness through the ROBINSTRUCT project (TIN2014- 58178-R) and the María de Maeztu grant to IRI (MDM-2016-0656). Additionally, we gratefully acknowledge financial support from the DIGIMAP project, funded under grant #860375 by the Austrian Research Promotion Agency (FFG)., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Contribución de congreso (ConferenceOutput)

Multi-modal embedding for main product detection in fashion

  • Rubio Romano, Antonio
  • Yu, LongLong
  • Simo-Serra, Edgar
  • Moreno-Noguer, Francesc
Trabajo presentado a la International Conference on Computer Vision Workshops (ICCVW), celebrada en Venice (Italy) del 22 al 29 de octubre de 2017., We present an approach to detect the main product in fashion images by exploiting the textual metadata associated with each image. Our approach is based on a Convolutional Neural Network and learns a joint embedding of object proposals and textual metadata to predict the main product in the image. We additionally use several complementary classification and overlap losses in order to improve training stability and performance. Our tests on a large-scale dataset taken from eight e-commerce sites show that our approach outperforms strong baselines and is able to accurately detect the main product in a wide diversity of challenging fashion images., This work is partly funded by the Spanish MINECO project RobInstruct TIN2014-58178-R. A. Rubio is supported by the industrial doctorate grant 2015-DI-010 of the AGAUR., Peer Reviewed