Skip to main content

PROCESADO DE SEÑALES MULTIMODALES Y APRENDIZAJE AUTOMATICO EN GRAFOS

TEC2016-75976-R

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos I+D+i Retos de Investigación
Call year 2016
State Plan 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITECNICA DE CATALUNYA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 67
Encontrada(s) 1 página(s)

Repositorio Digital de la UPF
Publication Artículo científico (JournalArticle)

NeAT: a nonlinear analysis toolbox for neuroimaging

  • Casamitjana, Adrià
  • Vilaplana, Verónica
  • Puch, Santi
  • Aduriz, Asier
  • López, Carlos
  • Operto, Grégory
  • Cacciaglia, Raffaele
  • Falcón, Carles
  • Molinuevo, José Luis
  • Gispert, Juan Domingo
  • Alzheimer’s Disease Neuroimaging Initiative
NeAT is a modular, flexible and user-friendly neuroimaging analysis toolbox for modeling linear and nonlinear effects overcoming the limitations of the standard neuroimaging methods which are solely based on linear models. NeAT provides a wide range of statistical and machine learning non-linear methods for model estimation, several metrics based on curve fitting and complexity for model inference and a graphical user interface (GUI) for visualization of results. We illustrate its usefulness on two study cases where non-linear effects have been previously established. Firstly, we study the nonlinear effects of Alzheimer's disease on brain morphology (volume and cortical thickness). Secondly, we analyze the effect of the apolipoprotein APOE-ε4 genotype on brain aging and its interaction with age. NeAT is fully documented and publicly distributed at https://imatge-upc.github.io/neat-tool/., This work has been partially supported by the project MALEGRA TEC2016-75976-R financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF). Adrià Casamitjana is supported by the Spanish “Ministerio de Educación, Cultura y Deporte” FPU Research Fellowship. Juan D. Gispert holds a “‘Ramón y Cajal’” fellowship (RYC-2013-13054).
Repositorio Digital de la UPF
Publication Contribución de congreso (ConferenceOutput)

Towards large scale multimedia indexing: a case study on person discovery in broadcast news

  • Le, Nam
  • Bredin, Hervé
  • Sargent, Gabriel
  • India, Miquel
  • López Otero, Paula
  • Barras, Claude
  • Guinaudeau, Camille
  • Gravier, Guillaume
  • Barbosa da Fonseca, Gabriel
  • Lyon Freire, Izabela
  • Patrocínio, Zenilton
  • Guimarães, Silvio Jamil F.
  • Martí Juan, Gerard
  • Morros, Josep Ramon
  • Hernando, Javier
  • Docio Fernández, Laura
  • García Mateo, Carmen
  • Meignier, Sylvain
  • Odobez, Jean-Marc
Comunicació presentada a: the 15th International Workshop on Content-Based Multimedia Indexing (CBMI'17), celebrat a Florència, Itàlia, del 19 al 21 de juny de 2017, The rapid growth of multimedia databases and the human interest in their peers make indices representing the location and identity of people in audio-visual documents essential for searching archives. Person discovery in the absence of prior identity knowledge requires accurate association of audio-visual cues and detected names. To this end, we present 3 different strategies to approach this problem: clustering-based naming, verification-based naming, and graph-based naming. Each of these strategies utilizes different recent advances in unsupervised face / speech representation, verification, and optimization. To have a better understanding of the approaches, this paper also provides a quantitative and qualitative comparative study of these approaches using the associated corpus of the Person Discovery challenge at MediaEval 2016. From the results of our experiments, we can observe the pros and cons of each approach, thus paving the way for future promising research directions., This work was supported by the EU project EUMSSI (FP7-611057), ANR project MetaDaTV (ANR-14-CE24-0024) project, Camomile project (PCIN-2013-067), and the projects TEC2013-43935-R, TEC2015-69266-P, TEC2016-75976-R, TEC2015-65345-P financed by the Spanish government and ERDF.
Repositori Obert UdL
Publication Artículo científico (JournalArticle)

Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio López, Eduard
The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html. Dades primàries associades a l'article http://hdl.handle.net/10459.1/68505, This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR646), the Spanish Ministry of Economy and Competitiveness (project AGL2013-48297-C2-2-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). Part of the work was also developed within the framework of the project TEC2016-75976-R, financed by the Spanish Ministry of Economy, Industry and Competitiveness and the European Regional Development Fund (ERDF). The Spanish Ministry of Educationis thanked for Mr. J.Gené’s pre-doctoral fellowships (FPU15/03355). We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Ernesto Membrillo and Roberto Maturino for their support in dataset labelling.
Repositori Obert UdL
Publication Artículo científico (JournalArticle)

Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities

  • Gené Mola, Jordi
  • Vilaplana Besler, Verónica
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection., This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s predoctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Adria Carbó for his assistance in Faster R-CNN implementation.
Repositori Obert UdL
Publication Contribución de congreso (ConferenceOutput)

Apple size estimation using photogrammetry-derived 3D point clouds

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Escolà i Agustí, Alexandre
  • Gregorio López, Eduard
In-field fruit monitoring at different growth stages provides important information for farmers. Recent advances have focused on the detection and location of fruits, although the development of accurate fruit size estimation systems is still a challenge that requires further attention. The present work proposes a novel methodology for in-field measurement of spherical fruits using SfMMVS- based point clouds and considering different percentages of fruit occlusions., This work was partly funded by the Secretaria d’Universitats i Recerca de la Generalitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (projects AGL2013-48297-C2-2-R and TEC2016-75976-R) and the Spanish Ministry of Science, Innovation and Universities (grant RTI2018- 094222-B-I00[PAgFRUIT project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union).
Repositori Obert UdL
Publication Artículo científico (JournalArticle)

KFuji RGB-DS database: Fuji apple multi-modal images for fruit detection with color, depth and range-corrected IR data

  • Gené Mola, Jordi
  • Vilaplana Besler, Verónica
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
This article contains data related to the research article entitle 'Multi-modal Deep Learning for Fruit Detection Using RGB-D Cameras and their Radiometric Capabilities' [1]. The development of reliable fruit detection and localization systems is essential for future sustainable agronomic management of high-value crops. RGB-D sensors have shown potential for fruit detection and localization since they provide 3D information with color data. However, the lack of substantial datasets is a barrier for exploiting the use of these sensors. This article presents the KFuji RGBDS database which is composed by 967 multi-modal images of Fuji apples on trees captured using Microsoft Kinect v2 (Microsoft, Redmond, WA, USA). Each image contains information from 3 different modalities: color (RGB), depth (D) and range corrected IR intensity (S). Ground truth fruit locations were manually annotated, labeling a total of 12,839 apples in all the dataset. The current dataset is publicly available at http://www.grap.udl.cat/publicacions/datasets.html., This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017 SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition.
Repositori Obert UdL
Publication Artículo científico (JournalArticle)

Fuji-SfM dataset: A collection of annotated images and point clouds for Fuji apple detection and location using structure-from-motion photogrammetry

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio López, Eduard
The present dataset contains colour images acquired in a commercial Fuji apple orchard (Malus domestica Borkh. cv. Fuji) to reconstruct the 3D model of 11 trees by using structure-from-motion (SfM) photogrammetry. The data provided in this article is related to the research article entitled “Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry” [1]. The Fuji-SfM dataset includes: (1) a set of 288 colour images and the corresponding annotations (apples segmentation masks) for training instance segmentation neural networks such as Mask-RCNN; (2) a set of 582 images defining a motion sequence of the scene which was used to generate the 3D model of 11 Fuji apple trees containing 1455 apples by using SfM; (3) the 3D point cloud of the scanned scene with the corresponding apple positions ground truth in global coordinates. With that, this is the first dataset for fruit detection containing images acquired in a motion sequence to build the 3D model of the scanned trees with SfM and including the corresponding 2D and 3D apple location annotations. This data allows the development, training, and test of fruit detection algorithms either based on RGB images, on coloured point clouds or on the combination of both types of data. Dades primàries associades a l'article http://hdl.handle.net/10459.1/68505, This work was partly funded by the Secretaria d'Universitats i Recerca del Departament d'Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (project AGL2013-48297-C2-2-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). Part of the work was also developed within the framework of the project TEC2016-75976-R, financed by the Spanish Ministry of Economy, Industry and Competitiveness and the European Regional Development Fund (ERDF). The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355).
Repositori Obert UdL
Publication Contribución de congreso (ConferenceOutput)

Fruit detection and 3D location using instance segmentation neural networks and SfM photogrammetry

  • Gregorio López, Eduard
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio López, Eduard
The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the effi-ciency of agriculture management. Most of the current fruit detec-tion systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location, combining the use of instance segmentation neu-ral networks and Structure-from-Motion (SfM) photogrammetry., This work was partly funded by the Secretaria d’Universitats i Recerca de la Gener-alitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (projects AGL2013-48297-C2-2-R and TEC2016-75976-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355).
Repositori Obert UdL
Publication Artículo científico (JournalArticle)

Fruit detection in an apple orchard using a mobile terrestrial laser scanner

  • Gené Mola, Jordi
  • Gregorio López, Eduard
  • Guevara, Javier
  • Auat Cheein, Fernando
  • Sanz Cortiella, Ricardo
  • Escolà i Agustí, Alexandre
  • Llorens Calveras, Jordi
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Rosell Polo, Joan Ramon
The development of reliable fruit detection and localization systems provides an opportunity to improve the crop value and management by limiting fruit spoilage and optimised harvesting practices. Most proposed systems for fruit detection are based on RGB cameras and thus are affected by intrinsic constraints, such as variable lighting conditions. This work presents a new technique that uses a mobile terrestrial laser scanner (MTLS) to detect and localise Fuji apples. An experimental test focused on Fuji apple trees (Malus domestica Borkh. cv. Fuji) was carried out. A 3D point cloud of the scene was generated using an MTLS composed of a Velodyne VLP-16 LiDAR sensor synchronised with an RTK-GNSS satellite navigation receiver. A reflectance analysis of tree elements was performed, obtaining mean apparent reflectance values of 28.9%, 29.1%, and 44.3% for leaves, branches and trunks, and apples, respectively. These results suggest that the apparent reflectance parameter (at 905 nm wavelength) can be useful to detect apples. For that purpose, a fourstep fruit detection algorithm was developed. By applying this algorithm, a localization success of 87.5%, an identification success of 82.4%, and an F1-score of 0.858 were obtained in relation to the total amount of fruits. These detection rates are similar to those obtained by RGB-based systems, but with the additional advantages of providing direct 3D fruit location information, which is not affected by sunlight variations. From the experimental results, it can be concluded that LiDAR-based technology and, particularly, its reflectance information, has potential for remote apple detection and 3D location., This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (projects AGL2013-48297-C2-2-Rand MALEGRA, TEC2016-75976-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355). The work of Jordi Llorens was supported by Spanish Ministry of Economy, Industry and Competitiveness through a postdoctoral position named Juan de la Cierva Incorporación (JDCI-2016-29464_N18003). We would also like to thank CONICYT/FONDECYT for grant 1171431 and CONICYT FB0008. Nufri (especially Santiago Salamero and Oriol Morreres) and Vicens Maquinària Agrícola S.A. are also thanked for their support during the data acquisition.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Contribución de congreso (ConferenceOutput)

How2Sign: A large-scale multimodal dataset for continuous American sign language

  • Duarte, Amanda
  • Palaskar, Shruti
  • Ventura, Lucas
  • Ghadiyaram, Deepti
  • DeHaan, Kenneth
  • Metze, Florian
  • Torres, Jordi
  • Giró i Nieto, Xavier
Trabajo presentado en la IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), celebrada de forma virtual del 19 al 25 de junio de 2021, One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth. A three-hour subset was further recorded in the Panoptic studio enabling detailed 3D pose estimation. To evaluate the potential of How2Sign for real-world impact, we conduct a study with ASL signers and show that synthesized videos using our dataset can indeed be understood. The study further gives insights on challenges that computer vision should address in order to make progress in this field. Dataset website: http://how2sign.github.io/, This work received funding from Facebook through gifts to CMU and UPC; through projects TEC2016-75976-R, TIN2015- 65316-P, SEV-2015-0493 and PID2019-107255GB-C22 of the Spanish Government and 2017-SGR-1414 of Generalitat de Catalunya. This work used XSEDE’s “Bridges” system at the Pittsburgh Supercomputing Center (NSF award ACI1445606). Amanda Duarte has received support from la Caixa Foundation (ID 100010434) under the fellowship code LCF/BQ/IN18/11660029. Shruti Palaskar was supported by the Facebook Fellowship program.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Multiresolution co-clustering for uncalibrated multiview segmentation

  • Ventura, Carles
  • Varas, David
  • Vilaplana Besler, Verónica
  • Giró Nieto, Xavier
  • Marqués Acosta, Fernando
We propose a technique for coherently co-clustering uncalibrated views of a scene with a contour-based representation. Our work extends the previous framework, an iterative algorithm for segmenting sequences with small variations, where the partition solution space is too restrictive for scenarios where consecutive images present larger variations. To deal with a more flexible scenario, we present three main contributions. First, motion information has been considered both for region adjacency and region similarity. Second, a two-step iterative architecture is proposed to increase the partition solution space. Third, a feasible global optimization that allows to jointly process all the views has been implemented. In addition to the previous contributions, which are based on low-level features, we have also considered introducing higher level features as semantic information in the co-clustering algorithm. We evaluate these techniques on multiview and temporal datasets, showing that they outperform state-of-the-art approaches., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Refinement network for unsupervised on the scene foreground segmentation

  • Pardàs Feliu, Montse
  • Canet Tarrés, Gemma
Unsupervised learning represents one of the most interesting challenges in computer vision today. The task has an immense practical value with many applications in artificial intelligence and emerging technologies, as large quantities of unlabeled images and videos can be collected at low cost. In this paper, we address the unsupervised learning problem in the context of segmenting the main foreground objects in single images. We propose an unsupervised learning system, which has two pathways, the teacher and the student, respectively. The system is designed to learn over several generations of teachers and students. At every generation the teacher performs unsupervised object discovery in videos or collections of images and an automatic selection module picks up good frame segmentations and passes them to the student pathway for training. At every generation multiple students are trained, with different deep network architectures to ensure a better diversity. The students at one iteration help in training a better selection module, forming together a more powerful teacher pathway at the next iteration. In experiments, we show that the improvement in the selection power, the training of multiple students and the increase in unlabeled data significantly improve segmentation accuracy from one generation to the next. Our method achieves top results on three current datasets for object discovery in video, unsupervised image segmentation and saliency detection. At test time, the proposed system is fast, being one to two orders of magnitude faster than published unsupervised methods. We also test the strength of our unsupervised features within a well known transfer learning setup and achieve competitive performance, proving that our unsupervised approach can be reliably used in a variety of computer vision tasks., During the development of this work the first author was a visitor at TOSHIBA Cambridge Research Lab. This work has been carried out with the support of this lab and project TEC2016-75976-R, by the Ministerio de Economia, Industria y Competitividad and the European Regional Development Fund., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Detection-aided liver lesion segmentation using deep learning

  • Bellver, Míriam
  • Maninis, Kevis-Kokitsi
  • Pont Tuset, Jordi
  • Giró Nieto, Xavier
  • Torres Viñals, Jordi
  • Van Gool, Luc
A fully automatic technique for segmenting the liver and localizing its unhealthy tissues is a convenient tool in order to diagnose hepatic diseases and assess the response to the according treatments. In this work we propose a method to segment the liver and its lesions from Computed Tomography (CT) scans using Convolutional Neural Networks (CNNs), that have proven good results in a variety of computer vision tasks, including medical imaging. The network that segments the lesions consists of a cascaded architecture, which first focuses on the region of the liver in order to segment the lesions on it. Moreover, we train a detector to localize the lesions, and mask the results of the segmentation network with the positive detections. The segmentation architecture is based on DRIU, a Fully Convolutional Network (FCN) with side outputs that work on feature maps of different resolutions, to finally benefit from the multi-scale information learned by different stages of the network. The main contribution of this work is the use of a detector to localize the lesions, which we show to be beneficial to remove false positives triggered by the segmentation network., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Shared latent structures between imaging features and biomarkers in early stages of Alzheimer's disease: a predictive study

  • Casamitjana Díaz, Adrià
  • Petrone, Paula
  • Molinuevo, José Luis
  • Gispert, Juan Domingo
  • Vilaplana Besler, Verónica
© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Magnetic resonance imaging (MRI) provides high resolution brain morphological information and is used as a biomarker in neurodegenerative diseases. Population studies of brain morphology often seek to identify pathological structural changes related to different diagnostic categories (e.g: controls, mild cognitive impairment or dementia) which normally describe highly heterogeneous groups with a single categorical variable. Instead, multiple biomarkers are used as a proxy for pathology and are more powerful in capturing structural variability. Hence, using the joint modeling of brain morphology and biomarkers, we aim at describing structural changes related to any brain condition by means of few underlying processes. In this regard, we use a multivariate approach based on Projection to Latent Structures in its regression variant (PLSR) to study structural changes related to aging and AD pathology. MRI volumetric and cortical thickness measurements are used for brain morphology and cerebrospinal fluid (CSF) biomarkers (t-tau, p-tau and amyloid-beta) are used as a proxy for AD pathology. By relating both sets of measurements, PLSR finds a low-dimensional latent space describing AD pathological effects on brain structure. The proposed framework allows to separately model aging effects on brain morphology as a confounder variable orthogonal to the pathological effect. The predictive power of the associated latent spaces (i.e. the capacity of predicting biomarker values) is assessed in a cross-validation framework., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Comparing fixed and adaptive computation time for recurrent neural networks

  • Fojo, Daniel
  • Campos Camunez, Victor
  • Giró Nieto, Xavier
Deep networks commonly perform better than shallow ones, but allocating the proper amount of computation for each particular input sample remains an open problem. This issue is particularly challenging in sequential tasks, where the required complexity may vary for different tokens in the input sequence. Adaptive Computation Time (ACT) was proposed as a method for dynamically adapting the computation at each step for Recurrent Neural Networks (RNN). ACT introduces two main modifications to the regular RNN formulation: (1) more than one RNN steps may be executed between an input sample is fed to the layer and and this layer generates an output, and (2) this number of steps is dynamically predicted depending on the input token and the hidden state of the network. In our work, we aim at gaining intuition about the contribution of these two factors to the overall performance boost observed when augmenting RNNs with ACT. We design a new baseline, Repeat-RNN, which performs a constant number of RNN state updates larger than one before generating an output. Surprisingly, such uniform distribution of the computational resources matches the performance of ACT in the studied tasks. We hope that this finding motivates new research efforts towards designing RNN architectures that are able to dynamically allocate computational resources. TL;DR: Comparison of Adaptive Computation Time with Fixed computation time for RNN's gives surprising results, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

One shot learning for generic instance segmentation in RGBD videos

  • Lin, Xiao
  • Casas Pla, Josep Ramon
  • Pardàs Feliu, Montse
Hand-crafted features employed in classical generic instance segmentation methods have limited discriminative power to distinguish different objects in the scene, while Convolutional Neural Networks (CNNs) based semantic segmentation is restricted to predefined semantics and not aware of object instances. In this paper, we combine the advantages of the two methodologies and apply the combined approach to solve a generic instance segmentation problem in RGBD video sequences. In practice, a classical generic instance segmentation method is employed to initially detect object instances and build temporal correspondences, whereas instance models are trained based on the few detected instance samples via CNNs to generate robust features for instance segmentation. We exploit the idea of one shot learning to deal with the small training sample size problem when training CNNs. Experiment results illustrate the promising performance of the proposed approach., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Cost-effective active learning for melanoma segmentation

  • Górriz, Marc
  • Giró Nieto, Xavier
  • Carlier, Axel
  • Faure, Emmanuel
We propose a novel Active Learning framework capable to train effectively a convolutional neural network for semantic segmentation of medical imaging, with a limited amount of training labeled data. Our contribution is a practical Cost-Effective Active Learning approach using dropout at test time as Monte Carlo sampling to model the pixel-wise uncertainty and to analyze the image information to improve the training performance., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Ship detection in SAR images based on Maxtree representation and graph signal processing

  • Salembier Clairon, Philippe Jean
  • Liesegang Maria, Sergi
  • López Martínez, Carlos
© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., This paper discusses an image processing architecture and tools to address the problem of ship detection in synthetic-aperture radar images. The detection strategy relies on a tree-based representation of images, here a Maxtree, and graph signal processing tools. Radiometric as well as geometric attributes are evaluated and associated with the Maxtree nodes. They form graph attribute signals which are processed with graph filters. The goal of this filtering step is to exploit the correlation existing between attribute values on neighboring tree nodes. Considering that trees are specific graphs where the connectivity toward ancestors and descendants may have a different meaning, we analyze several linear, nonlinear, and morphological filtering strategies. Beside graph filters, two new filtering notions emerge from this analysis: tree and branch filters. Finally, we discuss a ship detection architecture that involves graph signal filters and machine learning tools. This architecture demonstrates the interest of applying graph signal processing tools on the tree-based representation of images and of going beyond classical graph filters. The resulting approach significantly outperforms state-of-the-art algorithms. Finally, a MATLAB toolbox allowing users to experiment with the tools discussed in this paper on Maxtree or Mintree has been created and made public., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

SaltiNet: scan-path prediction on 360 degree images using saliency volumes

  • Assens, Marc
  • Giró Nieto, Xavier
  • McGuinness, Kevin
  • O'Connor, Noel
We introduce SaltiNet, a deep neural network for scan-path prediction trained on 360-degree images. The model is based on a temporal-aware novel representation of saliency information named the saliency volume. The first part of the network consists of a model trained to generate saliency volumes, whose parameters are fit by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency volumes. Sampling strategies over these volumes are used to generate scan-paths over the 360-degree images. Our experiments show the advantages of using saliency volumes, and how they can be used for related tasks. Our source code and trained models available at https://github.com/massens/saliency-360salient-2017., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Standardized assessment of automatic segmentation of white matter hyperintensities; results of the WMH segmentation challenge

  • Kuijf, Hugo J.
  • Biesbroek, J. Matthijs
  • De Bresser, Jeroen
  • Casamitjana Díaz, Adrià
  • Vilaplana Besler, Verónica
© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Quantification of white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Advanced measurements are obtained from manual segmentations on brain MR images, which is a laborious procedure. Automatic WMH segmentation methods exist, but a standardized comparison of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their method on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge (http://wmh.isi.uu.nl/). Sixty T1+FLAIR images from three MR scanners were released with manual WMH segmentations. A secret test set of 110 images from five MR scanners was used for evaluation. Methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: (1) Dice Similarity Coefficient, (2) modified Hausdorff distance (95th percentile), (3) absolute percentage volume difference, (4) sensitivity for detecting individual lesions, and (5) F1-score for individual lesions. Additionally, methods were ranked on their inter-scanner robustness. Twenty participants submitted their method for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods. There is one clear winner, which also has the best inter-scanner robustness. The challenge remains open for future submissions and provides a public platform for method evaluation., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Fruit detection and 3D location using instance segmentation neural networks and SfM photogrammetry

  • Gregorio López, Eduard
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio López, Eduard
The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the effi-ciency of agriculture management. Most of the current fruit detec-tion systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location, combining the use of instance segmentation neu-ral networks and Structure-from-Motion (SfM) photogrammetry., This work was partly funded by the Secretaria d’Universitats i Recerca de la Gener-alitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (projects AGL2013-48297-C2-2-R and TEC2016-75976-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355).
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Fruit detection in an apple orchard using a mobile terrestrial laser scanner

  • Gené Mola, Jordi
  • Gregorio López, Eduard
  • Guevara, Javier
  • Auat Cheein, Fernando
  • Sanz Cortiella, Ricardo
  • Escolà i Agustí, Alexandre
  • Llorens Calveras, Jordi
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Rosell Polo, Joan Ramon
The development of reliable fruit detection and localization systems provides an opportunity to improve the crop value and management by limiting fruit spoilage and optimised harvesting practices. Most proposed systems for fruit detection are based on RGB cameras and thus are affected by intrinsic constraints, such as variable lighting conditions. This work presents a new technique that uses a mobile terrestrial laser scanner (MTLS) to detect and localise Fuji apples. An experimental test focused on Fuji apple trees (Malus domestica Borkh. cv. Fuji) was carried out. A 3D point cloud of the scene was generated using an MTLS composed of a Velodyne VLP-16 LiDAR sensor synchronised with an RTK-GNSS satellite navigation receiver. A reflectance analysis of tree elements was performed, obtaining mean apparent reflectance values of 28.9%, 29.1%, and 44.3% for leaves, branches and trunks, and apples, respectively. These results suggest that the apparent reflectance parameter (at 905 nm wavelength) can be useful to detect apples. For that purpose, a fourstep fruit detection algorithm was developed. By applying this algorithm, a localization success of 87.5%, an identification success of 82.4%, and an F1-score of 0.858 were obtained in relation to the total amount of fruits. These detection rates are similar to those obtained by RGB-based systems, but with the additional advantages of providing direct 3D fruit location information, which is not affected by sunlight variations. From the experimental results, it can be concluded that LiDAR-based technology and, particularly, its reflectance information, has potential for remote apple detection and 3D location., This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (projects AGL2013-48297-C2-2-Rand MALEGRA, TEC2016-75976-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355). The work of Jordi Llorens was supported by Spanish Ministry of Economy, Industry and Competitiveness through a postdoctoral position named Juan de la Cierva Incorporación (JDCI-2016-29464_N18003). We would also like to thank CONICYT/FONDECYT for grant 1171431 and CONICYT FB0008. Nufri (especially Santiago Salamero and Oriol Morreres) and Vicens Maquinària Agrícola S.A. are also thanked for their support during the data acquisition.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Digitally stained confocal microscopy through deep learning

  • Combalia Escudero, Marc
  • Pérez Ankar, Javiera
  • García Herrera, Adriana
  • Alos, Llúcia
  • Vilaplana Besler, Verónica
  • Marqués Acosta, Fernando
  • Puig, Susana
  • Malvehy, Josep
Specialists have used confocal microscopy in the ex-vivo modality to identify Basal Cell Carcinoma tumors with an overall sensitivity of 96.6% and specificity of 89.2% (Chung et al., 2004). However, this technology hasn’t established yet in the standard clinical practice because most pathologists lack the knowledge to interpret its output. In this paper we propose a combination of deep learning and computer vision techniques to digitally stain confocal microscopy images into H&E-like slides, enabling pathologists to interpret these images without specific training. We use a fully convolutional neural network with a multiplicative residual connection to denoise the confocal microscopy images, and then stain them using a Cycle Consistency Generative Adversarial Network, Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Assessing knee OA severity with CNN attention-based end-to-end architectures

  • Górriz, Marc
  • Antony, Joseph
  • McGuinness, Kevin
  • Giró Nieto, Xavier
  • O'Connor, Noel
This work proposes a novel end-to-end convolutional neural network (CNN) architecture to automatically quantify the severity of knee osteoarthritis (OA) using X-Ray images, which incorporates trainable attention modules acting as unsupervised fine-grained detectors of the region of interest (ROI). The proposed attention modules can be applied at different levels and scales across any CNN pipeline helping the network to learn relevant attention patterns over the most informative parts of the image at different resolutions. We test the proposed attention mechanism on existing state-of-the-art CNN architectures as our base models, achieving promising results on the benchmark knee OA datasets from the osteoarthritis initiative (OAI) and multicenter osteoarthritis study (MOST)., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Wav2Pix: speech-conditioned face generation using generative adversarial networks

  • Cardoso Duarte, Amanda
  • Roldan, Francisco
  • Tubau, Miquel
  • Escur, Janna
  • Pascual de la Puente, Santiago
  • Salvador Aguilera, Amaia
  • Mohedano, Eva
  • McGuinness, Kevin
  • Torres Viñals, Jordi
  • Giró Nieto, Xavier
Speech is a rich biometric signal that contains information about the identity, gender and emotional state of the speaker. In this work, we explore its potential to generate face images of a speaker by conditioning a Generative Adversarial Network (GAN) with raw speech input. We propose a deep neural network that is trained from scratch in an end-to-end fashion, generating a face directly from the raw speech waveform without any additional identity information (e.g reference image or one-hot encoding). Our model is trained in a self-supervised approach by exploiting the audio and visual signals naturally aligned in videos. With the purpose of training from video data, we present a novel dataset collected for this work, with high-quality videos of youtubers with notable expressiveness in both the speech and visual signals., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Hierarchical stack filtering: a bitplane-based algorithm for massively parallel processors

  • Frías Velazquez, Andres
  • Morros Rubió, Josep Ramon
  • García Molina, Mario
  • Philips, Wilfried
With the development of novel parallel architectures for image processing, the implementation of well-known image operators needs to be reformulated to take advantage of the so-called massive parallelism. In this work, we propose a general algorithm that implements a large class of nonlinear filters, called stack filters, with a 2D-array processor. The proposed method consists of decomposing an image into bitplanes with the bitwise decomposition and then processes every bitplane hierarchically. The filtered image is reconstructed by simply stacking the filtered bitplanes according to their order of significance. Owing to its hierarchical structure, our algorithm allows us to trade-off between image quality and processing time, and to significantly reduce the computation time of low-entropy images. Also, experimental tests show that the processing time of our method is substantially lower than that of classical methods when using large structuring elements. All these features are of interest to a variety of real-time applications based on morphological operations such as video segmentation and video enhancement., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Uso de redes neuronales convolucionales para la detección remota de frutos con cámaras RGB-D

  • Gené Mola, Jordi
  • Vilaplana Besler, Verónica
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
La detección remota de frutos será una herramienta indispensable para la gestión agronómica optimizada y sostenible de las plantaciones frutícolas del futuro, con aplicaciones en previsión de cosecha, robotización de la recolección y elaboración de mapas de producción. Este trabajo propone el uso de cámaras de profundidad RGB-D para la detección y la posterior localización 3D de los frutos. El material utilizado para la adquisición de datos consiste en una plataforma terrestre autopropulsada equipada con dos sensores Kinect v2 de Microsoft y un sistema de posicionamiento RTK-GNSS. Con este equipo se escanearon 3 filas de manzanos Fuji de una explotación comercial. El conjunto de datos adquiridos está compuesto por 110 capturas que contienen un total de 12,838 manzanas Fuji. La detección de frutos se realizó mediante los datos RGB (imágenes de color proporcionadas por el sensor). Para ello, se implementó y se entrenó la red neuronal convolucional de detección de objetos Faster R-CNN, la cual está compuesta por dos módulos: red de propuesta de regiones de interés y red de clasificación. Ambos módulos comparten las primeras capas convolucionales siguiendo el modelo VGG-16 pre-entrenado con la base de datos ImageNet. Los resultados de test muestran un porcentaje de detección del 91.4% de los frutos con un 15.9% de falsos positivos (F1-score = 0.876). La evaluación cualitativa de las detecciones muestra que los falsos positivos corresponden a zonas de la imagen que presentan un patrón muy similar a una manzana, donde, incluso a percepción del ojo humano, es difícil de determinar si hay o no manzana. Por otro lado, las manzanas no detectadas corresponden a aquellas que estaban ocultas casi en su totalidad por otros órganos vegetativos (hojas o ramas) o a manzanas cortadas por los márgenes de la imagen. De los resultados experimentales se concluye que el sensor Kinect v2 tiene un gran potencial para la detección y localización 3D de frutos. La principal limitación del sistema es que el rendimiento del sensor de profundidad se ve afectado en condiciones de alta iluminación., Este trabajo ha sido parcialmente financiado por la Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, el Ministerio de Economía y Competitividad de España, y the European Regional Development Fund (ERDF), mediante los proyectos 2017 SGR 646, AGL2013-48297-C2-2-R, MALEGRA TEC2016-75976-R. Se agradece al Ministerio de Educación de España por la beca pre-doctoral de J. Gené (FPU15/03355)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

3D hierarchical optimization for multi-view depth map coding

  • Maceira, Marc
  • Varas, David
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Marqués Acosta, Fernando
Depth data has a widespread use since the popularity of high resolution 3D sensors. In multi-view sequences, depth information is used to supplement the color data of each view. This article proposes a joint encoding of multiple depth maps with a unique representation. Color and depth images of each view are segmented independently and combined in an optimal Rate-Distortion fashion. The resulting partitions are projected to a reference view where a coherent hierarchy for the multiple views is built. A Rate-Distortion optimization is applied to obtain the nal segmentation choosing nodes of the hierarchy. The consistent segmentation is used to robustly encode depth maps of multiple views obtaining competitive results with HEVC coding standards., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Comparative study of upsampling methods for super-resolution in remote sensing

  • Salgueiro Romero, Luis Fernando
  • Marcello Ruiz, Javier
  • Vilaplana Besler, Verónica
Many remote sensing applications require high spatial resolution images, but the elevated cost of these images makes some studies unfeasible. Single-image super-resolution algorithms can improve the spatial resolution of a lowresolution image by recovering feature details learned from pairs of low-high resolution images. In this work, several configurations of ESRGAN, a state-of-the-art algorithm for image super-resolution, are tested. We make a comparison between several scenarios, with different modes of upsampling and channels involved. The best results are obtained training a model with RGB-IR channels and using progressive upsampling., This work has been partially supported by the ARTEMISAT-2 (CTM2016-77733-R) and MALEGRA TEC2016-75976-R projects, funded by the Spanish AEI, FEDER funds,and by the Spanish Ministerio de Economía y Competitividad, respectively. L.S.R. would like to acknowledge the BECAL (Becas Carlos Antonio López) scholarship for the financial support., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Demonstration of an open source framework for qualitative evaluation of CBIR systems

  • Gomez Duran, Paula
  • Mohedano, Eva
  • McGuinness, Kevin
  • Giró Nieto, Xavier
  • O'Connor, Noel
Evaluating image retrieval systems in a quantitative way, for example by computing measures like mean average precision, allows for objective comparisons with a ground-truth. However, in cases where ground-truth is not available, the only alternative is to collect feedback from a user. Thus, qualitative assessments become important to better understand how the system works. Visualizing the results could be, in some scenarios, the only way to evaluate the results obtained and also the only opportunity to identify that a system is failing. This necessitates developing a User Interface (UI) for a Content Based Image Retrieval (CBIR) system that allows visualization of results and improvement via capturing user relevance feedback. A well-designed UI facilitates understanding of the performance of the system, both in cases where it works well and perhaps more importantly those which highlight the need for improvement. Our open-source system implements three components to facilitate researchers to quickly develop these capabilities for their retrieval engine. We present: a web-based user interface to visualize retrieval results and collect user annotations; a server that simplifies connection with any underlying CBIR system; and a server that manages the search engine data., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

UPC multimodal speaker diarization system for the 2018 Albayzin challenge

  • India Massana, Miquel Àngel
  • Sagastiberri, Itziar
  • Palau Puigdevall, Ponç
  • Sayrol Clols, Elisa
  • Morros Rubió, Josep Ramon
  • Hernando Pericás, Francisco Javier
This paper presents the UPC system proposed for the Multimodal Speaker Diarization task of the 2018 Albayzin Challenge. This approach works by processing individually the speech and the image signal. In the speech domain, speaker diarization is performed using identity embeddings created by a triplet loss DNN that uses i-vectors as input. The triplet DNN is trained with an additional regularization loss that minimizes the variance of both positive and negative distances. A sliding windows is then used to compare speech segments with enrollment speaker targets using cosine distance between the embeddings. To detect identities from the face modality, a face detector followed by a face tracker has been used on the videos. For each cropped face a feature vector is obtained using a Deep Neural Network based on the ResNet 34 architecture, trained using a metric learning triplet loss (available from dlib library). For each track the face feature vector is obtained by averaging the features obtained for each one of the frames of that track. Then, this feature vector is compared with the features extracted from the images of the enrollment identities. The proposed system is evaluated on the RTVE2018 database., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

How2Sign: A large-scale multimodal dataset for continuous American sign language

  • Cardoso Duarte, Amanda
  • Palaskar, Shruti
  • Ventura Ripol, Lucas
  • Ghadiyaram, Deepti
  • DeHaan, Kenneth
  • Metze, Florian
  • Torres Viñals, Jordi
  • Giró Nieto, Xavier
One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth. A three-hour subset was further recorded in the Panoptic studio enabling detailed 3D pose estimation. To evaluate the potential of How2Sign for real-world impact, we conduct a study with ASL signers and show that synthesized videos using our dataset can indeed be understood. The study further gives insights on challenges that computer vision should address in order to make progress in this field. Dataset website: http://how2sign.github.io/, This work received funding from Facebook through gifts to CMU and UPC; through projects TEC2016-75976-R, TIN2015- 65316-P, SEV-2015-0493 and PID2019-107255GB-C22 of the Spanish Government and 2017-SGR-1414 of Generalitat de Catalunya. This work used XSEDE’s “Bridges” system at the Pittsburgh Supercomputing Center (NSF award ACI- 1445606). Amanda Duarte has received support from la Caixa Foundation (ID 100010434) under the fellowship code LCF/BQ/IN18/11660029. Shruti Palaskar was supported by the Facebook Fellowship program., Peer Reviewed, Objectius de Desenvolupament Sostenible::10 - Reducció de les Desigualtats, Objectius de Desenvolupament Sostenible::4 - Educació de Qualitat::4.5 - Per a 2030, eliminar les disparitats de gènere en l’educació i garantir l’accés en condicions d’igualtat a les persones vulnerables, incloses les persones amb discapacitat, els pobles indígenes i els nens i nenes en situacions de vulnerabilitat, a tots els nivells de l’ensenyament i la formació professional, Objectius de Desenvolupament Sostenible::10 - Reducció de les Desigualtats::10.2 - Per a 2030, potenciar i promoure la inclusió social, econòmica i política de totes les persones, independentment de l’edat, sexe, discapacitat, raça, ètnia, origen, religió, situació econòmica o altra condició, Objectius de Desenvolupament Sostenible::4 - Educació de Qualitat, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio, Eduard
The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Hyperparameter-free losses for model-based monocular reconstruction

  • Ramon Maldonado, Eduard
  • Ruiz, Guillermo
  • Batard, Thomas
  • Giró Nieto, Xavier
This work proposes novel hyperparameter-free losses for single view 3D reconstruction with morphable models (3DMM). We dispense with the hyperparameters used in other works by exploiting geometry, so that the shape of the object and the camera pose are jointly optimized in a sole term expression. This simplification reduces the optimization time and its complexity. Moreover, we propose a novel implicit regularization technique based on random virtual projections that does not require additional 2D or 3D annotations. Our experiments suggest that minimizing a shape reprojection error together with the proposed implicit regularization is especially suitable for applications that require precise alignment between geometry and image spaces, such as augmented reality. We evaluate our losses on a large scale dataset with 3D ground truth and publish our implementations to facilitate reproducibility and public benchmarking in this field., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Inverse cooking: recipe generation from food images

  • Salvador Aguilera, Amaia
  • Drozdzal, Michal
  • Giró Nieto, Xavier
  • Romero, Adriana
People enjoy food photography because they appreciate food. Behind each meal there is a story described in a complex recipe and, unfortunately, by simply looking at a food image we do not have access to its preparation process. Therefore, in this paper we introduce an inverse cooking system that recreates cooking recipes given food images. Our system predicts ingredients as sets by means of a novel architecture, modeling their dependencies without imposing any order, and then generates cooking instructions by attending to both image and its inferred ingredients simultaneously. We extensively evaluate the whole system on the large-scale Recipe1M dataset and show that (1) we improve performance w.r.t. previous baselines for ingredient prediction; (2) we are able to obtain high quality recipes by leveraging both image and ingredients; (3) our system is able to produce more compelling recipes than retrieval-based approaches according to human judgment. We make code and models publicly available., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Scanpath and saliency prediction on 360 degree images

  • Assens Reina, Marc
  • Giró Nieto, Xavier
  • McGuinness, Kevin
  • O'Connor, Noel
We introduce deep neural networks for scanpath and saliency prediction trained on 360-degree images. The scanpath prediction model called SaltiNet is based on a temporal-aware novel representation of saliency information named the saliency volume. The first part of the network consists of a model trained to generate saliency volumes, whose parameters are fit by back-propagation using a binary cross entropy (BCE) loss over downsampled versions of the saliency volumes. Sampling strategies over these volumes are used to generate scanpaths over the 360-degree images. Our experiments show the advantages of using saliency volumes, and how they can be used for related tasks. We also show how a similar architecture achieves state-of-the-art performance for the related task of saliency map prediction. Our source code and trained models available at https://github.com/massens/saliency-360salient-2017., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Fuji-SfM dataset: a collection of annotated images and point clouds for Fuji apple detection and location using structure-from-motion photogrammetry

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio, Eduard
The present dataset contains colour images acquired in a commercial Fuji apple orchard (Malus domestica Borkh. cv. Fuji) to reconstruct the 3D model of 11 trees by using structure-from-motion (SfM) photogrammetry. The data provided in this article is related to the research article entitled “Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry” [1]. The Fuji-SfM dataset includes: (1) a set of 288 colour images and the corresponding annotations (apples segmentation masks) for training instance segmentation neural networks such as Mask-RCNN; (2) a set of 582 images defining a motion sequence of the scene which was used to generate the 3D model of 11 Fuji apple trees containing 1455 apples by using SfM; (3) the 3D point cloud of the scanned scene with the corresponding apple positions ground truth in global coordinates. With that, this is the first dataset for fruit detection containing images acquired in a motion sequence to build the 3D model of the scanned trees with SfM and including the corresponding 2D and 3D apple location annotations. This data allows the development, training, and test of fruit detection algorithms either based on RGB images, on coloured point clouds or on the combination of both types of data., This work was partly funded by the Secretaria d'Universitats i Recerca del Departament d'Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (project AGL2013-48297-C2-2-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). Part of the work was also developed within the framework of the project TEC2016-75976-R, financed by the Spanish Ministry of Economy, Industry and Competitiveness and the European Regional Development Fund (ERDF). The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355). We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Ernesto Membrillo and Roberto Maturino for their support in dataset labelling., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Picking groups instead of samples: a close look at Static Pool-based Meta-Active Learning

  • Mas Méndez, Ignasi
  • Morros Rubió, Josep Ramon
  • Vilaplana Besler, Verónica
©2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Active Learning techniques are used to tackle learning problems where obtaining training labels is costly. In this work we use Meta-Active Learning to learn to select a subset of samples from a pool of unsupervised input for further annotation. This scenario is called Static Pool-based Meta-Active Learning. We propose to extend existing approaches by performing the selection in a manner that, unlike previous works, can handle the selection of each sample based on the whole selected subset., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Super-resolution of Sensinel-2 imagery using generative adversarial networks

  • Salgueiro Romero, Luis Fernando
  • Marcello, Javier
  • Vilaplana Besler, Verónica
Sentinel-2 satellites provide multi-spectral optical remote sensing images with four bands at 10 m of spatial resolution. These images, due to the open data distribution policy, are becoming an important resource for several applications. However, for small scale studies, the spatial detail of these images might not be sufficient. On the other hand, WorldView commercial satellites offer multi-spectral images with a very high spatial resolution, typically less than 2 m, but their use can be impractical for large areas or multi-temporal analysis due to their high cost. To exploit the free availability of Sentinel imagery, it is worth considering deep learning techniques for single-image super-resolution tasks, allowing the spatial enhancement of low-resolution (LR) images by recovering high-frequency details to produce high-resolution (HR) super-resolved images. In this work, we implement and train a model based on the Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) with pairs of WorldView-Sentinel images to generate a super-resolved multispectral Sentinel-2 output with a scaling factor of 5. Our model, named RS-ESRGAN, removes the upsampling layers of the network to make it feasible to train with co-registered remote sensing images. Results obtained outperform state-of-the-art models using standard metrics like PSNR, SSIM, ERGAS, SAM and CC. Moreover, qualitative visual analysis shows spatial improvements as well as the preservation of the spectral information, allowing the super-resolved Sentinel-2 imagery to be used in studies requiring very high spatial resolution, This research has been supported by the ARTEMISAT-2 (CTM2016-77733-R) and MALEGRA (TEC2016-75976-R) projects, funded by the Spanish Agencia Estatal de Investigación (AEI), by the Fondo Europeo de Desarrollo Regional (FEDER) and the Spanish Ministerio de Economía y Competitividad, respectively. L.S. would like to acknowledge the BECAL (Becas Carlos Antonio López) scholarship for the financial support., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication

MRI brain tumor segmentation and uncertainty estimation using 3D-UNet architectures

  • Mora Ballestar, Laura
  • Vilaplana Besler, Verónica
Automation of brain tumor segmentation in 3D magnetic resonance images (MRIs) is key to assess the diagnostic and treatment of the disease. In recent years, convolutional neural networks (CNNs) have shown improved results in the task. However, high memory consumption is still a problem in 3D-CNNs. Moreover, most methods do not include uncertainty information, which is especially critical in medical diagnosis. This work studies 3D encoder-decoder architectures trained with patch-based techniques to reduce memory consumption and decrease the effect of unbalanced data. The different trained models are then used to create an ensemble that leverages the properties of each model, thus increasing the performance. We also introduce voxel-wise uncertainty information, both epistemic and aleatoric using test-time dropout (TTD) and data-augmentation (TTA) respectively. In addition, a hybrid approach is proposed that helps increase the accuracy of the segmentation. The model and uncertainty estimation measurements proposed in this work have been used in the BraTS’20 Challenge for task 1 and 3 regarding tumor segmentation and uncertainty estimation., This work has been partially supported by the project MALEGRA TEC2016-75976-R financed by the Spanish Ministerio de Economía y Competitividad., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Fuji-SfM dataset: A collection of annotated images and point clouds for Fuji apple detection and location using structure-from-motion photogrammetry

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio López, Eduard
The present dataset contains colour images acquired in a commercial Fuji apple orchard (Malus domestica Borkh. cv. Fuji) to reconstruct the 3D model of 11 trees by using structure-from-motion (SfM) photogrammetry. The data provided in this article is related to the research article entitled “Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry” [1]. The Fuji-SfM dataset includes: (1) a set of 288 colour images and the corresponding annotations (apples segmentation masks) for training instance segmentation neural networks such as Mask-RCNN; (2) a set of 582 images defining a motion sequence of the scene which was used to generate the 3D model of 11 Fuji apple trees containing 1455 apples by using SfM; (3) the 3D point cloud of the scanned scene with the corresponding apple positions ground truth in global coordinates. With that, this is the first dataset for fruit detection containing images acquired in a motion sequence to build the 3D model of the scanned trees with SfM and including the corresponding 2D and 3D apple location annotations. This data allows the development, training, and test of fruit detection algorithms either based on RGB images, on coloured point clouds or on the combination of both types of data. Dades primàries associades a l'article http://hdl.handle.net/10459.1/68505, This work was partly funded by the Secretaria d'Universitats i Recerca del Departament d'Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (project AGL2013-48297-C2-2-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). Part of the work was also developed within the framework of the project TEC2016-75976-R, financed by the Spanish Ministry of Economy, Industry and Competitiveness and the European Regional Development Fund (ERDF). The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Skip RNN: learning to skip state updates in recurrent neural networks

  • Campos Camunez, Victor
  • Jou, Brendan
  • Giró Nieto, Xavier
  • Torres Viñals, Jordi
  • Chang, Shih-Fu
Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/, This work was partially supported by the Spanish Ministry of Economy and Competitivity and the European Regional Development Fund (ERDF) under contracts TEC2016-75976-R and TIN2015- 65316-P, by the BSC-CNS Severo Ochoa program SEV-2015-0493, and grant 2014-SGR-1051 by the Catalan Government. Víctor Campos was supported by Obra Social “la Caixa” through La Caixa-Severo Ochoa International Doctoral Fellowship program. We would also like to thank the technical support team at the Barcelona Supercomputing Center., Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Simple vs complex temporal recurrences for video saliency prediction

  • Linardos, Panagiotis
  • Mohedano, Eva
  • Nieto, Juan Jose
  • O'Connor, Noel
  • Giró Nieto, Xavier
  • McGuinness, Kevin
This paper investigates modifying an existing neural network architecture for static saliency prediction using two types of recurrences that integrate information from the temporal domain. The first modification is the addition of a ConvLSTM within the architecture, while the second is a conceptually simple exponential moving average of an internal convolutional state. We use weights pre-trained on the SALICON dataset and fine-tune our model on DHF1K. Our results show that both modifications achieve state-of-the-art results and produce similar saliency maps., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

PathGAN: visual scanpath prediction with generative adversarial networks

  • Assens, Marc
  • Giró Nieto, Xavier
  • McGuinness, Kevin
  • O'Connor, Noel
“This is a post-peer-review, pre-copyedit version of an article published in: Computer Vision – ECCV 2018 Workshops. The final authenticated version is available online at: http://dx.doi.org/10.1007/978-3-030-11021-5_25”., We introduce PathGAN, a deep neural network for visual scanpath prediction trained on adversarial examples. A visual scanpath is defined as the sequence of fixation points over an image defined by a human observer with its gaze. PathGAN is composed of two parts, the generator and the discriminator. Both parts extract features from images using off-the-shelf networks, and train recurrent layers to generate or discriminate scanpaths accordingly. In scanpath prediction, the stochastic nature of the data makes it very difficult to generate realistic predictions using supervised learning strategies, but we adopt adversarial training as a suitable alternative. Our experiments prove how PathGAN improves the state of the art of visual scanpath prediction on the iSUN and Salient360! datasets., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

2D–3D geometric fusion network using multi-neighbourhood graph convolution for RGB-D indoor scene classification

  • Mosella Montoro, Albert
  • Ruiz Hidalgo, Javier
Multi-modal fusion has been proved to help enhance the performance of scene classification tasks. This paper presents a 2D-3D Fusion stage that combines 3D Geometric Features with 2D Texture Features obtained by 2D Convolutional Neural Networks. To get a robust 3D Geometric embedding, a network that uses two novel layers is proposed. The first layer, Multi-Neighbourhood Graph Convolution, aims to learn a more robust geometric descriptor of the scene combining two different neighbourhoods: one in the Euclidean space and the other in the Feature space. The second proposed layer, Nearest Voxel Pooling, improves the performance of the well-known Voxel Pooling. Experimental results, using NYU-Depth-V2 and SUN RGB-D datasets, show that the proposed method outperforms the current state-of-the-art in RGB-D indoor scene classification task., This work was supported by Secretary of Universities and Research of the Generalitat de Catalunya and the European Social Fund via a PhD grant (FI2018) in the framework of project TEC2016-75976-R, financed by the Ministerio de Economía, Industria y Competitividad and the European Regional Development Fund (ERDF)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

RVOS: end-to-end recurrent network for video object segmentation

  • Ventura Royo, Carles
  • Bellver, Míriam
  • Girbau Xalabarder, Andreu
  • Salvador Aguilera, Amaia
  • Marqués Acosta, Fernando
  • Giró Nieto, Xavier
Multiple object video object segmentation is a challenging task, specially for the zero-shot case, when no object mask is given at the initial frame and the model has to find the objects to be segmented along the sequence. In our work, we propose a Recurrent network for multiple object Video Object Segmentation (RVOS) that is fully end-to-end trainable. Our model incorporates recurrence on two different domains: (i) the spatial, which allows to discover the different object instances within a frame, and (ii) the temporal, which allows to keep the coherence of the segmented objects along time. We train RVOS for zero-shot video object segmentation and are the first ones to report quantitative results for DAVIS-2017 and YouTube-VOS benchmarks. Further, we adapt RVOS for one-shot video object segmentation by using the masks obtained in previous time steps as inputs to be processed by the recurrent module. Our model reaches comparable results to state-of-the-art techniques in YouTube-VOS benchmark and outperforms all previous video object segmentation methods not using online learning in the DAVIS-2017 benchmark. Moreover, our model achieves faster inference runtimes than previous methods, reaching 44ms/frame on a P100 GPU., This research was supported by the Spanish Ministry ofEconomy and Competitiveness and the European RegionalDevelopment Fund (TIN2015-66951-C2-2-R, TIN2015-65316-P & TEC2016-75976-R), the BSC-CNS SeveroOchoa SEV-2015-0493 and LaCaixa-Severo Ochoa Inter-national Doctoral Fellowship programs, the 2017 SGR 1414and the Industrial Doctorates 2017-DI-064 & 2017-DI-028from the Government of Catalonia, Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities

  • Gené Mola, Jordi
  • Vilaplana Besler, Verónica
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection., This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s predoctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Adria Carbó for his assistance in Faster R-CNN implementation.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Leishmaniasis parasite segmentation and classification using deep learning

  • Górriz, Marc
  • Aparicio, Albert
  • Raventós, Berta
  • Vilaplana Besler, Verónica
  • Sayrol Clols, Elisa
  • López Codina, Daniel
Leishmaniasis is considered a neglected disease that causes thousands of deaths annually in some tropical and subtropical countries. There are various techniques to diagnose leishmaniasis of which manual microscopy is considered to be the gold standard. There is a need for the development of automatic techniques that are able to detect parasites in a robust and unsupervised manner. In this paper we present a procedure for automatizing the detection process based on a deep learning approach. We train a U-net model that successfully segments leismania parasites and classifies them into promastigotes, amastigotes and adhered parasites., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Online detection of action start in untrimmed, streaming videos

  • Shou, Zheng
  • Pan, Junting
  • Chan, Jonathan
  • Miyazawa, Kazuyuki
  • Mansour, Hassan
  • Vetro, Anthony
  • Giró Nieto, Xavier
  • Chang, Shih-Fu
This is a post-peer-review, pre-copyedit version of an article published in: Lecture Notes in Computer Sciences, vol. 11207. The final authenticated version is available online at: http://dx.doi.org/10.1007/978-3-030-01219-9_33, We aim to tackle a novel task in action detection - Online Detection of Action Start (ODAS) in untrimmed, streaming videos. The goal of ODAS is to detect the start of an action instance, with high categorization accuracy and low detection latency. ODAS is important in many applications such as early alert generation to allow timely security or emergency response. We propose three novel methods to specifically address the challenges in training ODAS models: (1) hard negative samples generation based on Generative Adversarial Network (GAN) to distinguish ambiguous background, (2) explicitly modeling the temporal consistency between data around action start and data succeeding action start, and (3) adaptive sampling strategy to handle the scarcity of training data. We conduct extensive experiments using THUMOS'14 and ActivityNet. We show that our proposed methods lead to significant performance gains and improve the state-of-the-art methods. An ablation study confirms the effectiveness of each proposed method., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Vilaplana Besler, Verónica
  • Gregorio López, Eduard
The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html. Dades primàries associades a l'article http://hdl.handle.net/10459.1/68505, This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR646), the Spanish Ministry of Economy and Competitiveness (project AGL2013-48297-C2-2-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). Part of the work was also developed within the framework of the project TEC2016-75976-R, financed by the Spanish Ministry of Economy, Industry and Competitiveness and the European Regional Development Fund (ERDF). The Spanish Ministry of Educationis thanked for Mr. J.Gené’s pre-doctoral fellowships (FPU15/03355). We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Ernesto Membrillo and Roberto Maturino for their support in dataset labelling.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Apple size estimation using photogrammetry-derived 3D point clouds

  • Gené Mola, Jordi
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Escolà i Agustí, Alexandre
  • Gregorio López, Eduard
In-field fruit monitoring at different growth stages provides important information for farmers. Recent advances have focused on the detection and location of fruits, although the development of accurate fruit size estimation systems is still a challenge that requires further attention. The present work proposes a novel methodology for in-field measurement of spherical fruits using SfMMVS- based point clouds and considering different percentages of fruit occlusions., This work was partly funded by the Secretaria d’Universitats i Recerca de la Generalitat de Catalunya (grant 2017 SGR 646), the Spanish Ministry of Economy and Competitiveness (projects AGL2013-48297-C2-2-R and TEC2016-75976-R) and the Spanish Ministry of Science, Innovation and Universities (grant RTI2018- 094222-B-I00[PAgFRUIT project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Automatic reminiscence therapy for dementia

  • Carós, Mariona
  • Garolera Freixa, Maite
  • Radeva, Petia
  • Giró Nieto, Xavier
With people living longer than ever, the number of cases with dementia such as Alzheimer's disease increases steadily. It affects more than 46 million people worldwide, and it is estimated that in 2050 more than 100 million will be affected. While there are no effective treatments for these terminal diseases, therapies such as reminiscence, that stimulate memories from the past are recommended. Currently, reminiscence therapy takes place in care homes and is guided by a therapist or a carer. In this work, we present an AI-based solution to automate the reminiscence therapy. This consists of a dialogue system that uses photos of the users as input to generate questions about their life. Overall, this paper presents how reminiscence therapy can be automated by using deep learning, and deployed to smartphones and laptops, making the therapy more accessible to every person affected by dementia., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

One perceptron to rule them all: language, vision, audio and speech

  • Giró Nieto, Xavier
Deep neural networks have boosted the convergence of multimedia data analytics in a unified framework shared by practitioners in natural language, vision and speech. Image captioning, lip reading or video sonorization are some of the first applications of a new and exciting field of research exploiting the generalization properties of deep neural representation. This tutorial will firstly review the basic neural architectures to encode and decode vision, text and audio, to later review the those models that have successfully translated information across modalities., This tutorial was supported by the the Industrial Doctorate 2017-DI-064 from the Government of Catalonia, and the Spanish Ministryof Economy and Competitiveness through the European RegionalDevelopment Fund (TEC2016-75976-R)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

NeAT: a nonlinear analysis toolbox for neuroimaging

  • Casamitjana, Adrià
  • Vilaplana, Verónica
  • Puch, Santi
  • Aduriz, Asier
  • López, Carlos
  • Operto, Grégory
  • Cacciaglia, Raffaele
  • Falcón, Carles
  • Molinuevo, José Luis
  • Gispert, Juan Domingo
  • Alzheimer’s Disease Neuroimaging Initiative
NeAT is a modular, flexible and user-friendly neuroimaging analysis toolbox for modeling linear and nonlinear effects overcoming the limitations of the standard neuroimaging methods which are solely based on linear models. NeAT provides a wide range of statistical and machine learning non-linear methods for model estimation, several metrics based on curve fitting and complexity for model inference and a graphical user interface (GUI) for visualization of results. We illustrate its usefulness on two study cases where non-linear effects have been previously established. Firstly, we study the nonlinear effects of Alzheimer's disease on brain morphology (volume and cortical thickness). Secondly, we analyze the effect of the apolipoprotein APOE-ε4 genotype on brain aging and its interaction with age. NeAT is fully documented and publicly distributed at https://imatge-upc.github.io/neat-tool/., This work has been partially supported by the project MALEGRA TEC2016-75976-R financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF). Adrià Casamitjana is supported by the Spanish “Ministerio de Educación, Cultura y Deporte” FPU Research Fellowship. Juan D. Gispert holds a “‘Ramón y Cajal’” fellowship (RYC-2013-13054).
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Oriented trajectories as a method for audience measurement

  • López Palma, Manuel
  • Morros Rubió, Josep Ramon
  • Gago Barrio, Javier
  • Corbalán Fuertes, Montserrat
The quantification of the attention received by advertisements is of paramount importance to determine its effectiveness. In this work, a simple and effective objective method for the assessment of the attention given to advertisements is provided. The proposed method is based on computing the oriented trajectory of the different test persons along with their head pose. This way, it is possible to determine if a given person is looking towards the advertisement. While other similar methods use more complex setups, requiring a camera at each advertisement location, our method needs only a single (or a few) ceiling camera. Even though the apparent simplicity, the method can compute attention measures at each point of the scene., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

FuCiTNet: improving the generalization of deep learning networks by the fusion of learned class-inherent transformations

  • Rey-Arena, Manuel
  • Guirado, Emilio
  • Tabik, Siham
  • Ruiz Hidalgo, Javier
© <2020>. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/, It is widely known that very small datasets produce overfitting in Deep Neural Networks (DNNs), i.e., the network becomes highly biased to the data it has been trained on. This issue is often alleviated using transfer learning, regularization techniques and/or data augmentation. This work presents a new approach, independent but complementary to the previous mentioned techniques, for improving the generalization of DNNs on very small datasets in which the involved classes share many visual features. The proposed model, called FuCiTNet (Fusion Class inherent Transformations Network), inspired by GANs, creates as many generators as classes in the problem. Each generator, k, learns the transformations that bring the input image into the k-class domain. We introduce a classification loss in the generators to drive the leaning of specific k-class transformations. Our experiments demonstrate that the proposed transformations improve the generalization of the classification model in three diverse datasets., This work partially supported by the Spanish Ministry of Science and Technology under the project TIN2017-89517-P and the project TEC2016-75976-R, financed by the Spanish Ministerio de Economía, Industria y Competitividad and the European Regional Development Fund (ERDF). S. Tabik was supported by the Ramon y Cajal Programme (RYC-2015-18136). E.G was supported by the European Research Council (ERC Grant agreement 647038 [BIODESERT]), with additional support from Generalitat Valenciana (CIDEGENT/2018/041)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Towards large scale multimedia indexing: a case study on person discovery in broadcast news

  • Le, Nam
  • Bredin, Hervé
  • Sargent, Gabriel
  • India, Miquel
  • López Otero, Paula
  • Barras, Claude
  • Guinaudeau, Camille
  • Gravier, Guillaume
  • Barbosa da Fonseca, Gabriel
  • Lyon Freire, Izabela
  • Patrocínio, Zenilton
  • Guimarães, Silvio Jamil F.
  • Martí Juan, Gerard
  • Morros, Josep Ramon
  • Hernando, Javier
  • Docio Fernández, Laura
  • García Mateo, Carmen
  • Meignier, Sylvain
  • Odobez, Jean-Marc
Comunicació presentada a: the 15th International Workshop on Content-Based Multimedia Indexing (CBMI'17), celebrat a Florència, Itàlia, del 19 al 21 de juny de 2017, The rapid growth of multimedia databases and the human interest in their peers make indices representing the location and identity of people in audio-visual documents essential for searching archives. Person discovery in the absence of prior identity knowledge requires accurate association of audio-visual cues and detected names. To this end, we present 3 different strategies to approach this problem: clustering-based naming, verification-based naming, and graph-based naming. Each of these strategies utilizes different recent advances in unsupervised face / speech representation, verification, and optimization. To have a better understanding of the approaches, this paper also provides a quantitative and qualitative comparative study of these approaches using the associated corpus of the Person Discovery challenge at MediaEval 2016. From the results of our experiments, we can observe the pros and cons of each approach, thus paving the way for future promising research directions., This work was supported by the EU project EUMSSI (FP7-611057), ANR project MetaDaTV (ANR-14-CE24-0024) project, Camomile project (PCIN-2013-067), and the projects TEC2013-43935-R, TEC2015-69266-P, TEC2016-75976-R, TEC2015-65345-P financed by the Spanish government and ERDF.
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Mask-guided sample selection for semi-supervised instance segmentation

  • Bellver Bueno, Míriam
  • Salvador Aguilera, Amaia
  • Torres Viñals, Jordi
  • Giró Nieto, Xavier
The final publication is available at Springer via http://dx.doi.org/10.1007/s11042-020-09235-4, Image segmentation methods are usually trained with pixel-level annotations, which require significant human effort to collect. Weakly-supervised pipelines are the most common solution to address this constraint because they are trained with lower forms of supervision, such as bounding boxes or scribbles. Semi-supervised methods are another option, that leverage a large amount of unlabeled data and a limited number of strongly-labeled samples. In this second setup, samples to be strongly-annotated can be selected randomly or with an active learning mechanism that chooses the ones that will maximize the model performance. In this work, we propose a sample selection approach to decide which samples to annotate for semi-supervised instance segmentation. Our method consists in first predicting pseudo-masks for the unlabeled pool of samples, together with a score predicting the quality of each mask. This score is an estimate of the Intersection Over Union (IoU) of the segment with the ground truth mask. We study which samples should be annotated based on the quality score, leading to an improved performance for semi-supervised instance segmentation with low annotation budgets., This work was partially supported by the Spanish Ministry of Economy and Competitivity under contracts TIN2012-34557 by the BSC-CNS Severo Ochoa program (SEV-2011-00067), and contracts TEC2013-43935-R and TEC2016-75976-R. It has also been supported by grants 2014-SGR-1051 and 2014-SGR-1421 by the Government of Catalonia, and the European Regional Development Fund (ERDF). We would also like to acknowledge the valuable discussions with Victor Campos., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Temporally coherent 3D point cloud video segmentation in generic scenes

  • Lin, Xiao
  • Casas Pla, Josep Ramon
  • Pardàs Feliu, Montse
© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Video segmentation is an important building block for high level applications, such as scene understanding and interaction analysis. While outstanding results are achieved in this field by the state-of-the-art learning and model-based methods, they are restricted to certain types of scenes or require a large amount of annotated training data to achieve object segmentation in generic scenes. On the other hand, RGBD data, widely available with the introduction of consumer depth sensors, provide actual world 3D geometry compared with 2D images. The explicit geometry in RGBD data greatly help in computer vision tasks, but the lack of annotations in this type of data may also hinder the extension of learning-based methods to RGBD. In this paper, we present a novel generic segmentation approach for 3D point cloud video (stream data) thoroughly exploiting the explicit geometry in RGBD. Our proposal is only based on low level features, such as connectivity and compactness. We exploit temporal coherence by representing the rough estimation of objects in a single frame with a hierarchical structure and propagating this hierarchy along time. The hierarchical structure provides an efficient way to establish temporal correspondences at different scales of object-connectivity and to temporally manage the splits and merges of objects. This allows updating the segmentation according to the evidence observed in the history. The proposed method is evaluated on several challenging data sets, with promising results for the presented approach., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

RVOS: end-to-end recurrent network for video object segmentation

  • Ventura Royo, Carles
  • Bellver Bueno, Míriam
  • Girbau Xalabarder, Andreu
  • Salvador Aguilera, Amaia
  • Marqués Acosta, Fernando
  • Giró Nieto, Xavier
Multiple object video object segmentation is a challenging task, specially for the zero-shot case, when no object mask is given at the initial frame and the model has to find the objects to be segmented along the sequence. In our work, we propose a Recurrent network for multiple object Video Object Segmentation (RVOS) that is fully end-to-end trainable. Our model incorporates recurrence on two different domains: (i) the spatial, which allows to discover the different object instances within a frame, and (ii) the temporal, which allows to keep the coherence of the segmented objects along time. We train RVOS for zero-shot video object segmentation and are the first ones to report quantitative results for DAVIS-2017 and YouTube-VOS benchmarks. Further, we adapt RVOS for one-shot video object segmentation by using the masks obtained in previous time steps as inputs to be processed by the recurrent module. Our model reaches comparable results to state-of-the-art techniques in YouTube-VOS benchmark and outperforms all previous video object segmentation methods not using online learning in the DAVIS-2017 benchmark. Moreover, our model achieves faster inference runtimes than previous methods, reaching 44ms/frame on a P100 GPU., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Prediction of amyloid pathology in cognitively unimpaired individuals using voxel-wise analysis of longitudinal structural brain MRI

  • Petrone, Paula
  • Casamitjana Díaz, Adrià
  • Falcón, Carlos
  • Artigues, Miquel
  • Operto, Grégory
  • Cacciaglia, Raffaele
  • Molinuevo, José Luis
  • Vilaplana Besler, Verónica
  • Gispert, Juan Domingo
Background Magnetic resonance imaging (MRI) has unveiled specific alterations at different stages of Alzheimer’s disease (AD) pathophysiologic continuum constituting what has been established as “AD signature”. To what extent MRI can detect amyloid-related cerebral changes from structural MRI in cognitively unimpaired individuals is still an area open for exploration. Method Longitudinal 3D-T1 MRI scans were acquired from a subset of the ADNI cohort comprising 403 subjects: 79 controls (Ctrls), 50 preclinical AD (PreAD), and 274 MCI and dementia due to AD (MCI/AD). Amyloid CSF was used as gold-standard measure with established cutoffs (<¿192¿pg/mL) to establish diagnostic categories. Cognitively unimpaired individuals were defined as Ctrls if were amyloid negative and PreAD otherwise. The MCI/AD group was amyloid positive. Only subjects with the same diagnostic category at baseline and follow-up visits were considered for the study. Longitudinal morphometric analysis was performed using SPM12 to calculate Jacobian determinant maps. Statistical analysis was carried out on these Jacobian maps to identify structural changes that were significantly different between diagnostic categories. A machine learning classifier was applied on Jacobian determinant maps to predict the presence of abnormal amyloid levels in cognitively unimpaired individuals. The performance of this classifier was evaluated using receiver operating characteristic curve analysis and as a function of the follow-up time between MRI scans. We applied a cost function to assess the benefit of using this classifier in the triaging of individuals in a clinical trial-recruitment setting. Results The optimal follow-up time for classification of Ctrls vs PreAD was ¿t¿>¿2.5¿years, and hence, only subjects within this temporal span are used for evaluation (15 Ctrls, 10 PreAD). The longitudinal voxel-based classifier achieved an AUC¿=¿0.87 (95%CI 0.72–0.97). The brain regions that showed the highest discriminative power to detect amyloid abnormalities were the medial, inferior, and lateral temporal lobes; precuneus; caudate heads; basal forebrain; and lateral ventricles. Conclusions Our work supports that machine learning applied to longitudinal brain volumetric changes can be used to predict, with high precision, the presence of amyloid abnormalities in cognitively unimpaired subjects. Used as a triaging method to identify a fixed number of amyloid-positive individuals, this longitudinal voxel-wise classifier is expected to avoid 55% of unnecessary CSF and/or PET scans and reduce economic cost by 40%., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Cross-modal embeddings for video and audio retrieval

  • Surís Coll-Vinent, Dídac
  • Duarte, Amanda
  • Salvador Aguilera, Amaia
  • Torres Viñals, Jordi
  • Giró Nieto, Xavier
In this work, we explore the multi-modal information provided by the Youtube-8M dataset by projecting the audio and visual features into a common feature space, to obtain joint audio-visual embeddings. These links are used to retrieve audio samples that fit well to a given silent video, and also to retrieve images that match a given query audio. The results in terms of Recall@K obtained over a subset of YouTube-8M videos show the potential of this unsupervised approach for cross-modal feature learning., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Projection to latent spaces disentangles pathological effects on brain morphology in the asymptomatic phase of Alzheimer's disease

  • Casamitjana Díaz, Adrià
  • Petrone, Paula
  • Molinuevo, Jose Luis
  • Gispert, Juan Domingo
  • Vilaplana Besler, Verónica
Alzheimer's disease (AD) continuum is defined as a cascade of several neuropathological processes that can be measured using biomarkers, such as cerebrospinal fluid (CSF) levels of Aß, p-tau, and t-tau. In parallel, brain anatomy can be characterized through imaging techniques, such as magnetic resonance imaging (MRI). In this work we relate both sets of measurements and seek associations between biomarkers and the brain structure that can be indicative of AD progression. The goal is to uncover underlying multivariate effects of AD pathology on regional brain morphological information. For this purpose, we used the projection to latent structures (PLS) method. Using PLS, we found a low dimensional latent space that best describes the covariance between both sets of measurements on the same subjects. Possible confounder effects (age and sex) on brain morphology are included in the model and regressed out using an orthogonal PLS model. We looked for statistically significant correlations between brain morphology and CSF biomarkers that explain part of the volumetric variance at each region-of-interest (ROI). Furthermore, we used a clustering technique to discover a small set of CSF-related patterns describing the AD continuum. We applied this technique to the study of subjects in the whole AD continuum, from the pre-clinical asymptomatic stages all the way through to the symptomatic groups. Subsequent analyses involved splitting the course of the disease into diagnostic categories: cognitively unimpaired subjects (CU), mild cognitively impaired subjects (MCI), and subjects with dementia (AD-dementia), where all symptoms were due to AD., This work has been partially supported by the project MALEGRA TEC2016-75976-R financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF). AC was supported by the Spanish Ministerio de Educación, Cultura y Deporte FPU Research Fellowship. JG holds a Ramón y Cajal fellowship (RYC-2013-13054)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

KFuji RGB-DS database: Fuji apple multi-modal images for fruit detection with color, depth and range-corrected IR data

  • Gené Mola, Jordi
  • Vilaplana Besler, Verónica
  • Rosell Polo, Joan Ramon
  • Morros Rubió, Josep Ramon
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
This article contains data related to the research article entitle 'Multi-modal Deep Learning for Fruit Detection Using RGB-D Cameras and their Radiometric Capabilities' [1]. The development of reliable fruit detection and localization systems is essential for future sustainable agronomic management of high-value crops. RGB-D sensors have shown potential for fruit detection and localization since they provide 3D information with color data. However, the lack of substantial datasets is a barrier for exploiting the use of these sensors. This article presents the KFuji RGBDS database which is composed by 967 multi-modal images of Fuji apples on trees captured using Microsoft Kinect v2 (Microsoft, Redmond, WA, USA). Each image contains information from 3 different modalities: color (RGB), depth (D) and range corrected IR intensity (S). Ground truth fruit locations were manually annotated, labeling a total of 12,839 apples in all the dataset. The current dataset is publicly available at http://www.grap.udl.cat/publicacions/datasets.html., This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017 SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s pre-doctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition.
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

What is going on in the world? A display platform for media understanding

  • Fernàndez, Dèlia
  • Varas, David
  • Espadaler, Joan
  • Masuda, Issey
  • Giró Nieto, Xavier
  • Riveiro, Juan Carlos
  • Bou Balust, Elisenda
News broadcasters and on-line publishers daily generate a large amount of articles and videos describing events currently happening in the world. In this work, we present a system that automatically indexes videos from a library and links them to stories developing in the news. The user interface displays in an intuitive manner the links between videos and stories and allows navigation through related content by using associated tags. This interface is a powerful industrial tool for publishers to index, retrieve and visualize their video content. It helps them identify which topics require more attention or retrieve related content that has already been published about the stories., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

NeAT: a nonlinear analysis toolbox for neuroimaging

  • Casamitjana Díaz, Adrià
  • Vilaplana Besler, Verónica
  • Puch Giner, Santi
  • Aduriz Saiz, Asier
  • Operto, Grégory
  • Cacciaglia, Raffaele
  • Falcón, Carlos
  • Molinuevo, José Luis
  • Gispert, Juan Domingo
  • López Molina, Carlos Alejandro
NeAT is a modular, flexible and user-friendly neuroimaging analysis toolbox for modeling linear and nonlinear effects overcoming the limitations of the standard neuroimaging methods which are solely based on linear models. NeAT provides a wide range of statistical and machine learning non-linear methods for model estimation, several metrics based on curve fitting and complexity for model inference and a graphical user interface (GUI) for visualization of results. We illustrate its usefulness on two study cases where non-linear effects have been previously established. Firstly, we study the nonlinear effects of Alzheimer’s disease on brain morphology (volume and cortical thickness). Secondly, we analyze the effect of the apolipoprotein APOE-e4 genotype on brain aging and its interaction with age. NeAT is fully documented and publicly distributed at https://imatge-upc.github.io/neat-tool/., This work has been partially supported by the project MALEGRA TEC2016-75976-R financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF). Adrià Casamitjana is supported by the Spanish “Ministerio de Educación, Cultura y Deporte” FPU Research Fellowship. Juan D. Gispert holds a “‘Ramón y Cajal’” fellowship (RYC-2013-13054). Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wpcontent/uploads/how to apply/ADNI Acknowledgement List.pdf., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Explore, discover and learn: unsupervised discovery of state-covering skills

  • Campos Camúñez, Víctor
  • Trott, Alex
  • Xiong, Caiming
  • Socher, Richard
  • Giró Nieto, Xavier
  • Torres Viñals, Jordi
Acquiring abilities in the absence of a task-oriented reward function is at the frontier of reinforcement learning research. This problem has been studied through the lens of empowerment, which draws a connection between option discovery and information theory. Information-theoretic skill discovery methods have garnered much interest from the community, but little research has been conducted in understanding their limitations. Through theoretical analysis and empirical evidence, we show that existing algorithms suffer from a common limitation -- they discover options that provide a poor coverage of the state space. In light of this, we propose 'Explore, Discover and Learn' (EDL), an alternative approach to information-theoretic skill discovery. Crucially, EDL optimizes the same information-theoretic objective derived from the empowerment literature, but addresses the optimization problem using different machinery. We perform an extensive evaluation of skill discovery methods on controlled environments and show that EDL offers significant advantages, such as overcoming the coverage problem, reducing the dependence of learned skills on the initial state, and allowing the user to define a prior over which behaviors should be learned., This work was partially supported by the Spanish Ministry of Science and Innovation and the European Regional Development Fund under contracts TEC2016-75976-R and TIN2015-65316-P, by the BSC-CNS Severo Ochoa program SEV-2015-0493, and by Generalitat de Catalunya under contracts 2017-SGR-1414 and 2017-DI-011. Víctor Campos was supported by Obra Social “la Caixa” through La Caixa-Severo Ochoa International Doctoral Fellowship program., Peer Reviewed, Postprint (published version)