Pasar al contenido principal

APRENDIZAJE PROFUNDO EFICIENTE PARA SECUENCIAS DE VIDEO Y NUBES DE PUNTOS

PID2020-117142GB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2020
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POLITECNICA DE CATALUNYA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 38
Encontrada(s) 1 página(s)

Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené Mola, Jordi
  • Ferrer Ferrer, Mar
  • Hemming, Jochen
  • Dalfsen, Pieter van
  • Hoog, Dirk de
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Vilaplana Besler, Verónica
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation”., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088 ), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition.
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

Video-Based Fruit Detection and Tracking for Apple Counting and Mapping

  • Gené Mola, Jordi
  • Felip Pomés, Marc
  • Net Barnés, Francesc
  • Morros Rubió, Josep Ramon
  • Miranda, Juan Carlos
  • Arnó Satorra, Jaume
  • Asin Jones, Luis
  • Lordan Sanahuja, Jaume
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
Automatic fruit counting systems have garnered interest from farmers and agronomists to monitor fruit production, predict yields in advance, and identify production variability across orchards. However, accurately counting fruits poses challenges, particularly due to occlusions. This study proposes a multi-view sensing approach using continuous motion videos captured by a camera moved along the row of trees, followed by fruit detection in all video frames and application of Multi-Object Tracking (MOT) algorithms to prevent double-counting. Three tracking methods, namely SORT, DeepSORT, and ByteTrack, are compared for fruit counting using the YOLOv5x object detector. The methodology is applied to map fruit production in an experimental apple orchard at two different dates: four weeks and one week before harvest. The results demonstrate that ByteTrack (MOTA=0.682; IDF1=0.837; HOTA=0.689) outperforms SORT and DeepSORT, indicating its superior tracking performance. Computational efficiency analysis reveals similar processing times between SORT and ByteTrack (about 15 ms), while DeepSORT requires significantly more processing time per image (128 ms). Fruit counting evaluation shows reasonably accurate yield predictions on both dates, with reduced errors and improved performance closer to the harvest date (MAPE=7.47 %; R2=0.70). The system proves effective in estimating orchard fruit production using computer vision technology, offering valuable insights for yield forecasting. These findings contribute to optimizing fruit production and supporting precision agriculture practices. The code and the dataset have been made publicly available and a video visualization of results is accessible at http://www.grap.udl.cat/en/publications/video_fruit_counting., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda’s pre doctoral fellowship (2020 FI_B 00586).
Repositori Obert UdL
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené Mola, Jordi
  • Ferrer Ferrer, Mar
  • Gregorio López, Eduard
  • Blok, Pieter
  • Hemming, Jochen
  • Morros Rubió, Josep Ramon
  • Rosell Polo, Joan Ramon
  • Vilaplana Besler, Verónica
  • Ruiz Hidalgo, Javier
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for sim ultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gen´e Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

A closer look at referring expressions for video object segmentation

  • Bellver, Miriam
  • Ventura, Carles
  • Silberer, Carina
  • Kazakos, Ioannis
  • Torres, Jordi
  • Giró i Nieto, Xavier
The task of Language-guided Video Object Segmentation (LVOS) aims at generating binary masks for an object referred by a linguistic expression. When this expression unambiguously describes an object in the scene, it is named referring expression (RE). Our work argues that existing benchmarks used for LVOS are mainly composed of trivial cases, in which referents can be identified with simple phrases. Our analysis relies on a new categorization of the referring expressions in the DAVIS-2017 and Actor-Action datasets into trivial and non-trivial REs, where the non-trivial REs are further annotated with seven RE semantic categories. We leverage these data to analyze the performance of RefVOS, a novel neural network that obtains competitive results for the task of language-guided image segmentation and state of the art results for LVOS. Our study indicates that the major challenges for the task are related to understanding motion and static actions., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was partially supported by the projects PID2019-107255GB-C22 and PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033 Spanish Ministry of Science, and the grant 2017-SGR-1414 of the Government of Catalonia. This work was also partially supported by the project RTI2018-095232-B-C22 funded by the Spanish Ministry of Science, Innovation and Universities.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

InstantAvatar: Efficient 3D Head Reconstruction via Surface Rendering

  • Canela, Antonio
  • Caselles, Pol
  • Malik, Ibrar
  • Ramon, Eduard
  • Martínez García, Jaime
  • Sánchez-Riera, Jordi
  • Triginer, Gil
  • Moreno-Noguer, Francesc
Recent advances in full-head reconstruction have been obtained by optimizing a neural field through differentiable surface or volume rendering to represent a single scene. While these techniques achieve an unprecedented accuracy, they take several minutes, or even hours, due to the expensive optimization process required. In this work, we introduce InstantAvatar, a method that recovers full-head avatars from few images (down to just one) in a few seconds on commodity hardware. In order to speed up the reconstruction process, we propose a system that combines, for the first time, a voxel-grid neural field representation with a surface renderer. Notably, a naive combination of these two techniques leads to unstable optimizations that do not converge to valid solutions. In order to overcome this limitation, we present a novel statistical model that learns a prior distribution over 3D head signed distance functions using a voxel-grid based architecture. The use of this prior model, in combination with other design choices, results into a system that achieves 3D head reconstructions with comparable accuracy as the state-of-the-art with a 100 × speed-up., This work has been partially supported by: MoHuCo PID2020-120049RB-I00 funded by MCIN/AEI/10.13039/501100011033; DeeLight PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033; and the industrial doctorate 2021 DI 00102 funded by the Government of Catalonia., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

DASPack: controlled data compression for distributed acoustic sensing

  • Seguí, Aleix
  • Ugalde, Arantza
  • Fichtner, Andreas
  • Ventosa, Sergio
  • Morros, Josep Ramon
16 pages, 13 figures, 1 table.-- Data Availability: DASPack is openly available at https://github.com/asleix/daspack, including examples to use the software. We used an existing entropy coding implementation named Constriction, https://github.com/bamler-lab/constriction. For published data sets listed in Table 1, details on open data access can be found in the corresponding publications. Unpublished data sets are available from the respective authors (A. Fichtner for Ids. 4 and 7; A. Ugalde for Id. 12) upon reasonable request. Data segments used in Figs 8–11 are available at https://doi.org/10.20350/digitalCSIC/17586 (Ugalde & Fitchner 2025), We present DASPack, a high-performance, open-source compression tool specifically designed for distributed acoustic sensing (DAS) data. As DAS becomes a key technology for real-time, high-density and long-range monitoring in fields such as geophysics, infrastructure surveillance and environmental sensing, the volume of collected data is rapidly increasing. Large-scale DAS deployments already generate hundreds of terabytes and are expected to increase in the coming years, making long-term storage a major challenge. Despite this urgent need, few compression methods have proven to be both practical and scalable in real-world scenarios. DASPack is a fully operational solution that consistently outperforms existing techniques for DAS data. It enables both controlled lossy and lossless compression by allowing users to choose the maximum absolute difference per datum between the original and compressed data. The compression pipeline combines wavelet transforms, linear predictive coding, and entropy coding to optimise efficiency. Our method achieves up to file size reductions for strain and strain rate data in lossless mode across diverse data sets. In lossy mode, compression improves to with near-perfect signal fidelity, and up to is reached with acceptable signal degradation. It delivers fast throughput (100–200 MB s using a single-thread and up to 750 MB s using 8-threads), enabling real-time deployment even under high data rates. We validated its performance on 15 data sets from a variety of acquisition environments, demonstrating its speed, robustness and broad applicability. DASPack provides a practical foundation for long-term, sustainable DAS data management in large-scale monitoring networks, This work was supported in part by the ‘Severo Ochoa Centre of Excellence’ program (CEX2019-000928-S-21-2); the European Union NextGenerationEU/PRTR Program under projects PSI (PLEC2021-007875), TREMORS (CPP2021-008869), the CSIC contract SAFE with Telxius Cable España, S. L. (ref. 20233069) and the Spanish Ministry of Science, Innovation and Universities under DeeLight project (PID2020-117142GB-I00), Peer reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

A closer look at referring expressions for video object segmentation

  • Bellver Bueno, Míriam
  • Ventura Royo, Carles|||0000-0002-3055-339X
  • Silberer, Carina
  • Kazakos, Ioannis
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Giró Nieto, Xavier|||0000-0002-9935-5332
The task of Language-guided Video Object Segmentation (LVOS) aims at generating binary masks for an object referred by a linguistic expression. When this expression unambiguously describes an object in the scene, it is named referring expression (RE). Our work argues that existing benchmarks used for LVOS are mainly composed of trivial cases, in which referents can be identified with simple phrases. Our analysis relies on a new categorization of the referring expressions in the DAVIS-2017 and Actor-Action datasets into trivial and non-trivial REs, where the non-trivial REs are further annotated with seven RE semantic categories. We leverage these data to analyze the performance of RefVOS, a novel neural network that obtains competitive results for the task of language-guided image segmentation and state of the art results for LVOS. Our study indicates that the major challenges for the task are related to understanding motion and static actions., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was partially supported by the projects PID2019-107255GB-C22 and PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033 Spanish Ministry of Science, and the grant 2017-SGR-1414 of the Government of Catalonia. This work was also partially supported by the project RTI2018-095232-B-C22 funded by the Spanish Ministry of Science, Innovation and Universities., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Single-image super-resolution of sentinel-2 low resolution bands with residual dense convolutional neural networks

  • Salgueiro Romero, Luis Fernando|||0000-0003-4048-8330
  • Marcello Ruiz, Javier
  • Vilaplana Besler, Verónica|||0000-0001-6924-9961
Sentinel-2 satellites have become one of the main resources for Earth observation images because they are free of charge, have a great spatial coverage and high temporal revisit. Sentinel-2 senses the same location providing different spatial resolutions as well as generating a multi-spectral image with 13 bands of 10, 20, and 60 m/pixel. In this work, we propose a single-image super-resolution model based on convolutional neural networks that enhances the low-resolution bands (20 m and 60 m) to reach the maximal resolution sensed (10 m) at the same time, whereas other approaches provide two independent models for each group of LR bands. Our proposed model, named Sen2-RDSR, is made up of Residual in Residual blocks that produce two final outputs at maximal resolution, one for 20 m/pixel bands and the other for 60 m/pixel bands. The training is done in two stages, first focusing on 20 m bands and then on the 60 m bands. Experimental results using six quality metrics (RMSE, SRE, SAM, PSNR, SSIM, ERGAS) show that our model has superior performance compared to other state-of-the-art approaches, and it is very effective and suitable as a preliminary step for land and coastal applications, as studies involving pixel-based classification for Land-Use-Land-Cover or the generation of vegetation indices., This work was funded by the Spanish Agencia Estatal de Investigación (AEI) under projects ARTEMISAT-2 (CTM2016-77733-R) and PID2020-117142GB-I00 of the call MCIN/AEI/10.13039/501100011033)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Co3DCAE: feature compression for intermediate fusion collaborative perception for autonomous driving

  • Wang, Mingrui
  • Salavedra Pujol, Judit|||0009-0004-5927-7132
  • Li, Dongjie|||0000-0002-1005-7890
  • Casas Pla, Josep Ramon|||0000-0003-4639-6904
  • Ruiz Hidalgo, Javier|||0000-0001-6774-685X
In recent years, multi-agent sensing technology has attracted considerable attention, particularly in the field of vehicle-to-vehicle (V2V) collaborative perception for autonomous driving. By sharing complementary information, connected autonomous vehicles (CAVs) can significantly improve scene understanding. However, this collaboration also presents challenges, such as balancing data transmission efficiency with the accuracy of environmental perception. To address these issues, we propose an autoencoder based on intermediate-level feature fusion. Specifically, we design a 3D convolutional autoencoder with a residual structure to maximize the compression of inter-vehicle bandwidth usage while retaining essential feature information. By encoding the spatial-channel dimensions of intermediate feature maps extracted from point clouds, network bandwidth is reduced by 3–10× under different configurations. Additionally, we propose a channel-wise autoencoder training strategy to reduce model parameters and enhance feature compression efficiency, while preserving collaborative perception performance. Finally, we conduct a comprehensive evaluation of perception accuracy and bandwidth consumption using a collaborative detection dataset collected via LiDAR. Extensive experimental results demonstrate that our method offers significant advantages in balancing object detection accuracy and bandwidth compression for CAVs., This work was partially supported by the Spanish Research Agency (AEI) under project PID2020-117142GB-I00 of the call MCIN/ AEI /10.13039/501100011033 and by the 6G-EWOC project from the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101139182., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

SEG-ESRGAN: A multi-task network for super-resolution and semantic segmentation of remote sensing images

  • Salgueiro Romero, Luis Fernando|||0000-0003-4048-8330
  • Marcello Ruiz, Javier
  • Vilaplana Besler, Verónica|||0000-0001-6924-9961
The production of highly accurate land cover maps is one of the primary challenges in remote sensing, which depends on the spatial resolution of the input images. Sometimes, high-resolution imagery is not available or is too expensive to cover large areas or to perform multitemporal analysis. In this context, we propose a multi-task network to take advantage of the freely available Sentinel-2 imagery to produce a super-resolution image, with a scaling factor of 5, and the corresponding high-resolution land cover map. Our proposal, named SEG-ESRGAN, consists of two branches: the super-resolution branch, that produces Sentinel-2 multispectral images at 2 m resolution, and an encoder–decoder architecture for the semantic segmentation branch, that generates the enhanced land cover map. From the super-resolution branch, several skip connections are retrieved and concatenated with features from the different stages of the encoder part of the segmentation branch, promoting the flow of meaningful information to boost the accuracy in the segmentation task. Our model is trained with a multi-loss approach using a novel dataset to train and test the super-resolution stage, which is developed from Sentinel-2 and WorldView-2 image pairs. In addition, we generated a dataset with ground-truth labels for the segmentation task. To assess the super-resolution improvement, the PSNR, SSIM, ERGAS, and SAM metrics were considered, while to measure the classification performance, we used the IoU, confusion matrix and the F1-score. Experimental results demonstrate that the SEG-ESRGAN model outperforms different full segmentation and dual network models (U-Net, DeepLabV3+, HRNet and Dual_DeepLab), allowing the generation of high-resolution land cover maps in challenging scenarios using Sentinel-2 10 m bands., This work was funded by the Spanish Agencia Estatal de Investigación (AEI) under projects ARTEMISAT-2 (CTM 2016-77733-R), PID2020-117142GB-I00 and PID2020-116907RB-I00 (MCIN/AEI call 10.13039/501100011033). L.S. would like to acknowledge the BECAL (Becas Carlos Antonio López) scholarship for the financial support., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Simultaneous fruit detection and size estimation using multitask deep neural networks

  • Ferrer Ferrer, Mar
  • Ruiz Hidalgo, Javier|||0000-0001-6774-685X
  • Gregorio López, Eduard
  • Vilaplana Besler, Verónica|||0000-0001-6924-9961
  • Morros Rubió, Josep Ramon|||0000-0002-1395-487X
  • Gené-Mola, Jordi
The measurement of fruit size is of great interest to estimate the yield and predict the harvest resources in advance. This work proposes a novel technique for in-field apple detection and measurement based on Deep Neural Networks. The proposed framework was trained with RGB-D data and consists of an end-to-end multitask Deep Neural Network architecture specifically designed to perform the following tasks: 1) detection and segmentation of each fruit from its surroundings; 2) estimation of the diameter of each detected fruit. The methodology was tested with a total of 15,335 annotated apples at different growth stages, with diameters varying from 27 mm to 95 mm. Fruit detection results reported an F1-score for apple detection of 0.88 and a mean absolute error of diameter estimation of 5.64 mm. These are state-of-the-art results with the additional advantages of: a) using an end-to-end multitask trainable network; b) an efficient and fast inference speed; and c) being based on RGB-D data which can be acquired with affordable depth cameras. On the contrary, the main disadvantage is the need of annotating a large amount of data with fruit masks and diameter ground truth to train the model. Finally, a fruit visibility analysis showed an improvement in the prediction when limiting the measurement to apples above 65% of visibility (mean absolute error of 5.09 mm). This suggests that future works should develop a method for automatically identifying the most visible apples and discard the prediction of highly occluded fruits., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00[PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gene Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union-NextGenerationEU., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

A Cartesian encoding graph neural network for crystal structure property prediction: application to thermal ellipsoid estimation

  • Solé, Àlex|||0000-0002-2071-9317
  • Mosella Montoro, Albert
  • Cardona Olives, Joan
  • Gómez Coca, Silvia
  • Aravena Ponce, Daniel
  • Ruiz Sabin, Eliseo
  • Ruiz Hidalgo, Javier|||0000-0001-6774-685X
In the diffraction resolution of crystal structures, thermal ellipsoids are a critical parameter that is usually more difficult to determine than atomic positions. These ellipsoids are quantified through Anisotropic Displacement Parameters (ADPs), which provide critical insights into atomic vibrations within crystalline structures. ADPs reflect the thermal behaviour and structural properties of crystal structures. However, traditional methods to compute ADPs are computationally intensive. This paper presents CartNet, a novel graph neural network (GNN) architecture designed to predict properties of crystal structures efficiently by encoding the atomic structural geometry to the Cartesian axes and the temperature of the crystal structure. Additionally, CartNet employs a neighbour equalization technique for message passing to help emphasise the covalent and contact interactions and a novel Cholesky-based head to ensure valid ADP predictions. Furthermore, a rotational SO(3) data augmentation technique has been proposed during the training phase to generalize unseen rotations. To corroborate this procedure, an ADP dataset with over 200 000 experimental crystal structures from the Cambridge Structural Database (CSD) has been curated. The model significantly reduces computational costs and outperforms existing previously reported methods for ADP prediction by 10.87%, while demonstrating a 34.77% improvement over the tested theoretical computation methods. Moreover, we have employed CartNet for other already known datasets that included different material properties, such as formation energy, band gap, total energy, energy above the convex hull, bulk moduli, and shear moduli. The proposed architecture outperformed previously reported methods by 7.71% in the JARVIS dataset and 13.16% in the Materials Project dataset, proving CarNet's capability to achieve state-of-the-art results in several tasks. The project website with online demo available at: https://www.ee.ub.edu/cartnet., Financial support from Ministerio de Ciencia e Innovación (project PID2020-117142GB-I00, PID2021-122464NB-I00, TED2021-129593B-I00, CNS2023-144561 and Maria de Maeztu CEX2021-001202-M) is acknowledged. We also acknowledge the Generalitat de Catalunya for the 2021-SGR-00286 grant. E. R. also acknowledges the Generalitat de Catalunya for an ICREA Academia grant., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

DASPack: controlled data compression for distributed acoustic sensing

  • Seguí Ugalde, Aleix
  • Ugalde Aguirre, Arantza
  • Fichtner, Andreas
  • Ventosa Rahuet, Sergi
  • Morros Rubió, Josep Ramon|||0000-0002-1395-487X
We present DASPack, a high-performance, open-source compression tool specifically de- signed for distributed acoustic sensing (DAS) data. As DAS becomes a key technology for real-time, high-density and long-range monitoring in fields such as geophysics, infrastructure surveillance and environmental sensing, the volume of collected data is rapidly increasing. Large-scale DAS deployments already generate hundreds of terabytes and are expected to increase in the coming years, making long-term storage a major challenge. Despite this urgent need, few compression methods have proven to be both practical and scalable in real-world scenarios. DASPack is a fully operational solution that consistently outperforms existing tech- niques for DAS data. It enables both controlled lossy and lossless compression by allowing users to choose the maximum absolute difference per datum between the original and com- pressed data. The compression pipeline combines wavelet transforms, linear predictive coding, and entropy coding to optimise efficiency. Our method achieves up to 3 × file size reductions for strain and strain rate data in lossless mode across diverse data sets. In lossy mode, compres- sion improves to 6 × with near-perfect signal fidelity, and up to 10 × is reached with acceptable signal degradation. It delivers fast throughput (100–200 MB s −1 using a single-thread and up to 750 MB s −1 using 8-threads), enabling real-time deployment even under high data rates. We validated its performance on 15 data sets from a variety of acquisition environments, demon- strating its speed, robustness and broad applicability. DASPack provides a practical foundation for long-term, sustainable DAS data management in large-scale monitoring networks., This work was supported in part by the ‘Severo Ochoa Centre of Excellence’ program (CEX2019-000928-S-21-2); the European Union NextGenerationEU/PRTR Program under projects PSI (PLEC2021-007875), TREMORS (CPP2021-008869), the CSIC contract SAFE with Telxius Cable España, S. L. (ref. 20233069) and the Spanish Ministry of Science, Innovation and Universities under DeeLight project (PID2020-117142GB-I00)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Adaptive fusion of LiDAR features for 3D object detection in autonomous driving

  • Wang, Mingrui
  • Li, Dongjie|||0000-0002-1005-7890
  • Casas Pla, Josep Ramon|||0000-0003-4639-6904
  • Ruiz Hidalgo, Javier|||0000-0001-6774-685X
In the field of autonomous driving, cooperative perception through vehicle-to-vehicle communication significantly enhances environmental understanding by leveraging multi-sensor data, including LiDAR, cameras, and radar. However, traditional early or late fusion methods face challenges such as high bandwidth and computational resources, which make it difficult to balance data transmission efficiency with the accuracy of perception of the surrounding environment, especially for the detection of smaller objects such as pedestrians. To address these challenges, this paper proposes a novel cooperative perception framework based on two-stage intermediate-level sensor feature fusion specifically designed for complex traffic scenarios where pedestrians and vehicles coexist. In such scenarios, the model demonstrates superior performance in detecting small objects like pedestrians compared to mainstream perception methods while also improving the cooperative perception accuracy for medium and large objects such as vehicles. Furthermore, to thoroughly validate the reliability of the proposed model, we conducted both qualitative and quantitative experiments on mainstream simulated and real-world datasets. The experimental results demonstrate that our approach outperforms state-of-the-art perception models in terms of mAP, achieving up to a 4.1% improvement in vehicle detection accuracy and a remarkable 29.2% enhancement in pedestrian detection accuracy., This work was partially supported by the Spanish Research Agency (AEI) under project PID2020-117142GB-I00 of call MCIN/AEI/10.13039/501100011033 and by the 6G-EWOC project of the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101139182., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené-Mola, Jordi
  • Ferrer Ferrer, Mar
  • Gregorio López, Eduard
  • Blok, Pieter
  • Hemming, Jochen
  • Morros Rubió, Josep Ramon|||0000-0002-1395-487X
  • Rosell Polo, Joan R.
  • Vilaplana Besler, Verónica|||0000-0001-6924-9961
  • Ruiz Hidalgo, Javier|||0000-0001-6774-685X
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

A dual network for super-resolution and semantic segmentation of sentinel-2 imagery

  • Abadal Lloret, Sauc
  • Salgueiro Romero, Luis Fernando|||0000-0003-4048-8330
  • Marcello Ruiz, Javier
  • Vilaplana Besler, Verónica|||0000-0001-6924-9961
There is a growing interest in the development of automated data processing workflows that provide reliable, high spatial resolution land cover maps. However, high-resolution remote sensing images are not always affordable. Taking into account the free availability of Sentinel-2 satellite data, in this work we propose a deep learning model to generate high-resolution segmentation maps from low-resolution inputs in a multi-task approach. Our proposal is a dual-network model with two branches: the Single Image Super-Resolution branch, that reconstructs a high-resolution version of the input image, and the Semantic Segmentation Super-Resolution branch, that predicts a high-resolution segmentation map with a scaling factor of 2. We performed several experiments to find the best architecture, training and testing on a subset of the S2GLC 2017 dataset. We based our model on the DeepLabV3+ architecture, enhancing the model and achieving an improvement of 5% on IoU and almost 10% on the recall score. Furthermore, our qualitative results demonstrate the effectiveness and usefulness of the proposed approach., This work has been supported by the Spanish Research Agency (AEI) under project PID2020-117142GB-I00 of the call MCIN/AEI/10.13039/501100011033. L.S. would like to acknowledge the BECAL (Becas Carlos Antonio López) scholarship for the financial support., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Exploring hyperfine coupling in molecular qubits

  • Cardona Olives, Joan
  • Solé, Àlex|||0000-0002-2071-9317
  • Mella Páez, Pablo
  • Aravena Ponce, Daniel
  • Ruiz Hidalgo, Javier|||0000-0001-6774-685X
  • Gómez Coca, Silvia
  • Ruiz Sabin, Eliseo
Molecular qubits represent a promising avenue for advancing quantum sensing and computing technologies, yet significant challenges remain in optimising their performance. Hyperfine coupling has a critical influence on molecular qubit properties. While previous studies have exhaustively investigated this phenomenon, a comprehensive understanding of the underlying mechanisms across different systems remains elusive. A benchmark test was performed using DFT to assess which methodology worked best to accurately predict hyperfine coupling constants in molecular qubits predominantly composed of VIV and CuII. We systematically analysed the decomposition of hyperfine coupling and examined how variations in coordination sphere and molecular geometry impact dipolar, isotropic and spin–orbit contributions. By modelling diverse systems, we demonstrate how molecular design can fine-tune hyperfine coupling contributions, either minimising overall interaction or enhancing coupling along specific axes. This study provides useful insights into the structure–property relationships governing hyperfine coupling mechanisms and assesses the accuracy of different choices of density functional, basis sets and relativistic corrections in the prediction of hyperfine coupling constants., Financial support from Ministerio de Ciencia e Innovación (project PID2020-117142GB-I00, PID2021-122464NB-I00, TED2021-129593 B-I00, CNS2023-144561 and Maria de Maeztu CEX2021-001202-M) is acknowledged. We also acknowledge the Generalitat de Catalunya for the 2021-SGR-00286 grant. E. R. also acknowledges the Generalitat de Catalunya for an ICREA Academia grant. We thank CSUC for the computational resources. P. M. thanks Universidad de Santiago de Chile, USACH, Project DICYT code 022242AP_Postdoc., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené Mola, Jordi
  • Ferrer Ferrer, Mar
  • Hemming, Jochen
  • van Dalfsen, Pieter
  • de Hoog, Dirk
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan R.
  • Morros Rubió, Josep Ramon|||0000-0002-1395-487X
  • Vilaplana Besler, Verónica|||0000-0001-6924-9961
  • Ruiz Hidalgo, Javier|||0000-0001-6774-685X
  • Gregorio López, Eduard
Refers to: Gené-Mola, J. [et al.]. Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation. "Computers and electronics in agriculture", Juny 2023, vol. 209, article 107854. https://doi.org/10.1016/j.compag.2023.107854 http://hdl.handle.net/2117/387035, The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation” [1]., This work was partly funded by the Departament de Recerca i Universitats de la Generali- tat de Catalunya (grant 2021 LLAV 0 0 088 ), the Spanish Ministry of Science, Innovation and Uni- versities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/50110 0 011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi GenéMola was supported by the Spanish Ministry of Universities through a Mar- garita Salas postdoctoral grant funded by the European Union - NextGenerationEU., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Simultaneous fruit detection and size estimation using multitask deep neural networks

  • Ferrer Ferrer, Mar
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
  • Vilaplana Besler, Verónica
  • Morros Rubió, Josep Ramon
  • Gené-Mola, Jordi
The measurement of fruit size is of great interest to estimate the yield and predict the harvest resources in advance. This work proposes a novel technique for in-field apple detection and measurement based on Deep Neural Networks. The proposed framework was trained with RGB-D data and consists of an end-to-end multitask Deep Neural Network architecture specifically designed to perform the following tasks: 1) detection and segmentation of each fruit from its surroundings; 2) estimation of the diameter of each detected fruit. The methodology was tested with a total of 15,335 annotated apples at different growth stages, with diameters varying from 27 mm to 95 mm. Fruit detection results reported an F1-score for apple detection of 0.88 and a mean absolute error of diameter estimation of 5.64 mm. These are state-of-the-art results with the additional advantages of: a) using an end-to-end multitask trainable network; b) an efficient and fast inference speed; and c) being based on RGB-D data which can be acquired with affordable depth cameras. On the contrary, the main disadvantage is the need of annotating a large amount of data with fruit masks and diameter ground truth to train the model. Finally, a fruit visibility analysis showed an improvement in the prediction when limiting the measurement to apples above 65% of visibility (mean absolute error of 5.09 mm). This suggests that future works should develop a method for automatically identifying the most visible apples and discard the prediction of highly occluded fruits., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00[PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gene Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union-NextGenerationEU., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A Cartesian encoding graph neural network for crystal structure property prediction: application to thermal ellipsoid estimation

  • Solé, Àlex
  • Mosella Montoro, Albert
  • Cardona Olives, Joan
  • Gómez Coca, Silvia
  • Aravena Ponce, Daniel
  • Ruiz Sabin, Eliseo
  • Ruiz Hidalgo, Javier
In the diffraction resolution of crystal structures, thermal ellipsoids are a critical parameter that is usually more difficult to determine than atomic positions. These ellipsoids are quantified through Anisotropic Displacement Parameters (ADPs), which provide critical insights into atomic vibrations within crystalline structures. ADPs reflect the thermal behaviour and structural properties of crystal structures. However, traditional methods to compute ADPs are computationally intensive. This paper presents CartNet, a novel graph neural network (GNN) architecture designed to predict properties of crystal structures efficiently by encoding the atomic structural geometry to the Cartesian axes and the temperature of the crystal structure. Additionally, CartNet employs a neighbour equalization technique for message passing to help emphasise the covalent and contact interactions and a novel Cholesky-based head to ensure valid ADP predictions. Furthermore, a rotational SO(3) data augmentation technique has been proposed during the training phase to generalize unseen rotations. To corroborate this procedure, an ADP dataset with over 200 000 experimental crystal structures from the Cambridge Structural Database (CSD) has been curated. The model significantly reduces computational costs and outperforms existing previously reported methods for ADP prediction by 10.87%, while demonstrating a 34.77% improvement over the tested theoretical computation methods. Moreover, we have employed CartNet for other already known datasets that included different material properties, such as formation energy, band gap, total energy, energy above the convex hull, bulk moduli, and shear moduli. The proposed architecture outperformed previously reported methods by 7.71% in the JARVIS dataset and 13.16% in the Materials Project dataset, proving CarNet's capability to achieve state-of-the-art results in several tasks. The project website with online demo available at: https://www.ee.ub.edu/cartnet., Financial support from Ministerio de Ciencia e Innovación (project PID2020-117142GB-I00, PID2021-122464NB-I00, TED2021-129593B-I00, CNS2023-144561 and Maria de Maeztu CEX2021-001202-M) is acknowledged. We also acknowledge the Generalitat de Catalunya for the 2021-SGR-00286 grant. E. R. also acknowledges the Generalitat de Catalunya for an ICREA Academia grant., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Adaptive fusion of LiDAR features for 3D object detection in autonomous driving

  • Wang, Mingrui
  • Li, Dongjie
  • Casas Pla, Josep Ramon
  • Ruiz Hidalgo, Javier
In the field of autonomous driving, cooperative perception through vehicle-to-vehicle communication significantly enhances environmental understanding by leveraging multi-sensor data, including LiDAR, cameras, and radar. However, traditional early or late fusion methods face challenges such as high bandwidth and computational resources, which make it difficult to balance data transmission efficiency with the accuracy of perception of the surrounding environment, especially for the detection of smaller objects such as pedestrians. To address these challenges, this paper proposes a novel cooperative perception framework based on two-stage intermediate-level sensor feature fusion specifically designed for complex traffic scenarios where pedestrians and vehicles coexist. In such scenarios, the model demonstrates superior performance in detecting small objects like pedestrians compared to mainstream perception methods while also improving the cooperative perception accuracy for medium and large objects such as vehicles. Furthermore, to thoroughly validate the reliability of the proposed model, we conducted both qualitative and quantitative experiments on mainstream simulated and real-world datasets. The experimental results demonstrate that our approach outperforms state-of-the-art perception models in terms of mAP, achieving up to a 4.1% improvement in vehicle detection accuracy and a remarkable 29.2% enhancement in pedestrian detection accuracy., This work was partially supported by the Spanish Research Agency (AEI) under project PID2020-117142GB-I00 of call MCIN/AEI/10.13039/501100011033 and by the 6G-EWOC project of the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101139182., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Hemming, Jochen
  • van Dalfsen, Pieter
  • de Hoog, Dirk
  • Sanz-Cortiella, Ricardo
  • Rosell-Polo, Joan R.
  • Morros, Josep Ramon
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation” [1]., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A closer look at referring expressions for video object segmentation

  • Bellver Bueno, Míriam
  • Ventura Royo, Carles
  • Silberer, Carina
  • Kazakos, Ioannis
  • Torres Viñals, Jordi
  • Giró Nieto, Xavier
The task of Language-guided Video Object Segmentation (LVOS) aims at generating binary masks for an object referred by a linguistic expression. When this expression unambiguously describes an object in the scene, it is named referring expression (RE). Our work argues that existing benchmarks used for LVOS are mainly composed of trivial cases, in which referents can be identified with simple phrases. Our analysis relies on a new categorization of the referring expressions in the DAVIS-2017 and Actor-Action datasets into trivial and non-trivial REs, where the non-trivial REs are further annotated with seven RE semantic categories. We leverage these data to analyze the performance of RefVOS, a novel neural network that obtains competitive results for the task of language-guided image segmentation and state of the art results for LVOS. Our study indicates that the major challenges for the task are related to understanding motion and static actions., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was partially supported by the projects PID2019-107255GB-C22 and PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033 Spanish Ministry of Science, and the grant 2017-SGR-1414 of the Government of Catalonia. This work was also partially supported by the project RTI2018-095232-B-C22 funded by the Spanish Ministry of Science, Innovation and Universities., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené-Mola, Jordi
  • Ferrer Ferrer, Mar
  • Gregorio López, Eduard
  • Blok, Pieter
  • Hemming, Jochen
  • Morros Rubió, Josep Ramon
  • Rosell Polo, Joan R.
  • Vilaplana Besler, Verónica
  • Ruiz Hidalgo, Javier
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené Mola, Jordi
  • Ferrer Ferrer, Mar
  • Gregorio López, Eduard
  • Blok, Pieter
  • Hemming, Jochen
  • Morros Rubió, Josep Ramon
  • Rosell Polo, Joan Ramon
  • Vilaplana Besler, Verónica
  • Ruiz Hidalgo, Javier
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for sim ultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gen´e Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené Mola, Jordi
  • Ferrer Ferrer, Mar
  • Hemming, Jochen
  • van Dalfsen, Pieter
  • de Hoog, Dirk
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan R.
  • Morros Rubió, Josep Ramon
  • Vilaplana Besler, Verónica
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
Refers to: Gené-Mola, J. [et al.]. Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation. "Computers and electronics in agriculture", Juny 2023, vol. 209, article 107854. https://doi.org/10.1016/j.compag.2023.107854 http://hdl.handle.net/2117/387035, The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation” [1]., This work was partly funded by the Departament de Recerca i Universitats de la Generali- tat de Catalunya (grant 2021 LLAV 0 0 088 ), the Spanish Ministry of Science, Innovation and Uni- versities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/50110 0 011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi GenéMola was supported by the Spanish Ministry of Universities through a Mar- garita Salas postdoctoral grant funded by the European Union - NextGenerationEU., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Exploring hyperfine coupling in molecular qubits

  • Cardona Olives, Joan
  • Solé, Àlex
  • Mella Páez, Pablo
  • Aravena Ponce, Daniel
  • Ruiz Hidalgo, Javier
  • Gómez Coca, Silvia
  • Ruiz Sabin, Eliseo
Molecular qubits represent a promising avenue for advancing quantum sensing and computing technologies, yet significant challenges remain in optimising their performance. Hyperfine coupling has a critical influence on molecular qubit properties. While previous studies have exhaustively investigated this phenomenon, a comprehensive understanding of the underlying mechanisms across different systems remains elusive. A benchmark test was performed using DFT to assess which methodology worked best to accurately predict hyperfine coupling constants in molecular qubits predominantly composed of VIV and CuII. We systematically analysed the decomposition of hyperfine coupling and examined how variations in coordination sphere and molecular geometry impact dipolar, isotropic and spin–orbit contributions. By modelling diverse systems, we demonstrate how molecular design can fine-tune hyperfine coupling contributions, either minimising overall interaction or enhancing coupling along specific axes. This study provides useful insights into the structure–property relationships governing hyperfine coupling mechanisms and assesses the accuracy of different choices of density functional, basis sets and relativistic corrections in the prediction of hyperfine coupling constants., Financial support from Ministerio de Ciencia e Innovación (project PID2020-117142GB-I00, PID2021-122464NB-I00, TED2021-129593 B-I00, CNS2023-144561 and Maria de Maeztu CEX2021-001202-M) is acknowledged. We also acknowledge the Generalitat de Catalunya for the 2021-SGR-00286 grant. E. R. also acknowledges the Generalitat de Catalunya for an ICREA Academia grant. We thank CSUC for the computational resources. P. M. thanks Universidad de Santiago de Chile, USACH, Project DICYT code 022242AP_Postdoc., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Video-Based Fruit Detection and Tracking for Apple Counting and Mapping

  • Gené Mola, Jordi
  • Felip Pomés, Marc
  • Net Barnés, Francesc
  • Morros Rubió, Josep Ramon
  • Miranda, Juan Carlos
  • Arnó Satorra, Jaume
  • Asin Jones, Luis
  • Lordan Sanahuja, Jaume
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
Automatic fruit counting systems have garnered interest from farmers and agronomists to monitor fruit production, predict yields in advance, and identify production variability across orchards. However, accurately counting fruits poses challenges, particularly due to occlusions. This study proposes a multi-view sensing approach using continuous motion videos captured by a camera moved along the row of trees, followed by fruit detection in all video frames and application of Multi-Object Tracking (MOT) algorithms to prevent double-counting. Three tracking methods, namely SORT, DeepSORT, and ByteTrack, are compared for fruit counting using the YOLOv5x object detector. The methodology is applied to map fruit production in an experimental apple orchard at two different dates: four weeks and one week before harvest. The results demonstrate that ByteTrack (MOTA=0.682; IDF1=0.837; HOTA=0.689) outperforms SORT and DeepSORT, indicating its superior tracking performance. Computational efficiency analysis reveals similar processing times between SORT and ByteTrack (about 15 ms), while DeepSORT requires significantly more processing time per image (128 ms). Fruit counting evaluation shows reasonably accurate yield predictions on both dates, with reduced errors and improved performance closer to the harvest date (MAPE=7.47 %; R2=0.70). The system proves effective in estimating orchard fruit production using computer vision technology, offering valuable insights for yield forecasting. These findings contribute to optimizing fruit production and supporting precision agriculture practices. The code and the dataset have been made publicly available and a video visualization of results is accessible at http://www.grap.udl.cat/en/publications/video_fruit_counting., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda’s pre doctoral fellowship (2020 FI_B 00586).
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Simultaneous fruit detection and size estimation using multitask deep neural networks

  • Ferrer-Ferrer, Mar
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
  • Vilaplana, Verónica
  • Morros, Josep-Ramon
  • Gené-Mola, Jordi
The measurement of fruit size is of great interest to estimate the yield and predict the harvest resources in advance. This work proposes a novel technique for in-field apple detection and measurement based on Deep Neural Networks. The proposed framework was trained with RGB-D data and consists of an end-to-end multitask Deep Neural Network architecture specifically designed to perform the following tasks: 1) detection and segmentation of each fruit from its surroundings; 2) estimation of the diameter of each detected fruit. The methodology was tested with a total of 15,335 annotated apples at different growth stages, with diameters varying from 27 mm to 95 mm. Fruit detection results reported an F1-score for apple detection of 0.88 and a mean absolute error of diameter estimation of 5.64 mm. These are state-of-the-art results with the additional advantages of: a) using an end-to-end multitask trainable network; b) an efficient and fast inference speed; and c) being based on RGB-D data which can be acquired with affordable depth cameras. On the contrary, the main disadvantage is the need of annotating a large amount of data with fruit masks and diameter ground truth to train the model. Finally, a fruit visibility analysis showed an improvement in the prediction when limiting the measurement to apples above 65% of visibility (mean absolute error of 5.09 mm). This suggests that future works should develop a method for automatically identifying the most visible apples and discard the prediction of highly occluded fruits., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00[PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142 GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

DASPack: controlled data compression for distributed acoustic sensing

  • Seguí Ugalde, Aleix
  • Ugalde Aguirre, Arantza
  • Fichtner, Andreas
  • Ventosa Rahuet, Sergi
  • Morros Rubió, Josep Ramon
We present DASPack, a high-performance, open-source compression tool specifically de- signed for distributed acoustic sensing (DAS) data. As DAS becomes a key technology for real-time, high-density and long-range monitoring in fields such as geophysics, infrastructure surveillance and environmental sensing, the volume of collected data is rapidly increasing. Large-scale DAS deployments already generate hundreds of terabytes and are expected to increase in the coming years, making long-term storage a major challenge. Despite this urgent need, few compression methods have proven to be both practical and scalable in real-world scenarios. DASPack is a fully operational solution that consistently outperforms existing tech- niques for DAS data. It enables both controlled lossy and lossless compression by allowing users to choose the maximum absolute difference per datum between the original and com- pressed data. The compression pipeline combines wavelet transforms, linear predictive coding, and entropy coding to optimise efficiency. Our method achieves up to 3 × file size reductions for strain and strain rate data in lossless mode across diverse data sets. In lossy mode, compres- sion improves to 6 × with near-perfect signal fidelity, and up to 10 × is reached with acceptable signal degradation. It delivers fast throughput (100–200 MB s −1 using a single-thread and up to 750 MB s −1 using 8-threads), enabling real-time deployment even under high data rates. We validated its performance on 15 data sets from a variety of acquisition environments, demon- strating its speed, robustness and broad applicability. DASPack provides a practical foundation for long-term, sustainable DAS data management in large-scale monitoring networks., This work was supported in part by the ‘Severo Ochoa Centre of Excellence’ program (CEX2019-000928-S-21-2); the European Union NextGenerationEU/PRTR Program under projects PSI (PLEC2021-007875), TREMORS (CPP2021-008869), the CSIC contract SAFE with Telxius Cable España, S. L. (ref. 20233069) and the Spanish Ministry of Science, Innovation and Universities under DeeLight project (PID2020-117142GB-I00)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A dual network for super-resolution and semantic segmentation of sentinel-2 imagery

  • Abadal Lloret, Sauc
  • Salgueiro Romero, Luis Fernando
  • Marcello Ruiz, Javier
  • Vilaplana Besler, Verónica
There is a growing interest in the development of automated data processing workflows that provide reliable, high spatial resolution land cover maps. However, high-resolution remote sensing images are not always affordable. Taking into account the free availability of Sentinel-2 satellite data, in this work we propose a deep learning model to generate high-resolution segmentation maps from low-resolution inputs in a multi-task approach. Our proposal is a dual-network model with two branches: the Single Image Super-Resolution branch, that reconstructs a high-resolution version of the input image, and the Semantic Segmentation Super-Resolution branch, that predicts a high-resolution segmentation map with a scaling factor of 2. We performed several experiments to find the best architecture, training and testing on a subset of the S2GLC 2017 dataset. We based our model on the DeepLabV3+ architecture, enhancing the model and achieving an improvement of 5% on IoU and almost 10% on the recall score. Furthermore, our qualitative results demonstrate the effectiveness and usefulness of the proposed approach., This work has been supported by the Spanish Research Agency (AEI) under project PID2020-117142GB-I00 of the call MCIN/AEI/10.13039/501100011033. L.S. would like to acknowledge the BECAL (Becas Carlos Antonio López) scholarship for the financial support., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Single-image super-resolution of sentinel-2 low resolution bands with residual dense convolutional neural networks

  • Salgueiro Romero, Luis Fernando
  • Marcello Ruiz, Javier
  • Vilaplana Besler, Verónica
Sentinel-2 satellites have become one of the main resources for Earth observation images because they are free of charge, have a great spatial coverage and high temporal revisit. Sentinel-2 senses the same location providing different spatial resolutions as well as generating a multi-spectral image with 13 bands of 10, 20, and 60 m/pixel. In this work, we propose a single-image super-resolution model based on convolutional neural networks that enhances the low-resolution bands (20 m and 60 m) to reach the maximal resolution sensed (10 m) at the same time, whereas other approaches provide two independent models for each group of LR bands. Our proposed model, named Sen2-RDSR, is made up of Residual in Residual blocks that produce two final outputs at maximal resolution, one for 20 m/pixel bands and the other for 60 m/pixel bands. The training is done in two stages, first focusing on 20 m bands and then on the 60 m bands. Experimental results using six quality metrics (RMSE, SRE, SAM, PSNR, SSIM, ERGAS) show that our model has superior performance compared to other state-of-the-art approaches, and it is very effective and suitable as a preliminary step for land and coastal applications, as studies involving pixel-based classification for Land-Use-Land-Cover or the generation of vegetation indices., This work was funded by the Spanish Agencia Estatal de Investigación (AEI) under projects ARTEMISAT-2 (CTM2016-77733-R) and PID2020-117142GB-I00 of the call MCIN/AEI/10.13039/501100011033)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené Mola, Jordi
  • Ferrer Ferrer, Mar
  • Hemming, Jochen
  • Dalfsen, Pieter van
  • Hoog, Dirk de
  • Sanz Cortiella, Ricardo
  • Rosell Polo, Joan Ramon
  • Vilaplana Besler, Verónica
  • Ruiz Hidalgo, Javier
  • Gregorio López, Eduard
The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation”., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088 ), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition.
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Gregorio, Eduard
  • Blok, Pieter M.
  • Hemming, Jochen
  • Morros, Josep-Ramon
  • Rosell-Polo, Joan R.
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing, This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

SEG-ESRGAN: A multi-task network for super-resolution and semantic segmentation of remote sensing images

  • Salgueiro Romero, Luis Fernando
  • Marcello Ruiz, Javier
  • Vilaplana Besler, Verónica
The production of highly accurate land cover maps is one of the primary challenges in remote sensing, which depends on the spatial resolution of the input images. Sometimes, high-resolution imagery is not available or is too expensive to cover large areas or to perform multitemporal analysis. In this context, we propose a multi-task network to take advantage of the freely available Sentinel-2 imagery to produce a super-resolution image, with a scaling factor of 5, and the corresponding high-resolution land cover map. Our proposal, named SEG-ESRGAN, consists of two branches: the super-resolution branch, that produces Sentinel-2 multispectral images at 2 m resolution, and an encoder–decoder architecture for the semantic segmentation branch, that generates the enhanced land cover map. From the super-resolution branch, several skip connections are retrieved and concatenated with features from the different stages of the encoder part of the segmentation branch, promoting the flow of meaningful information to boost the accuracy in the segmentation task. Our model is trained with a multi-loss approach using a novel dataset to train and test the super-resolution stage, which is developed from Sentinel-2 and WorldView-2 image pairs. In addition, we generated a dataset with ground-truth labels for the segmentation task. To assess the super-resolution improvement, the PSNR, SSIM, ERGAS, and SAM metrics were considered, while to measure the classification performance, we used the IoU, confusion matrix and the F1-score. Experimental results demonstrate that the SEG-ESRGAN model outperforms different full segmentation and dual network models (U-Net, DeepLabV3+, HRNet and Dual_DeepLab), allowing the generation of high-resolution land cover maps in challenging scenarios using Sentinel-2 10 m bands., This work was funded by the Spanish Agencia Estatal de Investigación (AEI) under projects ARTEMISAT-2 (CTM 2016-77733-R), PID2020-117142GB-I00 and PID2020-116907RB-I00 (MCIN/AEI call 10.13039/501100011033). L.S. would like to acknowledge the BECAL (Becas Carlos Antonio López) scholarship for the financial support., Peer Reviewed, Postprint (published version)
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Gregorio, Eduard
  • Blok, Pieter M.
  • Hemming, Jochen
  • Morros, Josep-Ramon
  • Rosell-Polo, Joan R.
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing, This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

Simultaneous fruit detection and size estimation using multitask deep neural networks

  • Ferrer-Ferrer, Mar
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
  • Vilaplana, Verónica
  • Morros, Josep-Ramon
  • Gené-Mola, Jordi
The measurement of fruit size is of great interest to estimate the yield and predict the harvest resources in advance. This work proposes a novel technique for in-field apple detection and measurement based on Deep Neural Networks. The proposed framework was trained with RGB-D data and consists of an end-to-end multitask Deep Neural Network architecture specifically designed to perform the following tasks: 1) detection and segmentation of each fruit from its surroundings; 2) estimation of the diameter of each detected fruit. The methodology was tested with a total of 15,335 annotated apples at different growth stages, with diameters varying from 27 mm to 95 mm. Fruit detection results reported an F1-score for apple detection of 0.88 and a mean absolute error of diameter estimation of 5.64 mm. These are state-of-the-art results with the additional advantages of: a) using an end-to-end multitask trainable network; b) an efficient and fast inference speed; and c) being based on RGB-D data which can be acquired with affordable depth cameras. On the contrary, the main disadvantage is the need of annotating a large amount of data with fruit masks and diameter ground truth to train the model. Finally, a fruit visibility analysis showed an improvement in the prediction when limiting the measurement to apples above 65% of visibility (mean absolute error of 5.09 mm). This suggests that future works should develop a method for automatically identifying the most visible apples and discard the prediction of highly occluded fruits., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00[PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142 GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Hemming, Jochen
  • van Dalfsen, Pieter
  • de Hoog, Dirk
  • Sanz-Cortiella, Ricardo
  • Rosell-Polo, Joan R.
  • Morros, Josep Ramon
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation” [1]., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition., info:eu-repo/semantics/publishedVersion