Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 18
Encontrada(s) 1 página(s)

IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

Assessing automatic data processing algorithms for RGB-D cameras to predict fruit size and weight in apples

  • Miranda, Juan C.
  • Arnó, Jaume
  • Gené-Mola, Jordi
  • Lordan, Jaume
  • Asín, Luis
  • Gregorio, Eduard
Data acquired using an RGB-D Azure Kinect DK camera were used to assess different automatic algorithms to estimate the size, and predict the weight of non-occluded and occluded apples. The programming of the algorithms included: (i) the extraction of images of regions of interest (ROI) using manual delimitation of bounding boxes or binary masks; (ii) estimating the lengths of the major and minor geometric axes for the purpose of apple sizing; and (iii) predicting the final weight by allometric modelling. In addition to the use of bounding boxes, the algorithms also allowed other post-mask settings (circles, ellipses and rotated rectangles) to be implemented, and different depth options (distance between the RGB-D camera and the fruits detected) for subsequent sizing through the application of the thin lens theory. Both linear and nonlinear allometric models demonstrated the ability to predict apple weight with a high degree of accuracy (R2 greater than 0.942 and RMSE < 16 g). With respect to non-occluded apples, the best weight predictions were achieved using a linear allometric model including both the major and minor axes of the apples as predictors. The mean absolute percentage error (MAPE) ranged from 5.1% to 5.7% with respective RMSE of 11.09 g and 13.02 g, depending to whether circles, ellipses, or bounding boxes were used to adjust fruit shape. The results were therefore promising and open up the possibility of implementing reliable in-field apple measurements in real time. Importantly, final weight prediction error and intermediate size estimation errors (from sizing algorithms) interact but in a way that is not easily quantifiable when weight allometric models with implicit prediction error are used. In addition, allometric models should be reviewed when applied to other apple cultivars, fruit development stages or even for different fruit growth conditions depending on canopy management., This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017, SGR 646 and 2021 LLAV 00088), by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / ERDF (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project]) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / European Union NextGeneration / PRTR (grantTED2021-131871B-I00 [DIGIFRUIT project]). We would also like to thank the Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and the European Social Fund (ESF) for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

Fruit sizing using AI: A review of methods and challenges

  • Miranda, Juan C.
  • Gené-Mola, Jordi
  • Zude-Sasse, Manuela
  • Tsoulias, Nikos
  • Escolà, Alexandre
  • Arnó, Jaume
  • Rosell-Polo, Joan R.
  • Sanz-Cortiella, Ricardo
  • Martínez-Casasnovas, José A.
  • Gregorio, Eduard
Fruit size at harvest is an economically important variable for high-quality table fruit production in orchards and vineyards. In addition, knowing the number and size of the fruit on the tree is essential in the framework of precise production, harvest, and postharvest management. A prerequisite for analysis of fruit in a real-world environment is the detection and segmentation from background signal. In the last five years, deep learning convolutional neural network have become the standard method for automatic fruit detection, achieving F1-scores higher than 90 %, as well as real-time processing speeds. At the same time, different methods have been developed for, mainly, fruit size and, more rarely, fruit maturity estimation from 2D images and 3D point clouds. These sizing methods are focused on a few species like grape, apple, citrus, and mango, resulting in mean absolute error values of less than 4 mm in apple fruit. This review provides an overview of the most recent methodologies developed for in-field fruit detection/counting and sizing as well as few upcoming examples of maturity estimation. Challenges, such as sensor fusion, highly varying lighting conditions, occlusions in the canopy, shortage of public fruit datasets, and opportunities for research transfer, are discussed., This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646 and 2021 LLAV 00088) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / FEDER (grants RTI2018-094222-B-I00 [PAgFRUIT project] and PID2021-126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

LiDAR-derived indices and their relationship with productivity and oil quality attributes in high-density olive orchards

  • Sandonis-Pozo, L.
  • Rufat, Josep
  • Pascual, M.
  • Villar, J.M.
  • Arnó, J.
  • Escolà, A.
  • Rosell-Polo, J.R.
  • Martínez-Casasnovas, J.A.
Optimising the leaf area index (LAI) and light exposure are key aspects to improve yield and quality in high-density olive orchards. Traditionally, these characteristics are estimated using ceptometers, but this method is both labour intensive and dependent on light conditions, with limitations in effectiveness for precision horticulture management purposes. This study investigates the potential of four LiDAR-derived indices to: i) estimate the LAI and canopy light interception under different fertigation treatments, and ii) to assess their impact on yield and oil quality throughout a growing season in a high-density olive orchard. The treatments included two nitrogen dose rates (N0: 0 kg ha-¹ and N50: 50 kg ha-¹) combined with full irrigation (FI) and deficit irrigation (DI). The results provide strong evidence for the effectiveness of these indices in monitoring canopy response to treatments over time. The N50 and FI treatments increased the LAI and yield but led to poorer light attenuation within the canopy and lower oil quality. In contrast, N0 and DI promoted more light-permeable canopies and improved oil quality, with higher phenolic content, better oxidative stability and improved organoleptic oil attributes. These indices can improve the understanding of canopy light environments, help to optimise management practices, and ultimately improve the sustainability and profitability of intensive olive cultivation., This work was carried out with data acquired in the project IRTA2012–00059-C02–01 funded by Spain’s National Institute for Agricultural and Food Research and Technology, as well as in the projects AGL2010–22304-C04–03 and AGL2013–48297-C2–2-R funded by the Spanish Ministry of Economy and Competitiveness. The study was carried out under the framework of the grants RTI2018–094222-B-I00 (PAgFRUIT project) and PID2021–126648OB-I00 (PAgPROTECT project) through MCIN/AEI/10.13039/501100011033 and the European Union’s “ERDF, a way of making Europe”., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Gregorio, Eduard
  • Blok, Pieter M.
  • Hemming, Jochen
  • Morros, Josep-Ramon
  • Rosell-Polo, Joan R.
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing, This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Capítulo o parte de un libro (BookPart)

How can precision agriculture prescription maps contribute to the 50% pesticide reduction goal of the farm-to-fork strategy in viticulture

  • Llorens, J.
  • Román, C.
  • Escolà, A.
  • Gené-Mola, Jordi
  • Martínez-Casasnovas, J.A.
The Farm-to-Fork (F2F) strategy of the European Green Deal demands a 50 % reduction in the use of pesticides by 2030. In this work, several Precision Agriculture solutions were used in a vineyard spray application and the results were compared to a conventional treatment. Three different treatments were applied with the farm air-blast sprayer in a commercial 2.5 ha vineyard. Treatment 1 was the conventional farm application of a uniform 450 L ha-1 volume rate. In Treatment 2, the decision support system (DSS) DOSA3D was used to decide an adjusted uniform volume rate of 430 L ha-1. Treatment 3 took advantage of several PA techniques: a) electronic canopy characterization using a mobile terrestrial laser scanner to map canopy geometric parameters; b) management zones delineation; c) DSS to decide two different dose rates to create a prescription map (low vigour: 340 L ha-1 and high vigour: 430 L ha-1); d) variable-rate technology to execute the prescription map implemented in the farm sprayer. The spray deposit on leaves was the parameter used to assess differences between treatments. The statistical analysis did not show significant differences between treatments in high vigour vines. However, in low vigour vines, the mean deposits of both uniform treatments increased and showed significant differences with the variable-rate treatment., This work was partly funded by the Spanish Ministry of Science, Innovation and Universities through the projects PAgFRUIT (RTI2018-094222-B-I00) and PAgPROTECT (PID2021-126648OB-I00) (by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/acceptedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

A methodology for the realistic assessment of 3D point clouds of fruit trees in full 3D context

  • Lavaquiol-Colell, Bernat
  • Escolà, Alexandre
  • Sanz-Cortiella, Ricardo
  • Arnó, Jaume
  • Gené-Mola, Jordi
  • Gregorio, Eduard
  • Rosell-Polo, Joan R.
  • Ninot, Jérôme
  • Llorens-Calveras, Jordi
The aim of this paper is to address the lack of standard methodologies for the assessment of 3D point clouds. We present a methodology to realistically assess the accuracy of 3D point clouds, enabling the evaluation in a full 3D context rather than based on isolated points. Additionally, it introduces three significant innovations: a) it bridges the gap related to the unknown error of the reference ground-truth point cloud; b) it provides separate metrics for location error and reconstruction error; and c) it introduces a procedure to compute the location error that eliminates the bias in the selection of point-pair picking between the DGT points and their corresponding pairs in the point cloud being assessed. The geometry and structure of trees are related to the vegetative parameters and productivity in fruit orchards. In consequence, obtaining a precise and accurate geometric characterization of canopies is of interest for implementing site-specific management strategies that optimize input rates and minimize the costs and environmental risks of agricultural operations. Among the different sensing technologies, sensors based on the principle of light detection and ranging (LiDAR) have emerged as the primary choice for accurate geometric characterization of orchards. However, to make informed orchard management decisions based on LiDAR-derived geometric and structural data, it is essential to assess the accuracy of LiDAR-based scanning systems. Unfortunately, there is currently a lack of standard methodologies to evaluate the accuracy of LiDAR-based systems in agricultural environments. This research paper presents a novel methodology to assess the location error and the reconstruction error of 3D point clouds in full 3D context. The methodology involves comparing LiDAR-derived point clouds to an accurate high-resolution 3D digital ground truth (DGT) obtained using digital photogrammetric techniques. One of the main difficulties when using a reference point cloud to assess point cloud errors is the selection of the points to be compared so that they can be considered as corresponding point pairs. When developing the methodology, four procedures of point pair selection and distance calculation were compared. The best performing procedure was selected and proposed as a standard for accuracy assessment of 3D point clouds. The proposed procedure minimizes the error attributed to the selection of the corresponding point pairs between the assessed point cloud and the reference DGT point cloud. Subsequently, the proposed methodology was tested and validated by assessing the accuracy of 46 different point clouds. The conclusions regarding the accuracy, applicability, and practical utility of the proposed methodology are supported by the determination of reconstruction errors and location errors in 46 point clouds obtained with the 3 different MTLS systems operated with different settings. The proposed methodology will be very useful for scanning system manufacturers, researchers, advisors and eventually advanced farmers to quantify the errors committed when characterizing tree canopies. This is crucial to enable accurate management operations in the framework of Precision Agriculture based on canopy variability. Furthermore, the methodology is expected to facilitate the design of new applications requiring high accuracy to be implemented in the near future., The present study is part of the PAgPROTECT project PID2021-126648OB-I00 funded by the Spanish Ministry of Science and Innovation/AEI/10.13039/501100011033/FEDER, UE. This work was also partly funded by the Spanish Ministry of Science and Innovation/AEI/10.13039/501100011033/European Union NextGeneration/PRTR (grant TED2021-131871B-I00 [DIGIFRUIT project])., info:eu-repo/semantics/acceptedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

Simultaneous fruit detection and size estimation using multitask deep neural networks

  • Ferrer-Ferrer, Mar
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
  • Vilaplana, Verónica
  • Morros, Josep-Ramon
  • Gené-Mola, Jordi
The measurement of fruit size is of great interest to estimate the yield and predict the harvest resources in advance. This work proposes a novel technique for in-field apple detection and measurement based on Deep Neural Networks. The proposed framework was trained with RGB-D data and consists of an end-to-end multitask Deep Neural Network architecture specifically designed to perform the following tasks: 1) detection and segmentation of each fruit from its surroundings; 2) estimation of the diameter of each detected fruit. The methodology was tested with a total of 15,335 annotated apples at different growth stages, with diameters varying from 27 mm to 95 mm. Fruit detection results reported an F1-score for apple detection of 0.88 and a mean absolute error of diameter estimation of 5.64 mm. These are state-of-the-art results with the additional advantages of: a) using an end-to-end multitask trainable network; b) an efficient and fast inference speed; and c) being based on RGB-D data which can be acquired with affordable depth cameras. On the contrary, the main disadvantage is the need of annotating a large amount of data with fruit masks and diameter ground truth to train the model. Finally, a fruit visibility analysis showed an improvement in the prediction when limiting the measurement to apples above 65% of visibility (mean absolute error of 5.09 mm). This suggests that future works should develop a method for automatically identifying the most visible apples and discard the prediction of highly occluded fruits., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00[PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142 GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Hemming, Jochen
  • van Dalfsen, Pieter
  • de Hoog, Dirk
  • Sanz-Cortiella, Ricardo
  • Rosell-Polo, Joan R.
  • Morros, Josep Ramon
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation” [1]., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition., info:eu-repo/semantics/publishedVersion
IRTA Pubpro. Open Digital Archive
Publicación Artículo científico (JournalArticle)

AKFruitYield: Modular benchmarking and video analysis software for Azure Kinect cameras for fruit size and fruit yield estimation in apple orchards

  • Miranda, Juan Carlos
  • Arnó, Jaume
  • Gené-Mola, Jordi
  • Fountas, Spyros
  • Gregorio, Eduard
AKFruitYield is a modular software that allows orchard data from RGB-D Azure Kinect cameras to be processed for fruit size and fruit yield estimation. Specifically, two modules have been developed: i) AK_SW_BENCHMARKER that makes it possible to apply different sizing algorithms and allometric yield prediction models to manually labelled color and depth tree images; and ii) AK_VIDEO_ANALYSER that analyses videos on which to automatically detect apples, estimate their size and predict yield at the plot or per hectare scale using the appropriate algorithms. Both modules have easy-to-use graphical interfaces and provide reports that can subsequently be used by other analysis tools., This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646) and by the Spanish Ministry of Science and Innovation/AEI/10.13039/501100011033/ERDF (grant RTI2018–094222-B-I00 [PAgFRUIT project] and PID2021–126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda's pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. The authors would also like to thank the Institut de Recerca i Tecnologia Agroalimentàries (IRTA) for allowing the use of their experimental fields, and in particular Dr. Luís Asín and Dr. Jaume Lordán who have contributed to the success of this work., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

LiDAR-derived indices and their relationship with productivity and oil quality attributes in high-density olive orchards

  • Sandonis-Pozo, L.
  • Rufat, Josep
  • Pascual, M.
  • Villar, J.M.
  • Arnó, J.
  • Escolà, A.
  • Rosell-Polo, J.R.
  • Martínez-Casasnovas, J.A.
Optimising the leaf area index (LAI) and light exposure are key aspects to improve yield and quality in high-density olive orchards. Traditionally, these characteristics are estimated using ceptometers, but this method is both labour intensive and dependent on light conditions, with limitations in effectiveness for precision horticulture management purposes. This study investigates the potential of four LiDAR-derived indices to: i) estimate the LAI and canopy light interception under different fertigation treatments, and ii) to assess their impact on yield and oil quality throughout a growing season in a high-density olive orchard. The treatments included two nitrogen dose rates (N0: 0 kg ha-¹ and N50: 50 kg ha-¹) combined with full irrigation (FI) and deficit irrigation (DI). The results provide strong evidence for the effectiveness of these indices in monitoring canopy response to treatments over time. The N50 and FI treatments increased the LAI and yield but led to poorer light attenuation within the canopy and lower oil quality. In contrast, N0 and DI promoted more light-permeable canopies and improved oil quality, with higher phenolic content, better oxidative stability and improved organoleptic oil attributes. These indices can improve the understanding of canopy light environments, help to optimise management practices, and ultimately improve the sustainability and profitability of intensive olive cultivation., This work was carried out with data acquired in the project IRTA2012–00059-C02–01 funded by Spain’s National Institute for Agricultural and Food Research and Technology, as well as in the projects AGL2010–22304-C04–03 and AGL2013–48297-C2–2-R funded by the Spanish Ministry of Economy and Competitiveness. The study was carried out under the framework of the grants RTI2018–094222-B-I00 (PAgFRUIT project) and PID2021–126648OB-I00 (PAgPROTECT project) through MCIN/AEI/10.13039/501100011033 and the European Union’s “ERDF, a way of making Europe”., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

AmodalAppleSize_RGB-D dataset: RGB-D images of apple trees annotated with modal and amodal segmentation masks for fruit detection, visibility and size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Hemming, Jochen
  • van Dalfsen, Pieter
  • de Hoog, Dirk
  • Sanz-Cortiella, Ricardo
  • Rosell-Polo, Joan R.
  • Morros, Josep Ramon
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
The present dataset comprises a collection of RGB-D apple tree images that can be used to train and test computer vision-based fruit detection and sizing methods. This dataset encompasses two distinct sets of data obtained from a Fuji and an Elstar apple orchards. The Fuji apple orchard sub-set consists of 3925 RGB-D images containing a total of 15,335 apples annotated with both modal and amodal apple segmentation masks. Modal masks denote the visible portions of the apples, whereas amodal masks encompass both visible and occluded apple regions. Notably, this dataset is the first public resource to incorporate on-tree fruit amodal masks. This pioneering inclusion addresses a critical gap in existing datasets, enabling the development of robust automatic fruit sizing methods and accurate fruit visibility estimation, particularly in the presence of partial occlusions. Besides the fruit segmentation masks, the dataset also includes the fruit size (calliper) ground truth for each annotated apple. The second sub-set comprises 2731 RGB-D images capturing five Elstar apple trees at four distinct growth stages. This sub-set includes mean diameter information for each tree at every growth stage and serves as a valuable resource for evaluating fruit sizing methods trained with the first sub-set. The present data was employed in the research paper titled “Looking behind occlusions: a study on amodal segmentation for robust on-tree apple fruit size estimation” [1]., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). Data presented in this paper is also part of a Public Private Partnership project Precisie Tuinbouw, WP Fruit 4.0 (PPS KV 1604-025) and financed by Topsector Tuinbouw & Uitgangsmateriaal and various private companies. The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

A methodology for the realistic assessment of 3D point clouds of fruit trees in full 3D context

  • Lavaquiol-Colell, Bernat
  • Escolà, Alexandre
  • Sanz-Cortiella, Ricardo
  • Arnó, Jaume
  • Gené-Mola, Jordi
  • Gregorio, Eduard
  • Rosell-Polo, Joan R.
  • Ninot, Jérôme
  • Llorens-Calveras, Jordi
The aim of this paper is to address the lack of standard methodologies for the assessment of 3D point clouds. We present a methodology to realistically assess the accuracy of 3D point clouds, enabling the evaluation in a full 3D context rather than based on isolated points. Additionally, it introduces three significant innovations: a) it bridges the gap related to the unknown error of the reference ground-truth point cloud; b) it provides separate metrics for location error and reconstruction error; and c) it introduces a procedure to compute the location error that eliminates the bias in the selection of point-pair picking between the DGT points and their corresponding pairs in the point cloud being assessed. The geometry and structure of trees are related to the vegetative parameters and productivity in fruit orchards. In consequence, obtaining a precise and accurate geometric characterization of canopies is of interest for implementing site-specific management strategies that optimize input rates and minimize the costs and environmental risks of agricultural operations. Among the different sensing technologies, sensors based on the principle of light detection and ranging (LiDAR) have emerged as the primary choice for accurate geometric characterization of orchards. However, to make informed orchard management decisions based on LiDAR-derived geometric and structural data, it is essential to assess the accuracy of LiDAR-based scanning systems. Unfortunately, there is currently a lack of standard methodologies to evaluate the accuracy of LiDAR-based systems in agricultural environments. This research paper presents a novel methodology to assess the location error and the reconstruction error of 3D point clouds in full 3D context. The methodology involves comparing LiDAR-derived point clouds to an accurate high-resolution 3D digital ground truth (DGT) obtained using digital photogrammetric techniques. One of the main difficulties when using a reference point cloud to assess point cloud errors is the selection of the points to be compared so that they can be considered as corresponding point pairs. When developing the methodology, four procedures of point pair selection and distance calculation were compared. The best performing procedure was selected and proposed as a standard for accuracy assessment of 3D point clouds. The proposed procedure minimizes the error attributed to the selection of the corresponding point pairs between the assessed point cloud and the reference DGT point cloud. Subsequently, the proposed methodology was tested and validated by assessing the accuracy of 46 different point clouds. The conclusions regarding the accuracy, applicability, and practical utility of the proposed methodology are supported by the determination of reconstruction errors and location errors in 46 point clouds obtained with the 3 different MTLS systems operated with different settings. The proposed methodology will be very useful for scanning system manufacturers, researchers, advisors and eventually advanced farmers to quantify the errors committed when characterizing tree canopies. This is crucial to enable accurate management operations in the framework of Precision Agriculture based on canopy variability. Furthermore, the methodology is expected to facilitate the design of new applications requiring high accuracy to be implemented in the near future., The present study is part of the PAgPROTECT project PID2021-126648OB-I00 funded by the Spanish Ministry of Science and Innovation/AEI/10.13039/501100011033/FEDER, UE. This work was also partly funded by the Spanish Ministry of Science and Innovation/AEI/10.13039/501100011033/European Union NextGeneration/PRTR (grant TED2021-131871B-I00 [DIGIFRUIT project])., info:eu-repo/semantics/acceptedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

AKFruitYield: Modular benchmarking and video analysis software for Azure Kinect cameras for fruit size and fruit yield estimation in apple orchards

  • Miranda, Juan Carlos
  • Arnó, Jaume
  • Gené-Mola, Jordi
  • Fountas, Spyros
  • Gregorio, Eduard
AKFruitYield is a modular software that allows orchard data from RGB-D Azure Kinect cameras to be processed for fruit size and fruit yield estimation. Specifically, two modules have been developed: i) AK_SW_BENCHMARKER that makes it possible to apply different sizing algorithms and allometric yield prediction models to manually labelled color and depth tree images; and ii) AK_VIDEO_ANALYSER that analyses videos on which to automatically detect apples, estimate their size and predict yield at the plot or per hectare scale using the appropriate algorithms. Both modules have easy-to-use graphical interfaces and provide reports that can subsequently be used by other analysis tools., This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646) and by the Spanish Ministry of Science and Innovation/AEI/10.13039/501100011033/ERDF (grant RTI2018–094222-B-I00 [PAgFRUIT project] and PID2021–126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda's pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. The authors would also like to thank the Institut de Recerca i Tecnologia Agroalimentàries (IRTA) for allowing the use of their experimental fields, and in particular Dr. Luís Asín and Dr. Jaume Lordán who have contributed to the success of this work., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Assessing automatic data processing algorithms for RGB-D cameras to predict fruit size and weight in apples

  • Miranda, Juan C.
  • Arnó, Jaume
  • Gené-Mola, Jordi
  • Lordan, Jaume
  • Asín, Luis
  • Gregorio, Eduard
Data acquired using an RGB-D Azure Kinect DK camera were used to assess different automatic algorithms to estimate the size, and predict the weight of non-occluded and occluded apples. The programming of the algorithms included: (i) the extraction of images of regions of interest (ROI) using manual delimitation of bounding boxes or binary masks; (ii) estimating the lengths of the major and minor geometric axes for the purpose of apple sizing; and (iii) predicting the final weight by allometric modelling. In addition to the use of bounding boxes, the algorithms also allowed other post-mask settings (circles, ellipses and rotated rectangles) to be implemented, and different depth options (distance between the RGB-D camera and the fruits detected) for subsequent sizing through the application of the thin lens theory. Both linear and nonlinear allometric models demonstrated the ability to predict apple weight with a high degree of accuracy (R2 greater than 0.942 and RMSE < 16 g). With respect to non-occluded apples, the best weight predictions were achieved using a linear allometric model including both the major and minor axes of the apples as predictors. The mean absolute percentage error (MAPE) ranged from 5.1% to 5.7% with respective RMSE of 11.09 g and 13.02 g, depending to whether circles, ellipses, or bounding boxes were used to adjust fruit shape. The results were therefore promising and open up the possibility of implementing reliable in-field apple measurements in real time. Importantly, final weight prediction error and intermediate size estimation errors (from sizing algorithms) interact but in a way that is not easily quantifiable when weight allometric models with implicit prediction error are used. In addition, allometric models should be reviewed when applied to other apple cultivars, fruit development stages or even for different fruit growth conditions depending on canopy management., This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017, SGR 646 and 2021 LLAV 00088), by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / ERDF (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project]) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / European Union NextGeneration / PRTR (grantTED2021-131871B-I00 [DIGIFRUIT project]). We would also like to thank the Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and the European Social Fund (ESF) for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Simultaneous fruit detection and size estimation using multitask deep neural networks

  • Ferrer-Ferrer, Mar
  • Ruiz-Hidalgo, Javier
  • Gregorio, Eduard
  • Vilaplana, Verónica
  • Morros, Josep-Ramon
  • Gené-Mola, Jordi
The measurement of fruit size is of great interest to estimate the yield and predict the harvest resources in advance. This work proposes a novel technique for in-field apple detection and measurement based on Deep Neural Networks. The proposed framework was trained with RGB-D data and consists of an end-to-end multitask Deep Neural Network architecture specifically designed to perform the following tasks: 1) detection and segmentation of each fruit from its surroundings; 2) estimation of the diameter of each detected fruit. The methodology was tested with a total of 15,335 annotated apples at different growth stages, with diameters varying from 27 mm to 95 mm. Fruit detection results reported an F1-score for apple detection of 0.88 and a mean absolute error of diameter estimation of 5.64 mm. These are state-of-the-art results with the additional advantages of: a) using an end-to-end multitask trainable network; b) an efficient and fast inference speed; and c) being based on RGB-D data which can be acquired with affordable depth cameras. On the contrary, the main disadvantage is the need of annotating a large amount of data with fruit masks and diameter ground truth to train the model. Finally, a fruit visibility analysis showed an improvement in the prediction when limiting the measurement to apples above 65% of visibility (mean absolute error of 5.09 mm). This suggests that future works should develop a method for automatically identifying the most visible apples and discard the prediction of highly occluded fruits., This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00[PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142 GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Fruit sizing using AI: A review of methods and challenges

  • Miranda, Juan C.
  • Gené-Mola, Jordi
  • Zude-Sasse, Manuela
  • Tsoulias, Nikos
  • Escolà, Alexandre
  • Arnó, Jaume
  • Rosell-Polo, Joan R.
  • Sanz-Cortiella, Ricardo
  • Martínez-Casasnovas, José A.
  • Gregorio, Eduard
Fruit size at harvest is an economically important variable for high-quality table fruit production in orchards and vineyards. In addition, knowing the number and size of the fruit on the tree is essential in the framework of precise production, harvest, and postharvest management. A prerequisite for analysis of fruit in a real-world environment is the detection and segmentation from background signal. In the last five years, deep learning convolutional neural network have become the standard method for automatic fruit detection, achieving F1-scores higher than 90 %, as well as real-time processing speeds. At the same time, different methods have been developed for, mainly, fruit size and, more rarely, fruit maturity estimation from 2D images and 3D point clouds. These sizing methods are focused on a few species like grape, apple, citrus, and mango, resulting in mean absolute error values of less than 4 mm in apple fruit. This review provides an overview of the most recent methodologies developed for in-field fruit detection/counting and sizing as well as few upcoming examples of maturity estimation. Challenges, such as sensor fusion, highly varying lighting conditions, occlusions in the canopy, shortage of public fruit datasets, and opportunities for research transfer, are discussed., This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646 and 2021 LLAV 00088) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / FEDER (grants RTI2018-094222-B-I00 [PAgFRUIT project] and PID2021-126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

  • Gené-Mola, Jordi
  • Ferrer-Ferrer, Mar
  • Gregorio, Eduard
  • Blok, Pieter M.
  • Hemming, Jochen
  • Morros, Josep-Ramon
  • Rosell-Polo, Joan R.
  • Vilaplana, Verónica
  • Ruiz-Hidalgo, Javier
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing, This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study., info:eu-repo/semantics/publishedVersion
Recercat. Dipósit de la Recerca de Catalunya
Publicación Capítulo o parte de un libro (BookPart)

How can precision agriculture prescription maps contribute to the 50% pesticide reduction goal of the farm-to-fork strategy in viticulture

  • Llorens, J.
  • Román, C.
  • Escolà, A.
  • Gené-Mola, Jordi
  • Martínez-Casasnovas, J.A.
The Farm-to-Fork (F2F) strategy of the European Green Deal demands a 50 % reduction in the use of pesticides by 2030. In this work, several Precision Agriculture solutions were used in a vineyard spray application and the results were compared to a conventional treatment. Three different treatments were applied with the farm air-blast sprayer in a commercial 2.5 ha vineyard. Treatment 1 was the conventional farm application of a uniform 450 L ha-1 volume rate. In Treatment 2, the decision support system (DSS) DOSA3D was used to decide an adjusted uniform volume rate of 430 L ha-1. Treatment 3 took advantage of several PA techniques: a) electronic canopy characterization using a mobile terrestrial laser scanner to map canopy geometric parameters; b) management zones delineation; c) DSS to decide two different dose rates to create a prescription map (low vigour: 340 L ha-1 and high vigour: 430 L ha-1); d) variable-rate technology to execute the prescription map implemented in the farm sprayer. The spray deposit on leaves was the parameter used to assess differences between treatments. The statistical analysis did not show significant differences between treatments in high vigour vines. However, in low vigour vines, the mean deposits of both uniform treatments increased and showed significant differences with the variable-rate treatment., This work was partly funded by the Spanish Ministry of Science, Innovation and Universities through the projects PAgFRUIT (RTI2018-094222-B-I00) and PAgPROTECT (PID2021-126648OB-I00) (by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU., info:eu-repo/semantics/acceptedVersion