Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards

Abstract

Robotic harvesting of fruits in orchards is a challenging task, since high density and overlapping of fruits and branches can heavily impact the success rate of robotic harvesting. Therefore, the vision system is demanded to provide comprehensive information of the working environment to guide the manipulator and gripping system to successful detach the target fruits. In this study, a deep learning based one-stage detector DaSNet-V2 is developed to perform the multi-task vision sensing in the working environment of apple orchards. DaSNet-V2 combines the detection and instance segmentation of fruits and semantic segmentation of branch into a single network architecture. Meanwhile, a light-weight backbone network LW-net is utilised in the DaSNet-V2 model to improve the computational efficiency of the model. In the experiment, DaSNet-V2 is tested and evaluated on the RGB-D images of the orchard. From the experiment results, DaSNet-V2 with lightweight backbone achieves 0.844, 0.858, and 0.795 on the F 1 score of the detection, and mean intersection of union on the instance segmentation of fruits and semantic segmentation of branches, respectively. To provide a direct-viewing of the working environment in orchards, the obtained sensing results are illustrated by 3D visualisation . The robustness and efficiency of the DaSNet-V2 in detection and segmentation are validated by the experiments in the real-environment of apple orchard.

17 pages, 7 figures

References in corpus (4)

• Deep learning in agriculture: A survey
• Fully Convolutional Networks for Semantic Segmentation
• Real-Time Fruit Recognition and Grasping Estimation for Autonomous Apple Harvesting
• Single Pixel Reconstruction for One-stage Instance Segmentation

Cited by in corpus (11)

• Geometry-Aware Fruit Grasping Estimation for Robotic Harvesting in Orchards
• Visual Perception and Modelling in Unstructured Orchard for Apple Harvesting Robots
• Accurate Fruit Localisation for Robotic Harvesting using High Resolution LiDAR-Camera Fusion
• Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards
• Tomato Maturity Recognition with Convolutional Transformers
• A Novel Perception and Semantic Mapping Method for Robot Autonomy in Orchards
• Semantic Segmentation for Partially Occluded Apple Trees Based on Deep Learning
• Dataset and Performance Comparison of Deep Learning Architectures for Plum Detection and Robotic Harvesting
• Review of Fruit Tree Image Segmentation
• Horticultural Temporal Fruit Monitoring via 3D Instance Segmentation and Re-Identification using Colored Point Clouds
• Minimizing Labeling Effort for Tree Skeleton Segmentation using an Automated Iterative Training Methodology