ORB-Livox: A real-time dynamic system for fruit detection and localization

Accurate fruit localization plays an essential role in many agricultural tasks and is one of the most vital components of an autonomous harvesting robot. Although fruit localization in stationary conditions has been extensively studied, the method to perform fruit localization in dynamic conditions is rarely studied. Fruit localization under dynamic conditions is much more challenging, since movement will introduce continuously irregular ego-motion on visual sensors, which can lead to considerable accuracy degeneration or even system failure. However, fruit localization under dynamic conditions is an essential task since it can significantly improve the harvesting efficiency. In this study, a fruit localization method is proposed to perform accurate fruit localization under dynamic conditions. Firstly, an ORB-SLAM3 SLAM algorithm is utilized to estimate the camera's pose under dynamic conditions. Secondly, YOLO-V5 is used to identify apples in RGB images. Moreover, Lidar is used to measure the depth of each apple accurately. Finally, a clustering technique based on the Gaussian distribution is proposed to mitigate the accuracy loss under dynamic conditions to optimize apple positions. Our method is extensively evaluated in indoor and outdoor experiments. The results of indoor experiments show that the average localization error is 21.1 mm, with 90% of the errors are less than 30 mm. The experimental result in an orchard environment demonstrates that our proposed method can detect and localize apples under dynamic conditions in unstructured natural environments.