A method for determining the optimal placement of litchi clusters using improved YOLACT and a distributed target fault-tolerance mechanism

Yuanhong Li, Jing Wang, Jiapeng Liao, Yubin Lan

Abstract


Abstract: At present, there is no efficient and accurate method for locating the litchi picking point.  Different from the grapes and tomatoes picking, litchi have lush leaves, thick and hard stems, and the biological characteristics (picking points) are random.  This paper proposes a fault-tolerant mechanism for distributed target picking.  This mechanism combines the morphological distribution characteristics of single litchi and the occluded targets completion method, and transforms the image processing problem into supervised learning and nonlinear regression question.  We researched the characteristics of litchi stems and growth laws, and divided the picking situation into two categories.  For the first time, we design the target fault-tolerant shearing path by utilizing the projection distribution of the normal vector of a single litchi onto the image coordinate system.  This approach addresses the challenge of litchi picking with irregular deviation angles caused by the influence of gravity.  To sum up, the distributed target fault-tolerance mechanism proposed in this paper combines the morphological characteristics of litchis and artificial intelligence technology, which fundamentally improves the positioning accuracy of litchi picking points and creates a common and intelligent picking positioning technology method for fruit agricultural robots.

Keywords: distributed, litchi, smart picking, mask, target fault-tolerance

DOI: 10.33440/j.ijpaa.20230601.217

 

Citation: Lia Y H, Wang J, Liao J P, and Lan Y B.  A method for determining the optimal placement of litchi clusters using improved YOLACT and a distributed target fault-tolerance mechanism.  Int J Precis Agric Aviat, 2023; 6(1): 23–32.

Full Text:

PDF

References


Xie, J., Jing, T., Chen, B., Peng, J., Zhang, X., He, P., Yin, H., Sun, D., Wang, W., Xiao, A. and Lyu, S. Method for Segmentation of Litchi Branches Based on the Improved DeepLabv3+. Agronomy, 2022, 12(11): 2812. https://doi.org/10.3390/agronomy12112812

Peng, H., Huang, B., Shao, Y., Li, Z., Zhang, C., Chen, Y. and Xiong, J. General improved SSD model for picking object recognition of multiple fruits in natural environment. Transactions of the Chinese Society of Agricultural Engineering, 2018, 34(16): 155–162.

Q. Zhu, R. Lu, J. Lu, F. Li. Research status and development trend of litchi picking machinery. Forestry Machinery and Woodworking Equipment, 2021, 49(08): 11–19. (in Chinese)

J. Li, Y. Tang, X. Zou, G. Lin, H. Wang. Detection of fruit-bearing branches and localization of litchi clusters for vision-based harvesting robots. IEEE Access, 8 (2020) 117746–117758.30. DOI: 10.1109/ ACCESS.2020.3005386

J. Xiong, R. Lin, Z. Liu, Z. He, L. Tang, Z. Yang, X. Zou. The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment. Biosystems Engineering, 2018, 166: 44–57.440. DOI?10.1016/j.biosystemseng.2017.11.005

J. Xiong, X. Zou, L. Chen, H. Peng, D. Wu, et al. Fruit recognition and positioning technology of litchi picking manipulator. Journal of Jiangsu University-Natural Science Edition, 2012, 33(1): 1–5. DOI: 10.3969/ j.issn.1671-7775.2012.01.001

Pérez-Zavala, R., Torres-Torriti, M., Cheein, F.A., Troni, G. A pattern recognition strategy for visual grape bunch detection in vineyards. Comput. Electron. Agric., 2018, 151, 136–149.

Wu, G., Zhu, Q., Huang, M., Guo, Y. and Qin, J. Automatic recognition of juicy peaches on trees based on 3D contour features and colour data. Biosystems Engineering, 2019, 188, pp.1–13.

Li, Y., Feng, Q., Liu, C., Xiong, Z., Sun, Y., Xie, F., Li, T. and Zhao, C. MTA-YOLACT: Multitask-aware network on fruit bunch identification for cherry tomato robotic harvesting. European Journal of Agronomy, 2023, 146, p.126812.

Tian, D., Han, Y., Wang, B., Guan, T., Gu, H. and Wei, W. Review of object instance segmentation based on deep learning. Journal of Electronic Imaging, 2022, 31(4): 041205-041205.

He, K., Gkioxari, G., Dollár, P. and Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.

Gong, T., Chen, K., Wang, X., Chu, Q., Zhu, F., Lin, D., Yu, N. and Feng, H. May. Temporal ROI align for video object recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(2): 1442–1450.

Li, Y., Qi, H., Dai, J., Ji, X. and Wei, Y. Fully convolutional instance-aware semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2359–2367.

Bolya, D., Zhou, C., Xiao, F. and Lee, Y. J. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9157–9166.

Leung, T. and Malik, J. Representing and recognizing the visual appearance of materials using three-dimensional textons. International journal of computer vision, 2001, 43, pp.29–44.

Sivic and Zisserman. October. Video Google: A text retrieval approach to object matching in videos. In Proceedings ninth IEEE international conference on computer vision, 2003, pp. 1470-1477. IEEE.

Kim, E., Kim, S., Seo, M. and Yoon, S. XProtoNet: diagnosis in chest radiography with global and local explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 15719-15728.

Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C. and Su, J. K. 2019. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32.

Zhang, Q. and Gao, G. Grasping point detection of randomly placed fruit cluster using adaptive morphology segmentation and principal component classification of multiple features. Ieee Access, 2019, 7, pp.158035-158050.

R. Bogue. Fruit picking robots: has their time come?, Industrial Robot:the international journal of robotics research and application.

Z. Li, F. Miao, Z. Yang, P. Chai, S. Yang. Factors affecting human hand grasp type in tomato fruit-picking: A statistical investigation for ergonomic development of harvesting robot. Computers and Electronics in Agriculture, 2019, 157, 90–97. DOI: 10.1016/j.compag.2018.12.047

H. Si, J. Lv, K. Lin, J. Wu, J. Chen. A review of application of computer vision in fruit picking robot, in: International Conference on Intelligent Computing. Communication & Devices, Springer, 2019, pp. 346–355. DOI: 10.1007/978-981-15-5887-0_50

J. Zhang. Target extraction of fruit picking robot vision system, in: Journal of Physics: Conference Series, Vol. 1423, IOP Publishing, 2019, p. 012061.

G. Wang, Y. Lan, H. Qi, P. Chen, A. Hewitt, Y. Han. Field evaluation of an unmanned aerial vehicle (uav) sprayer: effect of spray volume on deposition and the control of pests and disease in wheat. Pest Management Science, 2019, 75(6): 1546–1555. DOI: 10.1088/ 1742-6596/1423/1/012061

Y. Zhan, P. Chen, W. Xu, S. Chen, Y. Han, Y. Lan, G. Wang. Influence of the downwash airflow distribution characteristics of a plant protection uav on spray deposit distribution. Biosystems Engineering, 2022, 216, 32–45. DOI: 10.1016/j.biosystemseng.2022.01.016

N. Saranya, K. Srinivasan, S. Pravin Kumar, V. Rukkumani, R. Ramya,450 Fruit classification using traditional machine learning and deep learning approach, in: International Conference On Computational Vision and BioInspired Computing, Springer, 2019, pp. 79–89. DOI:10.1007/978-3-030-37218-7_10

J. Zhuang, C. Hou, Y. Tang, Y. He, Q. Guo, Z. Zhong, S. Luo. Computer vision-based localisation of picking points for automatic litchi harvesting applications towards natural scenarios. Biosystems Engineering, 2019, 187, 1–20. DOI: 10.1016/j.biosystemseng.2019. 08.016

K. Nagraj, G. Diwan, N. L, et al. Effect of fruit load on yield and quality of litchi (litchi chinensis sonn.). Journal of Pharmacognosy and Phytochemistry, 2019, 8(6): 1929–1931.

K. Kumar, K. Madhumala, S. Sahay. Response of different sources of potas-sium on fruit quality and fruit colour enhancement in litchi. Journal of Pharmacognosy and Phytochemistry, 2019, 8(6): 1990–1993.

L. Wang, Y. Zhao, S. Liu, Y. Li, S. Chen, Y. Lan. Precision detection of dense plums in orchards using the improved yolov4 model. Frontiers in Plant Science, 2022, 13, 839269–839269. DOI: 10.3389/fpls.2022. 839269

Zhang, Q. and Gao, G. Grasping point detection of randomly placed fruit cluster using adaptive morphology segmentation and principal component classification of multiple features. IEEE Access, 2019, 7, pp.158035–158050.

Barisoni, L., Lafata, K. J., Hewitt, S. M., Madabhushi, A. and Balis, U. G. Digital pathology and computational image analysis in nephropathology. Nature Reviews Nephrology, 2020, 16(11): 669–685.

Bolya, Daniel, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9157–9166. 2019.

Bharati, P. and Pramanik, A., 2020. Deep learning techniques—R-CNN to mask R-CNN: a survey. Computational Intelligence in Pattern Recognition: Proceedings of CIPR 2019, pp.657-668.

Gong, Yuqi, Xuehui Yu, Yao Ding, Xiaoke Peng, Jian Zhao, and Zhenjun Han. Effective fusion factor in FPN for tiny object detection. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1160-1168. 2021.

Chen, Chaofan, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K. Su. This looks like that: deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 2019, 32.

J. A. Villasenor Alva, E. G. Estrada. A generalization of shapiro–wilk’s test for multivariate normality. Communications in Statistics—Theory and Methods, 2009, 38(11): 1870–1883. DOI: 10.1080/ 03610920802474465

Y. Ge, Y. Xiong, G. L. Tenorio, P. J. From, Fruit localization and environment perception for strawberry harvesting robots. IEEE Access 7 (2019) 147642–147652. DOI: 10.1109/ACCESS.2019.2946369

K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.

Y. Yu, K. Zhang, L. Yang, D. Zhang. Fruit detection for strawberry harvest-ing robot in non-structural environment based on mask-rcnn. Computers and Electronics in Agriculture, 2019, 163: 104846. DOI: 10.1016/j.compag.2019.06.001

X. Liu, D. Zhao, W. Jia, W. Ji, C. Ruan, Y. Sun. Cucumber fruits detection in greenhouses based on instance segmentation. IEEE Access, 2019, 7: 139635–139642. DOI: 10.1109/ACCESS.2019.2942144

R. Yang, Y. Hu, Y. Yao, M. Gao, R. Liu. Fruit target detection based on bco-yolov5 model. Mobile Information Systems, 2022. DOI: 10.1155/2022/8457173

K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, Q. Tian. Centernet: Keypointtriplets for object detection, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6569–6578


Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.