Fruit detection and segmentation for apple harvesting using visual sensor in orchards

Hanwen Kang, Chao Chen

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Autonomous harvesting shows a promising prospect in the future development of the agriculture industry, while the vision system is one of the most challenging components in the autonomous harvesting technologies. This work proposes a multi-function network to perform the real-time detection and semantic segmentation of apples and branches in orchard environments by using the visual sensor. The developed detection and segmentation network utilises the atrous spatial pyramid pooling and the gate feature pyramid network to enhance feature extraction ability of the network. To improve the real-time computation performance of the network model, a lightweight backbone network based on the residual network architecture is developed. From the experimental results, the detection and segmentation network with ResNet-101 backbone outperformed on the detection and segmentation tasks, achieving an F1 score of 0.832 on the detection of apples and 87.6% and 77.2% on the semantic segmentation of apples and branches, respectively. The network model with lightweight backbone showed the best computation efficiency in the results. It achieved an F1 score of 0.827 on the detection of apples and 86.5% and 75.7% on the segmentation of apples and branches, respectively. The weights size and computation time of the network model with lightweight backbone were 12.8 M and 32 ms, respectively. The experimental results show that the detection and segmentation network can effectively perform the real-time detection and segmentation of apples and branches in orchards.

Original languageEnglish
Article number4599
Number of pages16
JournalSensors
Volume19
Issue number20
DOIs
Publication statusPublished - 2 Oct 2019

Keywords

  • Automated harvesting robot
  • Deep learning
  • Machine vision
  • Real-time fruit detection
  • Semantic segmentation
  • Visual sensor

Cite this

@article{31d740b37f7a405d8ddadf2a63fde13e,
title = "Fruit detection and segmentation for apple harvesting using visual sensor in orchards",
abstract = "Autonomous harvesting shows a promising prospect in the future development of the agriculture industry, while the vision system is one of the most challenging components in the autonomous harvesting technologies. This work proposes a multi-function network to perform the real-time detection and semantic segmentation of apples and branches in orchard environments by using the visual sensor. The developed detection and segmentation network utilises the atrous spatial pyramid pooling and the gate feature pyramid network to enhance feature extraction ability of the network. To improve the real-time computation performance of the network model, a lightweight backbone network based on the residual network architecture is developed. From the experimental results, the detection and segmentation network with ResNet-101 backbone outperformed on the detection and segmentation tasks, achieving an F1 score of 0.832 on the detection of apples and 87.6{\%} and 77.2{\%} on the semantic segmentation of apples and branches, respectively. The network model with lightweight backbone showed the best computation efficiency in the results. It achieved an F1 score of 0.827 on the detection of apples and 86.5{\%} and 75.7{\%} on the segmentation of apples and branches, respectively. The weights size and computation time of the network model with lightweight backbone were 12.8 M and 32 ms, respectively. The experimental results show that the detection and segmentation network can effectively perform the real-time detection and segmentation of apples and branches in orchards.",
keywords = "Automated harvesting robot, Deep learning, Machine vision, Real-time fruit detection, Semantic segmentation, Visual sensor",
author = "Hanwen Kang and Chao Chen",
year = "2019",
month = "10",
day = "2",
doi = "10.3390/s19204599",
language = "English",
volume = "19",
journal = "Sensors",
issn = "1424-8220",
publisher = "MDPI AG",
number = "20",

}

Fruit detection and segmentation for apple harvesting using visual sensor in orchards. / Kang, Hanwen; Chen, Chao.

In: Sensors, Vol. 19, No. 20, 4599, 02.10.2019.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Fruit detection and segmentation for apple harvesting using visual sensor in orchards

AU - Kang, Hanwen

AU - Chen, Chao

PY - 2019/10/2

Y1 - 2019/10/2

N2 - Autonomous harvesting shows a promising prospect in the future development of the agriculture industry, while the vision system is one of the most challenging components in the autonomous harvesting technologies. This work proposes a multi-function network to perform the real-time detection and semantic segmentation of apples and branches in orchard environments by using the visual sensor. The developed detection and segmentation network utilises the atrous spatial pyramid pooling and the gate feature pyramid network to enhance feature extraction ability of the network. To improve the real-time computation performance of the network model, a lightweight backbone network based on the residual network architecture is developed. From the experimental results, the detection and segmentation network with ResNet-101 backbone outperformed on the detection and segmentation tasks, achieving an F1 score of 0.832 on the detection of apples and 87.6% and 77.2% on the semantic segmentation of apples and branches, respectively. The network model with lightweight backbone showed the best computation efficiency in the results. It achieved an F1 score of 0.827 on the detection of apples and 86.5% and 75.7% on the segmentation of apples and branches, respectively. The weights size and computation time of the network model with lightweight backbone were 12.8 M and 32 ms, respectively. The experimental results show that the detection and segmentation network can effectively perform the real-time detection and segmentation of apples and branches in orchards.

AB - Autonomous harvesting shows a promising prospect in the future development of the agriculture industry, while the vision system is one of the most challenging components in the autonomous harvesting technologies. This work proposes a multi-function network to perform the real-time detection and semantic segmentation of apples and branches in orchard environments by using the visual sensor. The developed detection and segmentation network utilises the atrous spatial pyramid pooling and the gate feature pyramid network to enhance feature extraction ability of the network. To improve the real-time computation performance of the network model, a lightweight backbone network based on the residual network architecture is developed. From the experimental results, the detection and segmentation network with ResNet-101 backbone outperformed on the detection and segmentation tasks, achieving an F1 score of 0.832 on the detection of apples and 87.6% and 77.2% on the semantic segmentation of apples and branches, respectively. The network model with lightweight backbone showed the best computation efficiency in the results. It achieved an F1 score of 0.827 on the detection of apples and 86.5% and 75.7% on the segmentation of apples and branches, respectively. The weights size and computation time of the network model with lightweight backbone were 12.8 M and 32 ms, respectively. The experimental results show that the detection and segmentation network can effectively perform the real-time detection and segmentation of apples and branches in orchards.

KW - Automated harvesting robot

KW - Deep learning

KW - Machine vision

KW - Real-time fruit detection

KW - Semantic segmentation

KW - Visual sensor

UR - http://www.scopus.com/inward/record.url?scp=85074146740&partnerID=8YFLogxK

U2 - 10.3390/s19204599

DO - 10.3390/s19204599

M3 - Article

VL - 19

JO - Sensors

JF - Sensors

SN - 1424-8220

IS - 20

M1 - 4599

ER -