Deepfruits: A fruit detection system using deep neural networks

Inkyu Sa, Zongyuan Ge, Feras Dayoub, Ben Upcroft, Tristan Perez, Chris McCool

Research output: Contribution to journalArticleResearchpeer-review

Abstract

This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0.807 to 0.838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit.

Original languageEnglish
Article number1222
Number of pages23
JournalSensors
Volume16
Issue number8
DOIs
Publication statusPublished - 3 Aug 2016
Externally publishedYes

Keywords

  • Agricultural robotics
  • Deep convolutional neural network
  • Harvesting robots
  • Horticulture
  • Multi-modal
  • Rapid training
  • Real-time performance
  • Visual fruit detection

Cite this

Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., & McCool, C. (2016). Deepfruits: A fruit detection system using deep neural networks. Sensors, 16(8), [1222]. https://doi.org/10.3390/s16081222
Sa, Inkyu ; Ge, Zongyuan ; Dayoub, Feras ; Upcroft, Ben ; Perez, Tristan ; McCool, Chris. / Deepfruits : A fruit detection system using deep neural networks. In: Sensors. 2016 ; Vol. 16, No. 8.
@article{ea89f0bf8b1c4b799b98b6d26e1ae6f2,
title = "Deepfruits: A fruit detection system using deep neural networks",
abstract = "This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0.807 to 0.838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit.",
keywords = "Agricultural robotics, Deep convolutional neural network, Harvesting robots, Horticulture, Multi-modal, Rapid training, Real-time performance, Visual fruit detection",
author = "Inkyu Sa and Zongyuan Ge and Feras Dayoub and Ben Upcroft and Tristan Perez and Chris McCool",
year = "2016",
month = "8",
day = "3",
doi = "10.3390/s16081222",
language = "English",
volume = "16",
journal = "Sensors",
issn = "1424-8220",
publisher = "MDPI AG",
number = "8",

}

Sa, I, Ge, Z, Dayoub, F, Upcroft, B, Perez, T & McCool, C 2016, 'Deepfruits: A fruit detection system using deep neural networks' Sensors, vol. 16, no. 8, 1222. https://doi.org/10.3390/s16081222

Deepfruits : A fruit detection system using deep neural networks. / Sa, Inkyu; Ge, Zongyuan; Dayoub, Feras; Upcroft, Ben; Perez, Tristan; McCool, Chris.

In: Sensors, Vol. 16, No. 8, 1222, 03.08.2016.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Deepfruits

T2 - A fruit detection system using deep neural networks

AU - Sa, Inkyu

AU - Ge, Zongyuan

AU - Dayoub, Feras

AU - Upcroft, Ben

AU - Perez, Tristan

AU - McCool, Chris

PY - 2016/8/3

Y1 - 2016/8/3

N2 - This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0.807 to 0.838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit.

AB - This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0.807 to 0.838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit.

KW - Agricultural robotics

KW - Deep convolutional neural network

KW - Harvesting robots

KW - Horticulture

KW - Multi-modal

KW - Rapid training

KW - Real-time performance

KW - Visual fruit detection

UR - http://www.scopus.com/inward/record.url?scp=84982682350&partnerID=8YFLogxK

U2 - 10.3390/s16081222

DO - 10.3390/s16081222

M3 - Article

VL - 16

JO - Sensors

JF - Sensors

SN - 1424-8220

IS - 8

M1 - 1222

ER -

Sa I, Ge Z, Dayoub F, Upcroft B, Perez T, McCool C. Deepfruits: A fruit detection system using deep neural networks. Sensors. 2016 Aug 3;16(8). 1222. https://doi.org/10.3390/s16081222