The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research

Jurgen Leitner; Adam W. Tow; Niko Sunderhauf; Jake E. Dean; Joseph W. Durham; Matthew Cooper; Markus Eich; Christopher Lehnert; Ruben Mangels; Christopher McCool; Peter T. Kujala; Lachlan Nicholson; Trung Pham; James Sergeant; Liao Wu; Fangyi Zhang; Ben Upcroft; Peter Corke

doi:10.1109/ICRA.2017.7989545

The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research

Jurgen Leitner, Adam W. Tow, Niko Sunderhauf, Jake E. Dean, Joseph W. Durham, Matthew Cooper, Markus Eich, Christopher Lehnert, Ruben Mangels, Christopher McCool, Peter T. Kujala, Lachlan Nicholson, Trung Pham, James Sergeant, Liao Wu, Fangyi Zhang, Ben Upcroft, Peter Corke

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Other › peer-review

68 Citations (Scopus)

Abstract

Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark. Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems - including perception and manipulation - instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.

Original language	English
Title of host publication	2017 IEEE International Conference on Robotics and Automation (ICRA 2017)
Editors	Haoyong Yu, Zhidong Wang
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	4705-4712
Number of pages	8
ISBN (Electronic)	9781509046331
ISBN (Print)	9781509046338
DOIs	https://doi.org/10.1109/ICRA.2017.7989545
Publication status	Published - 2017
Externally published	Yes
Event	IEEE International Conference on Robotics and Automation 2017 - Sands Expo and Convention Centre, Singapore, Singapore Duration: 29 May 2017 → 3 Jun 2017 http://www.icra2017.org/ https://ieeexplore.ieee.org/xpl/conhome/7960754/proceeding (Proceedings)

Publication series

Name	Proceedings - IEEE International Conference on Robotics and Automation
Publisher	IEEE, Institute of Electrical and Electronics Engineers
ISSN (Print)	1050-4729

Conference

Conference	IEEE International Conference on Robotics and Automation 2017
Abbreviated title	ICRA 2017
Country/Territory	Singapore
City	Singapore
Period	29/05/17 → 3/06/17
Internet address	http://www.icra2017.org/ https://ieeexplore.ieee.org/xpl/conhome/7960754/proceeding (Proceedings)

Access to Document

10.1109/ICRA.2017.7989545

Cite this

Leitner, J., Tow, A. W., Sunderhauf, N., Dean, J. E., Durham, J. W., Cooper, M., Eich, M., Lehnert, C., Mangels, R., McCool, C., Kujala, P. T., Nicholson, L., Pham, T., Sergeant, J., Wu, L., Zhang, F., Upcroft, B., & Corke, P. (2017). The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research. In H. Yu, & Z. Wang (Eds.), 2017 IEEE International Conference on Robotics and Automation (ICRA 2017) (pp. 4705-4712). (Proceedings - IEEE International Conference on Robotics and Automation). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICRA.2017.7989545

Leitner, Jurgen ; Tow, Adam W. ; Sunderhauf, Niko et al. / The ACRV picking benchmark : a robotic shelf picking benchmark to foster reproducible research. 2017 IEEE International Conference on Robotics and Automation (ICRA 2017). editor / Haoyong Yu ; Zhidong Wang. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2017. pp. 4705-4712 (Proceedings - IEEE International Conference on Robotics and Automation).

@inproceedings{9cd151c0b16d480faf6d00134bba57e8,

title = "The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research",

abstract = "Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark. Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems - including perception and manipulation - instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.",

author = "Jurgen Leitner and Tow, {Adam W.} and Niko Sunderhauf and Dean, {Jake E.} and Durham, {Joseph W.} and Matthew Cooper and Markus Eich and Christopher Lehnert and Ruben Mangels and Christopher McCool and Kujala, {Peter T.} and Lachlan Nicholson and Trung Pham and James Sergeant and Liao Wu and Fangyi Zhang and Ben Upcroft and Peter Corke",

note = "Funding Information: This research was supported by the Australian Research Council Centre of Excellence for Robotic Vision (ACRV) (project number CE140100016). The participation at the APC was supported by Amazon Robotics LLC. 1Authors are with the Australian Centre for Robotic Vision (ACRV) 2Authors are with the Queensland University of Technology (QUT), Brisbane, QLD 4001 Australia. 3JWD is with Amazon Robotics LLC, North Reading, MA 01864 USA. 4TP is with the University of Adelaide, Adelaide, SA 5005 Australia. Publisher Copyright: {\textcopyright} 2017 IEEE.; IEEE International Conference on Robotics and Automation 2017, ICRA 2017 ; Conference date: 29-05-2017 Through 03-06-2017",

year = "2017",

doi = "10.1109/ICRA.2017.7989545",

language = "English",

isbn = "9781509046338",

series = "Proceedings - IEEE International Conference on Robotics and Automation",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "4705--4712",

editor = "Haoyong Yu and Zhidong Wang",

booktitle = "2017 IEEE International Conference on Robotics and Automation (ICRA 2017)",

address = "United States of America",

url = "http://www.icra2017.org/, https://ieeexplore.ieee.org/xpl/conhome/7960754/proceeding",

}

Leitner, J, Tow, AW, Sunderhauf, N, Dean, JE, Durham, JW, Cooper, M, Eich, M, Lehnert, C, Mangels, R, McCool, C, Kujala, PT, Nicholson, L, Pham, T, Sergeant, J, Wu, L, Zhang, F, Upcroft, B & Corke, P 2017, The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research. in H Yu & Z Wang (eds), 2017 IEEE International Conference on Robotics and Automation (ICRA 2017). Proceedings - IEEE International Conference on Robotics and Automation, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 4705-4712, IEEE International Conference on Robotics and Automation 2017, Singapore, Singapore, 29/05/17. https://doi.org/10.1109/ICRA.2017.7989545

The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research. / Leitner, Jurgen; Tow, Adam W.; Sunderhauf, Niko et al.
2017 IEEE International Conference on Robotics and Automation (ICRA 2017). ed. / Haoyong Yu; Zhidong Wang. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2017. p. 4705-4712 (Proceedings - IEEE International Conference on Robotics and Automation).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Other › peer-review

TY - GEN

T1 - The ACRV picking benchmark

T2 - IEEE International Conference on Robotics and Automation 2017

AU - Leitner, Jurgen

AU - Tow, Adam W.

AU - Sunderhauf, Niko

AU - Dean, Jake E.

AU - Durham, Joseph W.

AU - Cooper, Matthew

AU - Eich, Markus

AU - Lehnert, Christopher

AU - Mangels, Ruben

AU - McCool, Christopher

AU - Kujala, Peter T.

AU - Nicholson, Lachlan

AU - Pham, Trung

AU - Sergeant, James

AU - Wu, Liao

AU - Zhang, Fangyi

AU - Upcroft, Ben

AU - Corke, Peter

N1 - Funding Information: This research was supported by the Australian Research Council Centre of Excellence for Robotic Vision (ACRV) (project number CE140100016). The participation at the APC was supported by Amazon Robotics LLC. 1Authors are with the Australian Centre for Robotic Vision (ACRV) 2Authors are with the Queensland University of Technology (QUT), Brisbane, QLD 4001 Australia. 3JWD is with Amazon Robotics LLC, North Reading, MA 01864 USA. 4TP is with the University of Adelaide, Adelaide, SA 5005 Australia. Publisher Copyright: © 2017 IEEE.

PY - 2017

Y1 - 2017

N2 - Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark. Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems - including perception and manipulation - instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.

AB - Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark. Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems - including perception and manipulation - instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.

UR - http://www.scopus.com/inward/record.url?scp=85028023594&partnerID=8YFLogxK

U2 - 10.1109/ICRA.2017.7989545

DO - 10.1109/ICRA.2017.7989545

M3 - Conference Paper

AN - SCOPUS:85028023594

SN - 9781509046338

T3 - Proceedings - IEEE International Conference on Robotics and Automation

SP - 4705

EP - 4712

BT - 2017 IEEE International Conference on Robotics and Automation (ICRA 2017)

A2 - Yu, Haoyong

A2 - Wang, Zhidong

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

Y2 - 29 May 2017 through 3 June 2017

ER -

Leitner J, Tow AW, Sunderhauf N, Dean JE, Durham JW, Cooper M et al. The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research. In Yu H, Wang Z, editors, 2017 IEEE International Conference on Robotics and Automation (ICRA 2017). Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2017. p. 4705-4712. (Proceedings - IEEE International Conference on Robotics and Automation). doi: 10.1109/ICRA.2017.7989545

The ACRV picking benchmark: a robotic shelf picking benchmark to foster reproducible research

Abstract

Publication series

Conference

Access to Document

Other files and links

Cite this