Benchmarking simulated robotic manipulation through a real world dataset

Jack Collins, Jessie McVicar, David Wedlock, Ross Brown, David Howard, Jurgen Leitner

Research output: Contribution to journalArticleResearchpeer-review

16 Citations (Scopus)


We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies. We apply our benchmark to two simulation environments, i.e., PyBullet and V-Rep, and publish the results. All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks' website11[Online]. Available:

Original languageEnglish
Pages (from-to)250-257
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number1
Publication statusPublished - Jan 2020
Externally publishedYes


  • force and tactile sensing
  • grasping
  • Performance evaluation and benchmarking
  • simulation and animation

Cite this