3D contact point cloud reconstruction from vision-based tactile flow

Yipai Du, Guanlan Zhang, Michael Yu Wang

Research output: Contribution to journalArticleResearchpeer-review


With the growing interest in vision-based tactile sensors, various types of sensors that utilize digital imaging are being developed. Among them, a group of sensors captures the tactile flow resulted from the contact deformation using the optical flow algorithm from computer vision and achieves full resolution deformation tracking on the tactile surface. In this work, a novel 3D contact reconstruction algorithm is proposed and evaluated. It exploits the contact geometry and projection relationship in the tactile flow, which are versatile for vision-based tactile sensors, unique for tactile perception but not inherited from computer vision. The resulted 3D contact point cloud representation is consistent with the tactile flow constraint, scale estimation, and contact edge estimation. It can be directly manipulated in downstream applications such as contact force estimation and contact pose estimation. Experiments and examples are provided that indicate the potential for the proposed tactile processing algorithm to connect tactile perception to tactile enabled robotic manipulation tasks.

Original languageEnglish
Pages (from-to)12177-12184
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number4
Publication statusPublished - Oct 2022
Externally publishedYes


  • Contact modeling
  • force and tactile sensing
  • perception for grasping and manipulation

Cite this