Joint prediction of depths, normals and surface curvature from RGB images using CNNs

Thanuja Dharmasiri, Andrew Spek, Tom Drummond

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

8 Citations (Scopus)

Abstract

Understanding the 3D structure of a scene is of vital importance, when it comes to developing fully autonomous robots. To this end, we present a novel deep learning based framework that estimates depth, surface normals and surface curvature by only using a single RGB image. To the best of our knowledge this is the first work to estimate surface curvature from colour using a machine learning approach. Additionally, we demonstrate that by tuning the network to infer well designed features, such as surface curvature, we can achieve improved performance at estimating depth and normals. This indicates that network guidance is still a useful aspect of designing and training a neural network. We run extensive experiments where the network is trained to infer different tasks while the model capacity is kept constant resulting in different feature maps based on the tasks at hand. We outperform the previous state-of-the-art benchmarks which jointly estimate depths and surface normals while predicting surface curvature in parallel.

Original languageEnglish
Title of host publication2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017)
EditorsRichard Vaughan
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages1505-1512
Number of pages8
ISBN (Electronic)9781538626825, 9781538626818
ISBN (Print)9781538626832
DOIs
Publication statusPublished - 13 Dec 2017
EventIEEE/RSJ International Conference on Intelligent Robots and Systems 2017: Friendly People, Friendly Robots - Vancouver, Canada
Duration: 24 Sep 201728 Sep 2017
Conference number: 30th
http://iros2017.org/

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

ConferenceIEEE/RSJ International Conference on Intelligent Robots and Systems 2017
Abbreviated titleIROS2017
CountryCanada
CityVancouver
Period24/09/1728/09/17
Internet address

Cite this

Dharmasiri, T., Spek, A., & Drummond, T. (2017). Joint prediction of depths, normals and surface curvature from RGB images using CNNs. In R. Vaughan (Ed.), 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017) (pp. 1505-1512). (IEEE International Conference on Intelligent Robots and Systems). Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/IROS.2017.8205954
Dharmasiri, Thanuja ; Spek, Andrew ; Drummond, Tom. / Joint prediction of depths, normals and surface curvature from RGB images using CNNs. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017). editor / Richard Vaughan. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2017. pp. 1505-1512 (IEEE International Conference on Intelligent Robots and Systems).
@inproceedings{e4ce47c49045462eb045a75a1b1fa5e5,
title = "Joint prediction of depths, normals and surface curvature from RGB images using CNNs",
abstract = "Understanding the 3D structure of a scene is of vital importance, when it comes to developing fully autonomous robots. To this end, we present a novel deep learning based framework that estimates depth, surface normals and surface curvature by only using a single RGB image. To the best of our knowledge this is the first work to estimate surface curvature from colour using a machine learning approach. Additionally, we demonstrate that by tuning the network to infer well designed features, such as surface curvature, we can achieve improved performance at estimating depth and normals. This indicates that network guidance is still a useful aspect of designing and training a neural network. We run extensive experiments where the network is trained to infer different tasks while the model capacity is kept constant resulting in different feature maps based on the tasks at hand. We outperform the previous state-of-the-art benchmarks which jointly estimate depths and surface normals while predicting surface curvature in parallel.",
author = "Thanuja Dharmasiri and Andrew Spek and Tom Drummond",
year = "2017",
month = "12",
day = "13",
doi = "10.1109/IROS.2017.8205954",
language = "English",
isbn = "9781538626832",
series = "IEEE International Conference on Intelligent Robots and Systems",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
pages = "1505--1512",
editor = "Richard Vaughan",
booktitle = "2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017)",
address = "United States of America",

}

Dharmasiri, T, Spek, A & Drummond, T 2017, Joint prediction of depths, normals and surface curvature from RGB images using CNNs. in R Vaughan (ed.), 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017). IEEE International Conference on Intelligent Robots and Systems, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 1505-1512, IEEE/RSJ International Conference on Intelligent Robots and Systems 2017, Vancouver, Canada, 24/09/17. https://doi.org/10.1109/IROS.2017.8205954

Joint prediction of depths, normals and surface curvature from RGB images using CNNs. / Dharmasiri, Thanuja; Spek, Andrew; Drummond, Tom.

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017). ed. / Richard Vaughan. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2017. p. 1505-1512 (IEEE International Conference on Intelligent Robots and Systems).

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - Joint prediction of depths, normals and surface curvature from RGB images using CNNs

AU - Dharmasiri, Thanuja

AU - Spek, Andrew

AU - Drummond, Tom

PY - 2017/12/13

Y1 - 2017/12/13

N2 - Understanding the 3D structure of a scene is of vital importance, when it comes to developing fully autonomous robots. To this end, we present a novel deep learning based framework that estimates depth, surface normals and surface curvature by only using a single RGB image. To the best of our knowledge this is the first work to estimate surface curvature from colour using a machine learning approach. Additionally, we demonstrate that by tuning the network to infer well designed features, such as surface curvature, we can achieve improved performance at estimating depth and normals. This indicates that network guidance is still a useful aspect of designing and training a neural network. We run extensive experiments where the network is trained to infer different tasks while the model capacity is kept constant resulting in different feature maps based on the tasks at hand. We outperform the previous state-of-the-art benchmarks which jointly estimate depths and surface normals while predicting surface curvature in parallel.

AB - Understanding the 3D structure of a scene is of vital importance, when it comes to developing fully autonomous robots. To this end, we present a novel deep learning based framework that estimates depth, surface normals and surface curvature by only using a single RGB image. To the best of our knowledge this is the first work to estimate surface curvature from colour using a machine learning approach. Additionally, we demonstrate that by tuning the network to infer well designed features, such as surface curvature, we can achieve improved performance at estimating depth and normals. This indicates that network guidance is still a useful aspect of designing and training a neural network. We run extensive experiments where the network is trained to infer different tasks while the model capacity is kept constant resulting in different feature maps based on the tasks at hand. We outperform the previous state-of-the-art benchmarks which jointly estimate depths and surface normals while predicting surface curvature in parallel.

UR - http://www.scopus.com/inward/record.url?scp=85041946693&partnerID=8YFLogxK

U2 - 10.1109/IROS.2017.8205954

DO - 10.1109/IROS.2017.8205954

M3 - Conference Paper

AN - SCOPUS:85041946693

SN - 9781538626832

T3 - IEEE International Conference on Intelligent Robots and Systems

SP - 1505

EP - 1512

BT - 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017)

A2 - Vaughan, Richard

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

ER -

Dharmasiri T, Spek A, Drummond T. Joint prediction of depths, normals and surface curvature from RGB images using CNNs. In Vaughan R, editor, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017). Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2017. p. 1505-1512. (IEEE International Conference on Intelligent Robots and Systems). https://doi.org/10.1109/IROS.2017.8205954