ODAM: Object Detection, Association, and Mapping using posed RGB video

Kejie Li; Daniel DeTone; Steven Chen; Minh Vo; Ian Reid; Hamid Rezatofighi; Chris Sweeney; Julian Straub; Richard Newcombe

doi:10.1109/ICCV48922.2021.00594

ODAM: Object Detection, Association, and Mapping using posed RGB video

Kejie Li
, Daniel DeTone
, Steven Chen
, Minh Vo
, Ian Reid
, Hamid Rezatofighi
, Chris Sweeney
, Julian Straub
, Richard Newcombe

Department of Data Science & AI

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

26 Citations (Scopus)

Abstract

Localizing objects and estimating their extent in 3D is an important step towards high-level 3D scene understanding, which has many applications in Augmented Reality and Robotics. We present ODAM, a system for 3D Object Detection, Association, and Mapping using posed RGB videos. The proposed system relies on a deep learning front-end to detect 3D objects from a given RGB frame and associate them to a global object-based map using a graph neural network (GNN). Based on these frame-to-model associations, our back-end optimizes object bounding volumes, represented as super-quadrics, under multi-view geometry constraints and the object scale prior. We validate the proposed system on ScanNet where we show a significant improvement over existing RGB-only methods.

Original language	English
Title of host publication	Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021
Editors	Eric Mortensen
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	5978-5988
Number of pages	11
ISBN (Electronic)	9781665428125
DOIs	https://doi.org/10.1109/ICCV48922.2021.00594
Publication status	Published - 2021
Event	IEEE International Conference on Computer Vision 2021 - Online, United States of America Duration: 11 Oct 2021 → 17 Oct 2021 https://iccv2021.thecvf.com/home (Website) https://ieeexplore.ieee.org/xpl/conhome/9709627/proceeding (Proceedings)

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
ISSN (Print)	1550-5499

Conference

Conference	IEEE International Conference on Computer Vision 2021
Abbreviated title	ICCV 2021
Country/Territory	United States of America
City	Online
Period	11/10/21 → 17/10/21
Internet address	https://iccv2021.thecvf.com/home (Website) https://ieeexplore.ieee.org/xpl/conhome/9709627/proceeding (Proceedings)

Access to Document

10.1109/ICCV48922.2021.00594

354731270-oaAccepted author manuscript, 1.39 MB

https://openaccess.thecvf.com/content/ICCV2021/html/Li_ODAM_Object_Detection_Association_and_Mapping_Using_Posed_RGB_Video_ICCV_2021_paper.html

Cite this

Li, K., DeTone, D., Chen, S., Vo, M., Reid, I., Rezatofighi, H., Sweeney, C., Straub, J., & Newcombe, R. (2021). ODAM: Object Detection, Association, and Mapping using posed RGB video. In E. Mortensen (Ed.), Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021 (pp. 5978-5988). (Proceedings of the IEEE International Conference on Computer Vision). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICCV48922.2021.00594

Li, Kejie ; DeTone, Daniel ; Chen, Steven et al. / ODAM : Object Detection, Association, and Mapping using posed RGB video. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. editor / Eric Mortensen. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2021. pp. 5978-5988 (Proceedings of the IEEE International Conference on Computer Vision).

@inproceedings{dbcbd8e2c3aa4bfdba0631dec5e1ac76,

title = "ODAM: Object Detection, Association, and Mapping using posed RGB video",

abstract = "Localizing objects and estimating their extent in 3D is an important step towards high-level 3D scene understanding, which has many applications in Augmented Reality and Robotics. We present ODAM, a system for 3D Object Detection, Association, and Mapping using posed RGB videos. The proposed system relies on a deep learning front-end to detect 3D objects from a given RGB frame and associate them to a global object-based map using a graph neural network (GNN). Based on these frame-to-model associations, our back-end optimizes object bounding volumes, represented as super-quadrics, under multi-view geometry constraints and the object scale prior. We validate the proposed system on ScanNet where we show a significant improvement over existing RGB-only methods.",

author = "Kejie Li and Daniel DeTone and Steven Chen and Minh Vo and Ian Reid and Hamid Rezatofighi and Chris Sweeney and Julian Straub and Richard Newcombe",

note = "Funding Information: KL and IR gratefully acknowledge the support of the ARC through the Centre of Excellence for Robotic Vision CE140100016. Funding Information: We presented ODAM, a system to localize and infer 3D oriented bounding volumes of objects given posed RGB-only videos. Key to ODAM is (1) an attention-based GNN for robust detection-to-map data association, and (2) a super-quadric-based multi-view optimization for accurate object bounding volume estimation from the associated 2D BB and class observations. ODAM is the best performing RGB-only method for object-based mapping. The fact that the proposed RGB-only methods can close the accuracy gap to RGB-D methods in a subset of object categories is encouraging and points to a future where depth cameras are unnecessary for 3D scene understanding. Acknowledgment KL and IR gratefully acknowledge the support of the ARC through the Centre of Excellence for Publisher Copyright: {\textcopyright} 2021 IEEE; IEEE International Conference on Computer Vision 2021, ICCV 2021 ; Conference date: 11-10-2021 Through 17-10-2021",

year = "2021",

doi = "10.1109/ICCV48922.2021.00594",

language = "English",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "5978--5988",

editor = "Eric Mortensen",

booktitle = "Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021",

address = "United States of America",

url = "https://iccv2021.thecvf.com/home, https://ieeexplore.ieee.org/xpl/conhome/9709627/proceeding",

}

Li, K, DeTone, D, Chen, S, Vo, M, Reid, I, Rezatofighi, H, Sweeney, C, Straub, J & Newcombe, R 2021, ODAM: Object Detection, Association, and Mapping using posed RGB video. in E Mortensen (ed.), Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. Proceedings of the IEEE International Conference on Computer Vision, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 5978-5988, IEEE International Conference on Computer Vision 2021, Online, United States of America, 11/10/21. https://doi.org/10.1109/ICCV48922.2021.00594

ODAM: Object Detection, Association, and Mapping using posed RGB video. / Li, Kejie; DeTone, Daniel; Chen, Steven et al.
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. ed. / Eric Mortensen. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2021. p. 5978-5988 (Proceedings of the IEEE International Conference on Computer Vision).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - ODAM

T2 - IEEE International Conference on Computer Vision 2021

AU - Li, Kejie

AU - DeTone, Daniel

AU - Chen, Steven

AU - Vo, Minh

AU - Reid, Ian

AU - Rezatofighi, Hamid

AU - Sweeney, Chris

AU - Straub, Julian

AU - Newcombe, Richard

N1 - Funding Information: KL and IR gratefully acknowledge the support of the ARC through the Centre of Excellence for Robotic Vision CE140100016. Funding Information: We presented ODAM, a system to localize and infer 3D oriented bounding volumes of objects given posed RGB-only videos. Key to ODAM is (1) an attention-based GNN for robust detection-to-map data association, and (2) a super-quadric-based multi-view optimization for accurate object bounding volume estimation from the associated 2D BB and class observations. ODAM is the best performing RGB-only method for object-based mapping. The fact that the proposed RGB-only methods can close the accuracy gap to RGB-D methods in a subset of object categories is encouraging and points to a future where depth cameras are unnecessary for 3D scene understanding. Acknowledgment KL and IR gratefully acknowledge the support of the ARC through the Centre of Excellence for Publisher Copyright: © 2021 IEEE

PY - 2021

Y1 - 2021

N2 - Localizing objects and estimating their extent in 3D is an important step towards high-level 3D scene understanding, which has many applications in Augmented Reality and Robotics. We present ODAM, a system for 3D Object Detection, Association, and Mapping using posed RGB videos. The proposed system relies on a deep learning front-end to detect 3D objects from a given RGB frame and associate them to a global object-based map using a graph neural network (GNN). Based on these frame-to-model associations, our back-end optimizes object bounding volumes, represented as super-quadrics, under multi-view geometry constraints and the object scale prior. We validate the proposed system on ScanNet where we show a significant improvement over existing RGB-only methods.

AB - Localizing objects and estimating their extent in 3D is an important step towards high-level 3D scene understanding, which has many applications in Augmented Reality and Robotics. We present ODAM, a system for 3D Object Detection, Association, and Mapping using posed RGB videos. The proposed system relies on a deep learning front-end to detect 3D objects from a given RGB frame and associate them to a global object-based map using a graph neural network (GNN). Based on these frame-to-model associations, our back-end optimizes object bounding volumes, represented as super-quadrics, under multi-view geometry constraints and the object scale prior. We validate the proposed system on ScanNet where we show a significant improvement over existing RGB-only methods.

UR - https://www.scopus.com/pages/publications/85124219564

U2 - 10.1109/ICCV48922.2021.00594

DO - 10.1109/ICCV48922.2021.00594

M3 - Conference Paper

AN - SCOPUS:85124219564

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 5978

EP - 5988

BT - Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

A2 - Mortensen, Eric

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

Y2 - 11 October 2021 through 17 October 2021

ER -

Li K, DeTone D, Chen S, Vo M, Reid I, Rezatofighi H et al. ODAM: Object Detection, Association, and Mapping using posed RGB video. In Mortensen E, editor, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2021. p. 5978-5988. (Proceedings of the IEEE International Conference on Computer Vision). doi: 10.1109/ICCV48922.2021.00594

ODAM: Object Detection, Association, and Mapping using posed RGB video

Abstract

Publication series

Conference

Access to Document

Other files and links

Cite this