Distilling knowledge from a deep pose regressor network

Muhamad Risqi U. Saputra; Pedro Gusmao; Yasin Almalioglu; Andrew Markham; Niki Trigoni

doi:10.1109/ICCV.2019.00035

Distilling knowledge from a deep pose regressor network

Muhamad Risqi U. Saputra, Pedro Gusmao, Yasin Almalioglu, Andrew Markham, Niki Trigoni

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

81 Citations (Scopus)

Abstract

This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.

Original language	English
Title of host publication	Proceedings - 2019 International Conference on Computer Vision, ICCV 2019
Editors	In So Kweon, Nikos Paragios, Ming-Hsuan Yang, Svetlana Lazebnik
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	263-272
Number of pages	10
ISBN (Electronic)	9781728148038
ISBN (Print)	9781728148045
DOIs	https://doi.org/10.1109/ICCV.2019.00035
Publication status	Published - 2019
Externally published	Yes
Event	IEEE International Conference on Computer Vision 2019 - Seoul, Korea, South Duration: 27 Oct 2019 → 2 Nov 2019 Conference number: 17th http://iccv2019.thecvf.com/ https://ieeexplore.ieee.org/xpl/conhome/8972782/proceeding (Proceedings)

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
Publisher	The Institute of Electrical and Electronics Engineers, Inc. All rights
Volume	2019-October
ISSN (Print)	1550-5499
ISSN (Electronic)	2380-7504

Conference

Conference	IEEE International Conference on Computer Vision 2019
Abbreviated title	ICCV 2019
Country/Territory	Korea, South
City	Seoul
Period	27/10/19 → 2/11/19
Internet address	http://iccv2019.thecvf.com/ https://ieeexplore.ieee.org/xpl/conhome/8972782/proceeding (Proceedings)

Access to Document

10.1109/ICCV.2019.00035

Cite this

Saputra, M. R. U., Gusmao, P., Almalioglu, Y., Markham, A., & Trigoni, N. (2019). Distilling knowledge from a deep pose regressor network. In I. So Kweon, N. Paragios, M.-H. Yang, & S. Lazebnik (Eds.), Proceedings - 2019 International Conference on Computer Vision, ICCV 2019 (pp. 263-272). (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2019-October). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICCV.2019.00035

Saputra, Muhamad Risqi U. ; Gusmao, Pedro ; Almalioglu, Yasin et al. / Distilling knowledge from a deep pose regressor network. Proceedings - 2019 International Conference on Computer Vision, ICCV 2019. editor / In So Kweon ; Nikos Paragios ; Ming-Hsuan Yang ; Svetlana Lazebnik. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. pp. 263-272 (Proceedings of the IEEE International Conference on Computer Vision).

@inproceedings{48d3b599abb348db838d9d76aab7ba3e,

title = "Distilling knowledge from a deep pose regressor network",

abstract = "This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.",

author = "Saputra, {Muhamad Risqi U.} and Pedro Gusmao and Yasin Almalioglu and Andrew Markham and Niki Trigoni",

note = "Funding Information: We have presented an approach to distill the knowledge from a deep pose regressor network to a much smaller network with a small loss of accuracy. We have shown that the teacher loss can be used as an effective attentive mechanism to transfer the knowledge between teacher and student. For future work, we will investigate whether another compression technique can be combined with our distillation approach to further reduce the computation time. Acknowledgement. This research is funded by the US National Institute of Standards and Technology (NIST) Grant No. 70NANB17H185. M. R. U. Saputra was supported by Indonesia Endowment Fund for Education (LPDP). Publisher Copyright: {\textcopyright} 2019 IEEE. Copyright: Copyright 2020 Elsevier B.V., All rights reserved.; IEEE International Conference on Computer Vision 2019, ICCV 2019 ; Conference date: 27-10-2019 Through 02-11-2019",

year = "2019",

doi = "10.1109/ICCV.2019.00035",

language = "English",

isbn = "9781728148045",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "263--272",

editor = "{So Kweon}, In and Nikos Paragios and Ming-Hsuan Yang and Svetlana Lazebnik",

booktitle = "Proceedings - 2019 International Conference on Computer Vision, ICCV 2019",

address = "United States of America",

url = "http://iccv2019.thecvf.com/, https://ieeexplore.ieee.org/xpl/conhome/8972782/proceeding",

}

Saputra, MRU, Gusmao, P, Almalioglu, Y, Markham, A & Trigoni, N 2019, Distilling knowledge from a deep pose regressor network. in I So Kweon, N Paragios, M-H Yang & S Lazebnik (eds), Proceedings - 2019 International Conference on Computer Vision, ICCV 2019. Proceedings of the IEEE International Conference on Computer Vision, vol. 2019-October, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 263-272, IEEE International Conference on Computer Vision 2019, Seoul, Korea, South, 27/10/19. https://doi.org/10.1109/ICCV.2019.00035

Distilling knowledge from a deep pose regressor network. / Saputra, Muhamad Risqi U.; Gusmao, Pedro; Almalioglu, Yasin et al.
Proceedings - 2019 International Conference on Computer Vision, ICCV 2019. ed. / In So Kweon; Nikos Paragios; Ming-Hsuan Yang; Svetlana Lazebnik. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2019. p. 263-272 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2019-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Distilling knowledge from a deep pose regressor network

AU - Saputra, Muhamad Risqi U.

AU - Gusmao, Pedro

AU - Almalioglu, Yasin

AU - Markham, Andrew

AU - Trigoni, Niki

N1 - Conference code: 17th

PY - 2019

Y1 - 2019

N2 - This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.

AB - This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.

UR - http://www.scopus.com/inward/record.url?scp=85079761935&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2019.00035

DO - 10.1109/ICCV.2019.00035

M3 - Conference Paper

AN - SCOPUS:85079761935

SN - 9781728148045

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 263

EP - 272

BT - Proceedings - 2019 International Conference on Computer Vision, ICCV 2019

A2 - So Kweon, In

A2 - Paragios, Nikos

A2 - Yang, Ming-Hsuan

A2 - Lazebnik, Svetlana

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

T2 - IEEE International Conference on Computer Vision 2019

Y2 - 27 October 2019 through 2 November 2019

ER -

Saputra MRU, Gusmao P, Almalioglu Y, Markham A, Trigoni N. Distilling knowledge from a deep pose regressor network. In So Kweon I, Paragios N, Yang MH, Lazebnik S, editors, Proceedings - 2019 International Conference on Computer Vision, ICCV 2019. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2019. p. 263-272. (Proceedings of the IEEE International Conference on Computer Vision). doi: 10.1109/ICCV.2019.00035

Distilling knowledge from a deep pose regressor network

Abstract

Publication series

Conference

Access to Document

Other files and links

Cite this