Geometry aware constrained optimization techniques for deep learning

Soumava Kumar Roy, Zakaria Mhammedi, Mehrtash Harandi

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

10 Citations (Scopus)

Abstract

In this paper, we generalize the Stochastic Gradient Descent (SGD) and RMSProp algorithms to the setting of Riemannian optimization. SGD is a popular method for large scale optimization. In particular, it is widely used to train the weights of Deep Neural Networks. However, gradients computed using standard SGD can have large variance, which is detrimental for the convergence rate of the algorithm. Other methods such as RMSProp and ADAM address this issue. Nevertheless, these methods cannot be directly applied to constrained optimization problems. In this paper, we extend some popular optimization algorithm to the Riemannian (constrained) setting. We substantiate our proposed extensions with a range of relevant problems in machine learning such as incremental Principal Component Analysis, computating the Riemannian centroids of SPD matrices, and Deep Metric Learning. We achieve competitive results against the state of the art for fine-grained object recognition datasets.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
EditorsDavid Forsyth, Ivan Laptev, Aude Oliva, Deva Ramanan
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages4460-4469
Number of pages10
ISBN (Electronic)9781538664209
ISBN (Print)9781538664216
DOIs
Publication statusPublished - 2018
Externally publishedYes
EventIEEE Conference on Computer Vision and Pattern Recognition 2018 - Salt Lake City, United States of America
Duration: 19 Jun 201821 Jun 2018
http://cvpr2018.thecvf.com/
https://ieeexplore.ieee.org/xpl/conhome/8576498/proceeding (Proceedings)

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISSN (Print)1063-6919

Conference

ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2018
Abbreviated titleCVPR 2018
CountryUnited States of America
CitySalt Lake City
Period19/06/1821/06/18
Internet address

Cite this