Constrained Stochastic Gradient Descent: The Good Practice

Soumava Kumar Roy, Mehrtash Harandi

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

1 Citation (Scopus)

Abstract

Stochastic Gradient Descent (SGD) is the method of choice for large scale problems, most notably in deep learning. Recent studies target improving convergence and speed of the SGD algorithm. In this paper, we equip the SGD algorithm and its advanced versions with an intriguing feature, namely handling constrained problems. Constraints such as orthogonality are pervasive in learning theory. Nevertheless and to some extent surprising, constrained SGD algorithms are rarely studied. Our proposal makes use of Riemannian geometry and accelerated optimization techniques to deliver efficient and constrained-aware SGD methods.We will assess and contrast our proposed approaches in a wide range of problems including incremental dimensionality reduction, karcher mean and deep metric learning.

Original languageEnglish
Title of host publication2017 International Conference on Digital Image Computing
Subtitle of host publicationTechniques and Applications (DICTA 2017)
EditorsYi Guo, Hongdong Li, Tom Cai, Manzur Murshed
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages8
ISBN (Electronic)9781538628393
ISBN (Print)9781538628409
DOIs
Publication statusPublished - 19 Dec 2017
Externally publishedYes
EventDigital Image Computing Techniques and Applications 2017
- Novotel Sydney Manly Pacific , Sydney, Australia
Duration: 29 Nov 20171 Dec 2017
http://dicta2017.dictaconference.org/

Conference

ConferenceDigital Image Computing Techniques and Applications 2017
Abbreviated titleDICTA 2017
CountryAustralia
CitySydney
Period29/11/171/12/17
Internet address

Cite this