Skip to main navigation Skip to search Skip to main content

Efficient Riemannian Meta-Optimization by Implicit Differentiation

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

To solve optimization problems with nonlinear constrains, the recently developed Riemannian meta-optimization methods show promise, which train neural networks as an optimizer to perform optimization on Riemannian manifolds. A key challenge is the heavy computational and memory burdens, because computing the meta-gradient with respect to the optimizer involves a series of time-consuming derivatives, and stores large computation graphs in memory. In this paper, we propose an efficient Riemannian meta-optimization method that decouples the complex computation scheme from the meta-gradient. We derive Riemannian implicit differentiation to compute the meta-gradient by establishing a link between Riemannian optimization and the implicit function theorem. As a result, the updating our optimizer is only related to the final two iterations, which in turn speeds up our method and reduces the memory footprint significantly. We theoretically study the computational load and memory footprint of our method for long optimization trajectories, and conduct an empirical study to demonstrate the benefits of the proposed method. Evaluations of three optimization problems on different Riemannian manifolds show that our method achieves state-of-the-art performance in terms of the convergence speed and the quality of optima.

Original languageEnglish
Title of host publicationThirty-Sixth AAAI Conference on Artificial Intelligence
EditorsVasant Honavar, Matthijs Spaan
Place of PublicationPalo Alto CA USA
PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
Pages3733-3740
Number of pages8
Volume36
ISBN (Electronic)1577358767, 9781577358763
DOIs
Publication statusPublished - 2022
EventAAAI Conference on Artificial Intelligence 2022 - Online, United States of America
Duration: 22 Feb 2022 → 1 Mar 2022
Conference number: 36th
https://aaai-2022.virtualchair.net/index.html (Website)
https://aaai.org/conference/aaai/aaai-22/
https://ojs.aaai.org/index.php/AAAI/issue/view/510 (Proceedings)

Conference

ConferenceAAAI Conference on Artificial Intelligence 2022
Abbreviated titleAAAI 2022
Country/TerritoryUnited States of America
CityOnline
Period22/02/22 → 1/03/22
Internet address

Keywords

  • Constraint Satisfaction And Optimization
  • Machine Learning

Cite this