Differentiable neural architecture search in equivalent space with exploration enhancement

Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, Zongyuan Ge, Steven Su

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

5 Citations (Scopus)

Abstract

Recent works on One-Shot Neural Architecture Search (NAS) mostly adopt a bilevel optimization scheme to alternatively optimize the supernet weights and architecture parameters after relaxing the discrete search space into a differentiable space. However, the non-negligible incongruence in their relaxation methods is hard to guarantee the differentiable optimization in the continuous space is equivalent to the optimization in the discrete space. Differently, this paper utilizes a variational graph autoencoder to injectively transform the discrete architecture space into an equivalently continuous latent space, to resolve the incongruence. A probabilistic exploration enhancement method is accordingly devised to encourage intelligent exploration during the architecture search in the latent space, to avoid local optimal in architecture search. As the catastrophic forgetting in differentiable One-Shot NAS deteriorates supernet predictive ability and makes the bilevel optimization inefficient, this paper further proposes an architecture complementation method to relieve this deficiency. We analyze the proposed method’s effectiveness, and a series of experiments have been conducted to compare the proposed method with state-of-the-art One-Shot NAS methods.

Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems 33 (NeurIPS 2020)
EditorsH. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, H. Lin
Place of PublicationSan Diego CA USA
PublisherNeural Information Processing Systems (NIPS)
Number of pages11
Publication statusPublished - 2020
EventAdvances of Neural Information Processing Systems 2020 - Online, Virtual, Online, United States of America
Duration: 6 Dec 202012 Dec 2020
Conference number: 34th
https://proceedings.neurips.cc/paper/2020 (Proceedings )
https://nips.cc/Conferences/2020 (Website)

Publication series

NameAdvances in Neural Information Processing Systems
PublisherMorgan Kaufmann Publishers
Volume2020-December
ISSN (Print)1049-5258

Conference

ConferenceAdvances of Neural Information Processing Systems 2020
Abbreviated titleNeurIPS 2020
Country/TerritoryUnited States of America
CityVirtual, Online
Period6/12/2012/12/20
Internet address

Cite this