TY - JOUR
T1 - A comprehensive survey of neural architecture search
T2 - challenges and solutions
AU - Ren, Pengzhen
AU - Xiao, Yun
AU - Chang, Xiaojun
AU - Huang, Po-Yao
AU - Li, Zhihui
AU - Chen, Xiaojiang
AU - Wang, Xin
N1 - Funding Information:
Pengzhen Ren and Yun Xiao contributed equally to this research. This work was partially supported by the NSFC under Grants No. 61972315 and No. 61906109, the Shaanxi Science and Technology Innovation Team Support Project under Grant Agreement No. 2018TD-026, and the Australian Research Council Discovery Early Career Researcher Award No. DE190100626. Authors’ addresses: P. Ren and Y. Xiao, No.1, Xuefu Avenue, Changan District, Xian City, Shanxi Province, 710127, China; emails: [email protected], [email protected]; X. Chang, Monash University, No.1, Xuefu Avenue, Changan District, Xian City, Shanxi Province, 710127, China; email: [email protected]; P.-Y. Huang, Carnegie Mellon University, 5000 Forbes AVE Pittsburgh PA 15213; Z. Li (corresponding author), Qilu University of Technology (Shandong Academy of Sciences), 19th Keyuan Road Lixia District, Jinan, Shandong Province China, 250014; X. Chen and X. Wang, Northwest University, 324 Wellington Rd, Clayton VIC 3800, Australia. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2021 Association for Computing Machinery. 0360-0300/2021/05-ART76 $15.00 https://doi.org/10.1145/3447582
Publisher Copyright:
© 2021 ACM.
PY - 2021/5
Y1 - 2021/5
N2 - Deep learning has made substantial breakthroughs in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers' prior knowledge and experience. And due to the limitations of humans' inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. In addition, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.
AB - Deep learning has made substantial breakthroughs in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers' prior knowledge and experience. And due to the limitations of humans' inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. In addition, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.
KW - AutoDL
KW - Continuous search strategy
KW - Incomplete training
KW - Modular search space
KW - Neural architecture recycle
KW - Neural architecture search
UR - https://www.scopus.com/pages/publications/85109212553
U2 - 10.1145/3447582
DO - 10.1145/3447582
M3 - Review Article
AN - SCOPUS:85109212553
SN - 0360-0300
VL - 54
JO - ACM Computing Surveys
JF - ACM Computing Surveys
IS - 4
M1 - 76
ER -