Towards effective low-bitwidth convolutional neural networks

Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, Ian Reid

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearch

42 Citations (Scopus)

Abstract

This paper tackles the problem of training a deep convolutional neural network with both low-precision weights and low-bitwidth activations. Optimizing a low-precision network is very challenging since the training process can easily get trapped in a poor local minima, which results in substantial accuracy loss. To mitigate this problem, we propose three simple-yet-effective approaches to improve the network training. First, we propose to use a two-stage optimization strategy to progressively find good local minima. Specifically, we propose to first optimize a net with quantized weights and then quantized activations. This is in contrast to the traditional methods which optimize them simultaneously. Second, following a similar spirit of the first method, we propose another progressive optimization approach which progressively decreases the bit-width from high-precision to low-precision during the course of training. Third, we adopt a novel learning scheme to jointly train a full-precision model alongside the low-precision one. By doing so, the full-precision model provides hints to guide the low-precision model training. Extensive experiments on various datasets (i.e., CIFAR-100 and ImageNet) show the effectiveness of the proposed methods. To highlight, using our methods to train a 4-bit precision network leads to no performance decrease in comparison with its full-precision counterpart with standard network architectures (i.e., AlexNet and ResNet-50).

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018
EditorsDavid Forsyth, Ivan Laptev, Aude Oliva, Deva Ramanan
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages7920-7928
Number of pages9
ISBN (Electronic)9781538664209
ISBN (Print)9781538664216
DOIs
Publication statusPublished - 2018
Externally publishedYes
EventIEEE Conference on Computer Vision and Pattern Recognition 2018 - Salt Lake City, United States of America
Duration: 19 Jun 201821 Jun 2018
http://cvpr2018.thecvf.com/
https://ieeexplore.ieee.org/xpl/conhome/8576498/proceeding (Proceedings)

Conference

ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2018
Abbreviated titleCVPR 2018
CountryUnited States of America
CitySalt Lake City
Period19/06/1821/06/18
Internet address

Cite this