Direct quantization for training highly accurate low bit-width deep neural networks

Tuan Hoang, Thanh-Toan Do, Tam V. Nguyen, Ngai-Man Cheung

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. However, this approach would result in some mismatch: the gradient descent updates full-precision weights, but it does not update the quantized weights. To address this issue, we propose a novel method that enables direct updating of quantized weights with learnable quantization levels to minimize the cost function using gradient descent. Second, to obtain low bit-width activations, existing works consider all channels equally. However, the activation quantizers could be biased toward a few channels with high-variance. To address this issue, we propose a method to take into account the quantization errors of individual channels. With this approach, we can learn activation quantizers that minimize the quantization errors in the majority of channels. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on the image classification task, using AlexNet, ResNet and MobileNetV2 architectures on CIFAR-100 and ImageNet datasets.

Original languageEnglish
Title of host publicationProceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
EditorsChristian Bessiere
Place of PublicationMarina del Rey CA USA
PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
Pages2111-2118
Number of pages8
ISBN (Electronic)9780999241165
DOIs
Publication statusPublished - 2020
Externally publishedYes
EventInternational Joint Conference on Artificial Intelligence-Pacific Rim International Conference on Artificial Intelligence 2020 - Yokohama, Japan
Duration: 7 Jan 202115 Jan 2021
Conference number: 29th/17th
https://www.ijcai.org/Proceedings/2020/ (Proceedings)
https://ijcai20.org (Website)

Conference

ConferenceInternational Joint Conference on Artificial Intelligence-Pacific Rim International Conference on Artificial Intelligence 2020
Abbreviated titleIJCAI-PRICAI 2020
CountryJapan
CityYokohama
Period7/01/2115/01/21
OtherIJCAI-PRICAI 2020, the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence!IJCAI-PRICAI2020 will take place January 7-15, 2021 online in a virtual reality in Japanese Standard Time (JST) zone.
Internet address

Keywords

  • Machine Learning
  • Deep Learning

Cite this