Parameterized rate-distortion stochastic encoder

Quan Hoang, Trung Le, Dinh Phung

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


We propose a novel gradient-based tractable approach for the Blahut-Arimoto (BA) algorithm to compute the rate-distortion function where the BA algorithm is fully parameterized. This results in a rich and flexible framework to learn a new class of stochastic encoders, termed PArameterized RAteDIstortion Stochastic Encoder (PARADISE). The framework can be applied to a wide range of settings from semi-supervised, multi-task to supervised and robust learning. We show that the training objective of PARADISE can be seen as a form of regularization that helps improve generalization. With an emphasis on robust learning we further develop a novel posterior matching objective to encourage smoothness on the loss function and show that PARADISE can significantly improve interpretability as well as robustness to adversarial attacks on the CIFAR-10 and ImageNet datasets. In particular, on the CIFAR-10 dataset, our model reduces standard and adversarial error rates in comparison to the state-of-the-art by 50% and 41%, respectively without the expensive computational cost of adversarial training.

Original languageEnglish
Title of host publication37th International Conference on Machine Learning (ICML 2020)
Subtitle of host publicationProceedings of Machine Learning Research Volume 119
EditorsHal Daume III, Aarti Singh
Place of PublicationStroudsburg PA USA
PublisherInternational Machine Learning Society (IMLS)
Number of pages11
ISBN (Electronic)9781713821120
Publication statusPublished - 2020
EventInternational Conference on Machine Learning 2020 - Online, Online, United States of America
Duration: 13 Jul 202018 Jul 2020
Conference number: 37th (Proceedings) (Website)

Publication series

Name37th International Conference on Machine Learning, ICML 2020
PublisherProceedings of Machine Learning Research


ConferenceInternational Conference on Machine Learning 2020
Abbreviated titleICML 2020
Country/TerritoryUnited States of America
Internet address

Cite this