On the effect of isotropy on VAE representations of text

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Injecting desired geometric properties into text representations has attracted a lot of attention. A property that has been argued for, due to its better utilisation of representation space, is isotropy. In parallel, VAEs have been successful in areas of NLP, but are known for their sub-optimal utilisation of the representation space. To address an aspect of this, we investigate the impact of injecting isotropy during training of VAEs. We achieve this by using an isotropic Gaussian posterior (IGP) instead of the ellipsoidal Gaussian posterior. We illustrate that IGP effectively encourages isotropy in the representations, inducing a more discriminative latent space. Compared to vanilla VAE, this translates into a much better classification performance, robustness to input perturbation, and generative behavior. Additionally, we offer insights about the representational properties encouraged by IGP.
Original languageEnglish
Title of host publicationACL 2022, The 60th Annual Meeting of the Association for Computational Linguistics
Subtitle of host publicationProceedings of the Conference, Vol. 2 (Short Papers)
EditorsDanilo Croce, Ryan Cotterell, Jordan Zhang
Place of PublicationStroudsburg PA USA
PublisherAssociation for Computational Linguistics (ACL)
Pages694–701
Number of pages8
ISBN (Electronic)9781955917223
Publication statusPublished - 2022
EventAnnual Meeting of the Association for Computational Linguistics 2022 - Dublin, Ireland
Duration: 22 May 202227 May 2022
Conference number: 60th
https://aclanthology.org/volumes/2022.acl-short/ (Proceedings - Short)
https://aclanthology.org/volumes/2022.acl-long/ (Proceedings - Long)
https://www.2022.aclweb.org/ (Website)

Conference

ConferenceAnnual Meeting of the Association for Computational Linguistics 2022
Abbreviated titleACL 2022
Country/TerritoryIreland
CityDublin
Period22/05/2227/05/22
Internet address

Cite this