Projects per year
Abstract
Decoding in autoregressive models (ARMs) consists of searching for a high scoring output sequence under the trained model. Standard decoding methods, based on unidirectional greedy algorithm or beam search, are suboptimal due to error propagation and myopic decisions which do not account for future steps in the generation process. In this paper we present a novel decoding approach based on the method of auxiliary coordinates (Carreira-Perpinan & Wang, 2014) to address the aforementioned shortcomings. Our method introduces discrete variables for output tokens, and auxiliary continuous variables representing the states of the underlying ARM. The auxiliary variables lead to a factor graph approximation of the ARM, whose maximum a posteriori (MAP) solution is found exactly using dynamic programming. The MAP solution is then used to recreate an improved factor graph approximation of the ARM via updated auxiliary variables. We then extend our approach to decode in an ensemble of ARMs, possibly with different generation orders, which is out of reach for the standard unidirectional decoding algorithms. Experiments on the text infilling task over SWAG and Daily Dialogue datasets show that our decoding method is superior to strong competing decoding methods.
Original language | English |
---|---|
Title of host publication | International Conference on Learning Representations, ICLR 2020 |
Editors | Shakir Mohamed |
Place of Publication | Portland OR USA |
Publisher | OpenReview |
Number of pages | 11 |
Publication status | Published - 2020 |
Event | International Conference on Learning Representations 2020 - Virtual, Addis Ababa, Ethiopia Duration: 26 Apr 2020 → 30 Apr 2020 Conference number: 8th https://iclr.cc/virtual_2020/index.html https://openreview.net/group?id=ICLR.cc/2020/Conference (Proceedings) |
Conference
Conference | International Conference on Learning Representations 2020 |
---|---|
Abbreviated title | ICLR 2020 |
Country/Territory | Ethiopia |
City | Addis Ababa |
Period | 26/04/20 → 30/04/20 |
Internet address |
Projects
- 1 Finished
-
Learning Deep Semantics for Automatic Translation between Human Languages
Haffari, R., Cohn, T. & Blunsom, P.
Australian Research Council (ARC)
1/01/16 → 31/12/18
Project: Research