Abstract
The task of vision-and-language navigation (VLN) requires an agent to follow text instructions to find its way through simulated household environments. A prominent challenge is to train an agent capable of generalising to new environments at test time, rather than one that simply memorises trajectories and visual details observed during training. We propose a new learning strategy that learns both from observations and generated counterfactual environments. We describe an effective algorithm to generate counterfactual observations on the fly for VLN, as linear combinations of existing environments. Simultaneously, we encourage the agent’s actions to remain stable between original and counterfactual environments through our novel training objective – effectively removing spurious features that would otherwise bias the agent. Our experiments show that this technique provides significant improvements in generalisation on benchmarks for Room-to-Room navigation and Embodied Question Answering.
Original language | English |
---|---|
Title of host publication | Advances in Neural Information Processing Systems 33 (NeurIPS 2020) |
Editors | H. Lorochelle, M. Ranzato, R. Hadsell, M.F. Balcan, H. Lin |
Place of Publication | San Diego CA USA |
Publisher | Neural Information Processing Systems (NIPS) |
Number of pages | 12 |
Publication status | Published - 2020 |
Externally published | Yes |
Event | Advances in Neural Information Processing Systems 2020 - Virtual, Online, United States of America Duration: 6 Dec 2020 → 12 Dec 2020 Conference number: 34th https://proceedings.neurips.cc/paper/2020 (Proceedings ) https://nips.cc/Conferences/2020 (Website) |
Publication series
Name | Advances in Neural Information Processing Systems |
---|---|
Publisher | Neural Information Processing Systems (NIPS) |
Volume | 2020-December |
ISSN (Electronic) | 1049-5258 |
Conference
Conference | Advances in Neural Information Processing Systems 2020 |
---|---|
Abbreviated title | NeurIPS 2020 |
Country/Territory | United States of America |
City | Virtual, Online |
Period | 6/12/20 → 12/12/20 |
Internet address |
|