Counterfactual vision-and-language navigation: Unravelling the unseen

Amin Parvaneh, Ehsan Abbasnejad, Damien Teney, Javen Qinfeng Shi, Anton van den Hengel

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

26 Citations (Scopus)

Abstract

The task of vision-and-language navigation (VLN) requires an agent to follow text instructions to find its way through simulated household environments. A prominent challenge is to train an agent capable of generalising to new environments at test time, rather than one that simply memorises trajectories and visual details observed during training. We propose a new learning strategy that learns both from observations and generated counterfactual environments. We describe an effective algorithm to generate counterfactual observations on the fly for VLN, as linear combinations of existing environments. Simultaneously, we encourage the agent’s actions to remain stable between original and counterfactual environments through our novel training objective – effectively removing spurious features that would otherwise bias the agent. Our experiments show that this technique provides significant improvements in generalisation on benchmarks for Room-to-Room navigation and Embodied Question Answering.

Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems 33 (NeurIPS 2020)
EditorsH. Lorochelle, M. Ranzato, R. Hadsell, M.F. Balcan, H. Lin
Place of PublicationSan Diego CA USA
PublisherNeural Information Processing Systems (NIPS)
Number of pages12
Publication statusPublished - 2020
Externally publishedYes
EventAdvances in Neural Information Processing Systems 2020 - Virtual, Online, United States of America
Duration: 6 Dec 202012 Dec 2020
Conference number: 34th
https://proceedings.neurips.cc/paper/2020 (Proceedings )
https://nips.cc/Conferences/2020 (Website)

Publication series

NameAdvances in Neural Information Processing Systems
PublisherNeural Information Processing Systems (NIPS)
Volume2020-December
ISSN (Electronic)1049-5258

Conference

ConferenceAdvances in Neural Information Processing Systems 2020
Abbreviated titleNeurIPS 2020
Country/TerritoryUnited States of America
CityVirtual, Online
Period6/12/2012/12/20
Internet address

Cite this