VMLoc: variational fusion for learning-based multimodal camera localization

Kaichen Zhou, Changhao Chen, Bing Wang, Muhamad Risqi U. Saputra, Niki Trigoni, Andrew Markham

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

11 Citations (Scopus)


Recent learning-based approaches have achieved impressive results in the field of single-shot camera localization. However, how best to fuse multiple modalities (e.g., image and depth) and to deal with degraded or missing input are less well studied. In particular, we note that previous approaches towards deep fusion do not perform significantly better than models employing a single modality. We conjecture that this is because of the naive approaches to feature space fusion through summation or concatenation which do not take into account the different strengths of each modality. To address this, we propose an end-to-end framework, termed VMLoc, to fuse different sensor inputs into a common latent space through a variational Product-of-Experts (PoE) followed by attention-based fusion. Unlike previous multimodal variational works directly adapting the objective function of vanilla variational auto-encoder, we show how camera localization can be accurately estimated through an unbiased objective function based on importance weighting. Our model is extensively evaluated on RGB-D datasets and the results prove the efficacy of our model. The source code is available at https://github.com/Zalex97/VMLoc.

Original languageEnglish
Title of host publicationProceedings of the AAAI Conference on Artificial Intelligence, AAAI-21
EditorsKevin Leyton-Brown, Mausam
Place of PublicationPalo Alto CA USA
PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
Number of pages9
ISBN (Electronic)9781577358664
Publication statusPublished - 2021
Externally publishedYes
EventAAAI Conference on Artificial Intelligence 2021 - Online, United States of America
Duration: 2 Feb 20219 Feb 2021
Conference number: 35th
https://aaai.org/Conferences/AAAI-21/ (Website)

Publication series

Name35th AAAI Conference on Artificial Intelligence, AAAI 2021


ConferenceAAAI Conference on Artificial Intelligence 2021
Abbreviated titleAAAI 2021
Country/TerritoryUnited States of America
Internet address


  • Localization
  • Mapping
  • Navigation

Cite this