Compact trilinear interaction for Visual Question Answering

Tuong Do, Huy Tran, Thanh-Toan Do, Erman Tjiputra, Quang Tran

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

3 Citations (Scopus)

Abstract

In Visual Question Answering (VQA), answers have a great correlation with question meaning and visual contents. Thus, to selectively utilize image, question and answer information, we propose a novel trilinear interaction model which simultaneously learns high level associations between these three inputs. In addition, to overcome the interaction complexity, we introduce a multimodal tensor-based PARALIND decomposition which efficiently parameterizes trilinear teraction between the three inputs. Moreover, knowledge distillation is first time applied in Free-form Opened-ended VQA. It is not only for reducing the computational cost and required memory but also for transferring knowledge from trilinear interaction model to bilinear interaction model. The extensive experiments on benchmarking datasets TDIUC, VQA-2.0, and Visual7W show that the proposed compact trilinear interaction model achieves state-of-the-art results when using a single model on all three datasets.

Original languageEnglish
Title of host publicationProceedings - 2019 International Conference on Computer Vision, ICCV 2019
EditorsIn So Kweon, Nikos Paragios, Ming-Hsuan Yang, Svetlana Lazebnik
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages392-401
Number of pages10
ISBN (Electronic)9781728148038
ISBN (Print)9781728148045
DOIs
Publication statusPublished - 2019
Externally publishedYes
EventIEEE International Conference on Computer Vision 2019 - Seoul, Korea, Republic of (South)
Duration: 27 Oct 20192 Nov 2019
Conference number: 17th
http://iccv2019.thecvf.com/
https://ieeexplore.ieee.org/xpl/conhome/8972782/proceeding (Proceedings)

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
PublisherThe Institute of Electrical and Electronics Engineers, Inc.
Volume2019-October
ISSN (Print)1550-5499
ISSN (Electronic)2380-7504

Conference

ConferenceIEEE International Conference on Computer Vision 2019
Abbreviated titleICCV 2019
CountryKorea, Republic of (South)
CitySeoul
Period27/10/192/11/19
Internet address

Cite this