MIML-FCN+: multi-instance multi-label learning via fully convolutional networks with privileged information

Hao Yang, Joey Tianyi Zhou, Jianfei Cai, Yew Soon Ong

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

19 Citations (Scopus)


Multi-instance multi-label (MIML) learning has many interesting applications in computer visions, including multi-object recognition and automatic image tagging. In these applications, additional information such as bounding-boxes, image captions and descriptions is often available during training phrase, which is referred as privileged information (PI). However, as existing works on learning using PI only consider instance-level PI (privileged instances), they fail to make use of bag-level PI (privileged bags) available in MIML learning. Therefore, in this paper, we propose a two-stream fully convolutional network, named MIML-FCN+, unified by a novel PI loss to solve the problem of MIML learning with privileged bags. Compared to the previous works on PI, the proposed MIML-FCN+ utilizes the readily available privileged bags, instead of hard-to-obtain privileged instances, making the system more general and practical in real world applications. As the proposed PI loss is convex and SGD-compatible and the framework itself is a fully convolutional network, MIML-FCN+ can be easily integrated with state-of-the-art deep learning networks. Moreover, the flexibility of convolutional layers allows us to exploit structured correlations among instances to facilitate more effective training and testing. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed MIML-FCN+, outperforming state-of-the-art methods in the application of multi-object recognition.

Original languageEnglish
Title of host publicationProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
EditorsYanxi Liu, James M. Rehg, Camillo J. Taylor, Ying Wu
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages9
ISBN (Electronic)9781538604571
ISBN (Print)9781538604588
Publication statusPublished - 2017
Externally publishedYes
EventIEEE Conference on Computer Vision and Pattern Recognition 2017 - Honolulu, United States of America
Duration: 21 Jul 201726 Jul 2017


ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2017
Abbreviated titleCVPR 2017
CountryUnited States of America
Internet address

Cite this