Unsupervised joint feature learning and encoding for RGB-D scene labeling

Anran Wang, Jiwen Lu, Jianfei Cai, Gang Wang, Tat Jen Cham

Research output: Contribution to journalArticleResearchpeer-review

19 Citations (Scopus)


Most existing approaches for RGB-D indoor scene labeling employ hand-crafted features for each modality independently and combine them in a heuristic manner. There has been some attempt on directly learning features from raw RGB-D data, but the performance is not satisfactory. In this paper, we propose an unsupervised joint feature learning and encoding (JFLE) framework for RGB-D scene labeling. The main novelty of our learning framework lies in the joint optimization of feature learning and feature encoding in a coherent way, which significantly boosts the performance. By stacking basic learning structure, higher level features are derived and combined with lower level features for better representing RGB-D data. Moreover, to explore the nonlinear intrinsic characteristic of data, we further propose a more general joint deep feature learning and encoding (JDFLE) framework that introduces the nonlinear mapping into JFLE. The experimental results on the benchmark NYU depth dataset show that our approaches achieve competitive performance, compared with the state-of-The-Art methods, while our methods do not need complex feature handcrafting and feature combination and can be easily applied to other data sets.

Original languageEnglish
Article number7185416
Pages (from-to)4459-4473
Number of pages15
JournalIEEE Transactions on Image Processing
Issue number11
Publication statusPublished - 11 Aug 2015
Externally publishedYes


  • joint feature learning and encoding
  • multi-modality
  • RGB-D scene labeling
  • unsupervised feature learning

Cite this