Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks

Yujun  Cai; Liuhao  Ge; Jun Liu; Jianfei Cai; Tat-Jen Cham; Junsong Yuan; Nadia Magnenat Thalmann

doi:10.1109/ICCV.2019.00236

Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks

Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, Nadia Magnenat Thalmann

Department of Data Science & AI

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

430 Citations (Scopus)

Abstract

Despite great progress in 3D pose estimation from single-view images or videos, it remains a challenging task due to the substantial depth ambiguity and severe self-
occlusions. Motivated by the effectiveness of incorporating spatial dependencies and temporal consistencies to alleviate these issues, we propose a novel graph-based method
to tackle the problem of 3D human body and 3D hand pose estimation from a short sequence of 2D joint detections. Particularly, domain knowledge about the human hand (body) configurations is explicitly incorporated into the graph convolutional operations to meet the specific demand of the 3D pose estimation. Furthermore, we introduce a local-to-global network architecture, which is capable of learning multi-scale features for the graph-based representations. We evaluate the proposed method on challenging benchmark datasets for both 3D hand pose estimation and 3D body pose estimation. Experimental results show that our method achieves state-of-the-art performance on both
tasks.

Original language	English
Title of host publication	Proceedings - IEEE International Conference on Computer Vision, ICCV 2019
Editors	In So Kweon, Nikos Paragios, Ming-Hsuan Yang, Svetlana Lazebnik
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	2272-2281
Number of pages	10
ISBN (Electronic)	9781728148038
ISBN (Print)	9781728148045
DOIs	https://doi.org/10.1109/ICCV.2019.00236
Publication status	Published - 2019
Event	IEEE International Conference on Computer Vision 2019 - Seoul, Korea, South Duration: 27 Oct 2019 → 2 Nov 2019 Conference number: 17th http://iccv2019.thecvf.com/ https://ieeexplore.ieee.org/xpl/conhome/8972782/proceeding (Proceedings)

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Volume	2019-October
ISSN (Print)	1550-5499
ISSN (Electronic)	2380-7504

Conference

Conference	IEEE International Conference on Computer Vision 2019
Abbreviated title	ICCV 2019
Country/Territory	Korea, South
City	Seoul
Period	27/10/19 → 2/11/19
Internet address	http://iccv2019.thecvf.com/ https://ieeexplore.ieee.org/xpl/conhome/8972782/proceeding (Proceedings)

Access to Document

10.1109/ICCV.2019.00236

296784820-oaFinal published version, 1.19 MB

Cite this

Cai, Y., Ge, L., Liu, J., Cai, J., Cham, T.-J., Yuan, J., & Thalmann, N. M. (2019). Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks. In I. So Kweon, N. Paragios, M.-H. Yang, & S. Lazebnik (Eds.), Proceedings - IEEE International Conference on Computer Vision, ICCV 2019 (pp. 2272-2281). (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2019-October). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICCV.2019.00236

Cai, Yujun ; Ge, Liuhao ; Liu, Jun et al. / Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks. Proceedings - IEEE International Conference on Computer Vision, ICCV 2019. editor / In So Kweon ; Nikos Paragios ; Ming-Hsuan Yang ; Svetlana Lazebnik. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. pp. 2272-2281 (Proceedings of the IEEE International Conference on Computer Vision).

@inproceedings{33d770fef4154acdb44c5e1407869e69,

title = "Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks",

abstract = "Despite great progress in 3D pose estimation from single-view images or videos, it remains a challenging task due to the substantial depth ambiguity and severe self-occlusions. Motivated by the effectiveness of incorporating spatial dependencies and temporal consistencies to alleviate these issues, we propose a novel graph-based methodto tackle the problem of 3D human body and 3D hand pose estimation from a short sequence of 2D joint detections. Particularly, domain knowledge about the human hand (body) configurations is explicitly incorporated into the graph convolutional operations to meet the specific demand of the 3D pose estimation. Furthermore, we introduce a local-to-global network architecture, which is capable of learning multi-scale features for the graph-based representations. We evaluate the proposed method on challenging benchmark datasets for both 3D hand pose estimation and 3D body pose estimation. Experimental results show that our method achieves state-of-the-art performance on bothtasks.",

author = "Yujun Cai and Liuhao Ge and Jun Liu and Jianfei Cai and Tat-Jen Cham and Junsong Yuan and Thalmann, {Nadia Magnenat}",

year = "2019",

doi = "10.1109/ICCV.2019.00236",

language = "English",

isbn = "9781728148045",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "2272--2281",

editor = "{So Kweon}, {In } and Paragios, {Nikos } and Yang, {Ming-Hsuan } and Svetlana Lazebnik",

booktitle = "Proceedings - IEEE International Conference on Computer Vision, ICCV 2019",

address = "United States of America",

note = "IEEE International Conference on Computer Vision 2019, ICCV 2019 ; Conference date: 27-10-2019 Through 02-11-2019",

url = "http://iccv2019.thecvf.com/, https://ieeexplore.ieee.org/xpl/conhome/8972782/proceeding",

}

Cai, Y, Ge, L, Liu, J, Cai, J, Cham, T-J, Yuan, J & Thalmann, NM 2019, Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks. in I So Kweon, N Paragios, M-H Yang & S Lazebnik (eds), Proceedings - IEEE International Conference on Computer Vision, ICCV 2019. Proceedings of the IEEE International Conference on Computer Vision, vol. 2019-October, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 2272-2281, IEEE International Conference on Computer Vision 2019, Seoul, Korea, South, 27/10/19. https://doi.org/10.1109/ICCV.2019.00236

Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks. / Cai, Yujun ; Ge, Liuhao ; Liu, Jun et al.
Proceedings - IEEE International Conference on Computer Vision, ICCV 2019. ed. / In So Kweon; Nikos Paragios; Ming-Hsuan Yang; Svetlana Lazebnik. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2019. p. 2272-2281 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2019-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks

AU - Cai, Yujun

AU - Ge, Liuhao

AU - Liu, Jun

AU - Cai, Jianfei

AU - Cham, Tat-Jen

AU - Yuan, Junsong

AU - Thalmann, Nadia Magnenat

N1 - Conference code: 17th

PY - 2019

Y1 - 2019

N2 - Despite great progress in 3D pose estimation from single-view images or videos, it remains a challenging task due to the substantial depth ambiguity and severe self-occlusions. Motivated by the effectiveness of incorporating spatial dependencies and temporal consistencies to alleviate these issues, we propose a novel graph-based methodto tackle the problem of 3D human body and 3D hand pose estimation from a short sequence of 2D joint detections. Particularly, domain knowledge about the human hand (body) configurations is explicitly incorporated into the graph convolutional operations to meet the specific demand of the 3D pose estimation. Furthermore, we introduce a local-to-global network architecture, which is capable of learning multi-scale features for the graph-based representations. We evaluate the proposed method on challenging benchmark datasets for both 3D hand pose estimation and 3D body pose estimation. Experimental results show that our method achieves state-of-the-art performance on bothtasks.

AB - Despite great progress in 3D pose estimation from single-view images or videos, it remains a challenging task due to the substantial depth ambiguity and severe self-occlusions. Motivated by the effectiveness of incorporating spatial dependencies and temporal consistencies to alleviate these issues, we propose a novel graph-based methodto tackle the problem of 3D human body and 3D hand pose estimation from a short sequence of 2D joint detections. Particularly, domain knowledge about the human hand (body) configurations is explicitly incorporated into the graph convolutional operations to meet the specific demand of the 3D pose estimation. Furthermore, we introduce a local-to-global network architecture, which is capable of learning multi-scale features for the graph-based representations. We evaluate the proposed method on challenging benchmark datasets for both 3D hand pose estimation and 3D body pose estimation. Experimental results show that our method achieves state-of-the-art performance on bothtasks.

UR - http://www.scopus.com/inward/record.url?scp=85077736660&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2019.00236

DO - 10.1109/ICCV.2019.00236

M3 - Conference Paper

AN - SCOPUS:85077736660

SN - 9781728148045

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 2272

EP - 2281

BT - Proceedings - IEEE International Conference on Computer Vision, ICCV 2019

A2 - So Kweon, In

A2 - Paragios, Nikos

A2 - Yang, Ming-Hsuan

A2 - Lazebnik, Svetlana

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

T2 - IEEE International Conference on Computer Vision 2019

Y2 - 27 October 2019 through 2 November 2019

ER -

Cai Y, Ge L, Liu J, Cai J, Cham TJ, Yuan J et al. Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks. In So Kweon I, Paragios N, Yang MH, Lazebnik S, editors, Proceedings - IEEE International Conference on Computer Vision, ICCV 2019. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2019. p. 2272-2281. (Proceedings of the IEEE International Conference on Computer Vision). doi: 10.1109/ICCV.2019.00236

Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks

Abstract

Publication series

Conference

Access to Document

Other files and links

Cite this