Vision-based System Identification and 3D keypoint discovery using dynamics constraints

Miguel Jaques; Martin Asenov; Michael Burke; Timothy Hospedales

Vision-based System Identification and 3D keypoint discovery using dynamics constraints

Miguel Jaques, Martin Asenov, Michael Burke, Timothy Hospedales

Electrical and Computer Systems Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach

Original language	English
Title of host publication	Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022
Editors	Roya Firoozi, Negar Mehr, Esen Yel, Rika Antonova, Jeannette Bohg, Mac Schwager, Mykel Kochenderfer
Place of Publication	London UK
Publisher	Proceedings of Machine Learning Research (PMLR)
Pages	316-329
Number of pages	14
Publication status	Published - 2022
Event	Annual Learning for Dynamics and Control Conference 2022 - Stanford, United States of America Duration: 23 Jun 2022 → 24 Jun 2022 Conference number: 4th https://proceedings.mlr.press/v168/ (Proceedings) https://l4dc.stanford.edu/ (Website)

Publication series

Name	Proceedings of Machine Learning Research
Volume	168
ISSN (Electronic)	2640-3498

Conference

Conference	Annual Learning for Dynamics and Control Conference 2022
Abbreviated title	L4DC
Country/Territory	United States of America
City	Stanford
Period	23/06/22 → 24/06/22
Internet address	https://proceedings.mlr.press/v168/ (Proceedings) https://l4dc.stanford.edu/ (Website)

Access to Document

403924415-oaFinal published version, 9.12 MB

https://proceedings.mlr.press/v168/jaques22a.html

Cite this

Jaques, M., Asenov, M., Burke, M., & Hospedales, T. (2022). Vision-based System Identification and 3D keypoint discovery using dynamics constraints. In R. Firoozi, N. Mehr, E. Yel, R. Antonova, J. Bohg, M. Schwager, & M. Kochenderfer (Eds.), Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022 (pp. 316-329). (Proceedings of Machine Learning Research; Vol. 168). Proceedings of Machine Learning Research (PMLR). https://proceedings.mlr.press/v168/jaques22a.html

Jaques, Miguel ; Asenov, Martin ; Burke, Michael et al. / Vision-based System Identification and 3D keypoint discovery using dynamics constraints. Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022. editor / Roya Firoozi ; Negar Mehr ; Esen Yel ; Rika Antonova ; Jeannette Bohg ; Mac Schwager ; Mykel Kochenderfer. London UK : Proceedings of Machine Learning Research (PMLR), 2022. pp. 316-329 (Proceedings of Machine Learning Research).

@inproceedings{2058bb3f5d4e41a38977e997c45a1a14,

title = "Vision-based System Identification and 3D keypoint discovery using dynamics constraints",

abstract = "This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach",

author = "Miguel Jaques and Martin Asenov and Michael Burke and Timothy Hospedales",

year = "2022",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "Proceedings of Machine Learning Research (PMLR)",

pages = "316--329",

editor = "Roya Firoozi and Negar Mehr and Esen Yel and Rika Antonova and Jeannette Bohg and Mac Schwager and Mykel Kochenderfer",

booktitle = "Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022",

note = "Annual Learning for Dynamics and Control Conference 2022, L4DC ; Conference date: 23-06-2022 Through 24-06-2022",

url = "https://proceedings.mlr.press/v168/, https://l4dc.stanford.edu/",

}

Jaques, M, Asenov, M, Burke, M & Hospedales, T 2022, Vision-based System Identification and 3D keypoint discovery using dynamics constraints. in R Firoozi, N Mehr, E Yel, R Antonova, J Bohg, M Schwager & M Kochenderfer (eds), Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022. Proceedings of Machine Learning Research, vol. 168, Proceedings of Machine Learning Research (PMLR), London UK, pp. 316-329, Annual Learning for Dynamics and Control Conference 2022, Stanford, California, United States of America, 23/06/22. <https://proceedings.mlr.press/v168/jaques22a.html>

Vision-based System Identification and 3D keypoint discovery using dynamics constraints. / Jaques, Miguel; Asenov, Martin; Burke, Michael et al.
Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022. ed. / Roya Firoozi; Negar Mehr; Esen Yel; Rika Antonova; Jeannette Bohg; Mac Schwager; Mykel Kochenderfer. London UK: Proceedings of Machine Learning Research (PMLR), 2022. p. 316-329 (Proceedings of Machine Learning Research; Vol. 168).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Vision-based System Identification and 3D keypoint discovery using dynamics constraints

AU - Jaques, Miguel

AU - Asenov, Martin

AU - Burke, Michael

AU - Hospedales, Timothy

N1 - Conference code: 4th

PY - 2022

Y1 - 2022

N2 - This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach

AB - This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach

M3 - Conference Paper

T3 - Proceedings of Machine Learning Research

SP - 316

EP - 329

BT - Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022

A2 - Firoozi, Roya

A2 - Mehr, Negar

A2 - Yel, Esen

A2 - Antonova, Rika

A2 - Bohg, Jeannette

A2 - Schwager, Mac

A2 - Kochenderfer, Mykel

PB - Proceedings of Machine Learning Research (PMLR)

CY - London UK

T2 - Annual Learning for Dynamics and Control Conference 2022

Y2 - 23 June 2022 through 24 June 2022

ER -

Jaques M, Asenov M, Burke M, Hospedales T. Vision-based System Identification and 3D keypoint discovery using dynamics constraints. In Firoozi R, Mehr N, Yel E, Antonova R, Bohg J, Schwager M, Kochenderfer M, editors, Proceedings of The 4th Annual Learning for Dynamics and Control Conference 2022. London UK: Proceedings of Machine Learning Research (PMLR). 2022. p. 316-329. (Proceedings of Machine Learning Research).