GMFlow: learning optical flow via global matching

Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Dacheng Tao

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

72 Citations (Scopus)

Abstract

Learning-based optical flow estimation has been dominated with the pipeline of cost volume with convolutions for flow regression, which is inherently limited to local correlations and thus is hard to address the long-standing challenge of large displacements. To alleviate this, the state-of-the-art framework RAFT gradually improves its prediction quality by using a large number of iterative refinements, achieving remarkable performance but introducing linearly increasing inference time. To enable both high accuracy and efficiency, we completely revamp the dominant flow regression pipeline by reformulating optical flow as a global matching problem, which identifies the correspondences by directly comparing feature similarities. Specifically, we propose a GMFlow framework, which consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. We further introduce a refinement step that reuses GMFlow at higher feature resolution for residual flow prediction. Our new framework outperforms 31-refinements RAFT on the challenging Sintel benchmark, while using only one refinement and running faster, suggesting a new paradigm for accurate and efficient optical flow estimation. Code is available at https://github.com/haofeixu/gmflow.

Original languageEnglish
Title of host publicationProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
EditorsKristin Dana, Gang Hua, Stefan Roth, Dimitris Samaras, Richa Singh
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages8111-8120
Number of pages10
ISBN (Electronic)9781665469463
DOIs
Publication statusPublished - 2022
EventIEEE Conference on Computer Vision and Pattern Recognition 2022 - New Orleans, United States of America
Duration: 19 Jun 202224 Jun 2022
https://ieeexplore.ieee.org/xpl/conhome/9878378/proceeding (Proceedings)
https://cvpr2022.thecvf.com
https://cvpr2022.thecvf.com/ (Website)

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume2022-June
ISSN (Print)1063-6919

Conference

ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2022
Abbreviated titleCVPR 2022
Country/TerritoryUnited States of America
CityNew Orleans
Period19/06/2224/06/22
Internet address

Keywords

  • Low-level vision
  • Motion and tracking

Cite this