Dueling deep Q-network for unsupervised inter-frame eye movement correction in optical coherence tomography volumes

Yasmeen George, Suman Sedai, Bhavna J. Antony, Hiroshi Ishikawa, Gadi Wollstein, Joel S. Schuman, Rahil Garnavi

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


In optical coherence tomography (OCT) volumes of retina, the sequential acquisition of the individual slices makes this modality prone to motion artifacts, misalignments between adjacent slices being the most noticeable. Any distortion in OCT volumes can bias structural analysis and influence the outcome of longitudinal studies. The presence of speckle noise characteristic of this imaging modality leads to inaccuracies when traditional registration techniques are employed. Also, the lack of a well-defined ground truth makes supervised deep-learning techniques ill-posed to tackle the problem. In this paper, we tackle these issues by using deep reinforcement learning to correct inter-frame movements in an unsupervised manner. Specifically, we use dueling deep Q-network to train an artificial agent to find the optimal policy, i.e. a sequence of actions, that best improves the alignment by maximizing the sum of reward signals. Instead of relying on the ground-truth of transformation parameters to guide the rewarding system, for the first time, we use a combination of intensity based image similarity metrics. Further, to avoid the agent bias towards speckle noise, we ensure the agent can see retinal layers as part of the interacting environment. For quantitative evaluation, we simulate the eye movement artifacts by applying 2D rigid transformations on individual B-scans. The proposed model achieves an average of 0.985 and 0.914 for normalized mutual information and correlation coefficient, respectively. We also compare our model with elastix intensity based medical image registration approach, where significant improvement is achieved by our model for both noisy and denoised volumes.

Original languageEnglish
Title of host publication2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI 2021)
EditorsTom Vercauteren, Gustavo Rohde
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages5
ISBN (Electronic)9781665412469, 9781665412452
ISBN (Print)9781665429474
Publication statusPublished - 2021
Externally publishedYes
EventIEEE International Symposium on Biomedical Imaging (ISBI) 2021 - Online, Nice, France
Duration: 13 Apr 202116 Apr 2021
Conference number: 18th
https://ieeexplore.ieee.org/xpl/conhome/9433749/proceeding (Proceedings)
https://biomedicalimaging.org/2021/ (Website)

Publication series

NameProceedings - International Symposium on Biomedical Imaging
PublisherInstitute of Electrical and Electronics Engineers, Inc.
ISSN (Print)1945-7928
ISSN (Electronic)1945-8452


ConferenceIEEE International Symposium on Biomedical Imaging (ISBI) 2021
Abbreviated titleISBI 2021
Internet address


  • Artificial agents
  • Dueling deep Q-network
  • Motion correction
  • Optical coherence tomography
  • Reinforcement learning

Cite this