ActionNet

vision-based workflow action recognition from programming screencasts

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, Guoqiang Li

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Programming screencasts have two important applications in software engineering context: study developer behaviors, information needs and disseminate software engineering knowledge. Although programming screencasts are easy to produce, they are not easy to analyze or index due to the image nature of the data. Existing techniques extract only content from screencasts, but ignore workflow actions by which developers accomplish programming tasks. This significantly limits the effective use of programming screencasts in downstream applications. In this paper, we are the first to present a novel technique for recognizing workflow actions in programming screencasts. Our technique exploits image differencing and Convolutional Neural Network (CNN) to analyze the correspondence and change of consecutive frames, based on which nine classes of frequent developer actions can be recognized from programming screencasts. Using programming screencasts from Youtube, we evaluate different configurations of our CNN model and the performance of our technique for developer action recognition across developers, working environments and programming languages. Using screencasts of developers' real work, we demonstrate the usefulness of our technique in a practical application for actionaware extraction of key-code frames in developers' work.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019
EditorsTevfik Bultan, Jon Whittle
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages350-361
Number of pages12
ISBN (Electronic)9781728108698
ISBN (Print)9781728108704
DOIs
Publication statusPublished - 2019
EventInternational Conference on Software Engineering 2019 - Montreal, Canada
Duration: 25 May 201931 May 2019
Conference number: 41st
https://2019.icse-conferences.org/home

Conference

ConferenceInternational Conference on Software Engineering 2019
Abbreviated titleICSE 2019
CountryCanada
CityMontreal
Period25/05/1931/05/19
Internet address

Keywords

  • Action Recognition
  • Deep learning
  • Programming Screencast

Cite this

Zhao, D., Xing, Z., Chen, C., Xia, X., & Li, G. (2019). ActionNet: vision-based workflow action recognition from programming screencasts. In T. Bultan, & J. Whittle (Eds.), Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019 (pp. 350-361). [8811922] Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICSE.2019.00049
Zhao, Dehai ; Xing, Zhenchang ; Chen, Chunyang ; Xia, Xin ; Li, Guoqiang. / ActionNet : vision-based workflow action recognition from programming screencasts. Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019. editor / Tevfik Bultan ; Jon Whittle. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. pp. 350-361
@inproceedings{ff966b8e5cfd44e4a8f1f1316b416906,
title = "ActionNet: vision-based workflow action recognition from programming screencasts",
abstract = "Programming screencasts have two important applications in software engineering context: study developer behaviors, information needs and disseminate software engineering knowledge. Although programming screencasts are easy to produce, they are not easy to analyze or index due to the image nature of the data. Existing techniques extract only content from screencasts, but ignore workflow actions by which developers accomplish programming tasks. This significantly limits the effective use of programming screencasts in downstream applications. In this paper, we are the first to present a novel technique for recognizing workflow actions in programming screencasts. Our technique exploits image differencing and Convolutional Neural Network (CNN) to analyze the correspondence and change of consecutive frames, based on which nine classes of frequent developer actions can be recognized from programming screencasts. Using programming screencasts from Youtube, we evaluate different configurations of our CNN model and the performance of our technique for developer action recognition across developers, working environments and programming languages. Using screencasts of developers' real work, we demonstrate the usefulness of our technique in a practical application for actionaware extraction of key-code frames in developers' work.",
keywords = "Action Recognition, Deep learning, Programming Screencast",
author = "Dehai Zhao and Zhenchang Xing and Chunyang Chen and Xin Xia and Guoqiang Li",
year = "2019",
doi = "10.1109/ICSE.2019.00049",
language = "English",
isbn = "9781728108704",
pages = "350--361",
editor = "Bultan, {Tevfik } and Whittle, {Jon }",
booktitle = "Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
address = "United States of America",

}

Zhao, D, Xing, Z, Chen, C, Xia, X & Li, G 2019, ActionNet: vision-based workflow action recognition from programming screencasts. in T Bultan & J Whittle (eds), Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019., 8811922, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 350-361, International Conference on Software Engineering 2019, Montreal, Canada, 25/05/19. https://doi.org/10.1109/ICSE.2019.00049

ActionNet : vision-based workflow action recognition from programming screencasts. / Zhao, Dehai; Xing, Zhenchang; Chen, Chunyang; Xia, Xin; Li, Guoqiang.

Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019. ed. / Tevfik Bultan; Jon Whittle. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2019. p. 350-361 8811922.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - ActionNet

T2 - vision-based workflow action recognition from programming screencasts

AU - Zhao, Dehai

AU - Xing, Zhenchang

AU - Chen, Chunyang

AU - Xia, Xin

AU - Li, Guoqiang

PY - 2019

Y1 - 2019

N2 - Programming screencasts have two important applications in software engineering context: study developer behaviors, information needs and disseminate software engineering knowledge. Although programming screencasts are easy to produce, they are not easy to analyze or index due to the image nature of the data. Existing techniques extract only content from screencasts, but ignore workflow actions by which developers accomplish programming tasks. This significantly limits the effective use of programming screencasts in downstream applications. In this paper, we are the first to present a novel technique for recognizing workflow actions in programming screencasts. Our technique exploits image differencing and Convolutional Neural Network (CNN) to analyze the correspondence and change of consecutive frames, based on which nine classes of frequent developer actions can be recognized from programming screencasts. Using programming screencasts from Youtube, we evaluate different configurations of our CNN model and the performance of our technique for developer action recognition across developers, working environments and programming languages. Using screencasts of developers' real work, we demonstrate the usefulness of our technique in a practical application for actionaware extraction of key-code frames in developers' work.

AB - Programming screencasts have two important applications in software engineering context: study developer behaviors, information needs and disseminate software engineering knowledge. Although programming screencasts are easy to produce, they are not easy to analyze or index due to the image nature of the data. Existing techniques extract only content from screencasts, but ignore workflow actions by which developers accomplish programming tasks. This significantly limits the effective use of programming screencasts in downstream applications. In this paper, we are the first to present a novel technique for recognizing workflow actions in programming screencasts. Our technique exploits image differencing and Convolutional Neural Network (CNN) to analyze the correspondence and change of consecutive frames, based on which nine classes of frequent developer actions can be recognized from programming screencasts. Using programming screencasts from Youtube, we evaluate different configurations of our CNN model and the performance of our technique for developer action recognition across developers, working environments and programming languages. Using screencasts of developers' real work, we demonstrate the usefulness of our technique in a practical application for actionaware extraction of key-code frames in developers' work.

KW - Action Recognition

KW - Deep learning

KW - Programming Screencast

UR - http://www.scopus.com/inward/record.url?scp=85072301361&partnerID=8YFLogxK

U2 - 10.1109/ICSE.2019.00049

DO - 10.1109/ICSE.2019.00049

M3 - Conference Paper

SN - 9781728108704

SP - 350

EP - 361

BT - Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019

A2 - Bultan, Tevfik

A2 - Whittle, Jon

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

ER -

Zhao D, Xing Z, Chen C, Xia X, Li G. ActionNet: vision-based workflow action recognition from programming screencasts. In Bultan T, Whittle J, editors, Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2019. p. 350-361. 8811922 https://doi.org/10.1109/ICSE.2019.00049