ActionNet: vision-based workflow action recognition from programming screencasts

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, Guoqiang Li

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

17 Citations (Scopus)


Programming screencasts have two important applications in software engineering context: study developer behaviors, information needs and disseminate software engineering knowledge. Although programming screencasts are easy to produce, they are not easy to analyze or index due to the image nature of the data. Existing techniques extract only content from screencasts, but ignore workflow actions by which developers accomplish programming tasks. This significantly limits the effective use of programming screencasts in downstream applications. In this paper, we are the first to present a novel technique for recognizing workflow actions in programming screencasts. Our technique exploits image differencing and Convolutional Neural Network (CNN) to analyze the correspondence and change of consecutive frames, based on which nine classes of frequent developer actions can be recognized from programming screencasts. Using programming screencasts from Youtube, we evaluate different configurations of our CNN model and the performance of our technique for developer action recognition across developers, working environments and programming languages. Using screencasts of developers' real work, we demonstrate the usefulness of our technique in a practical application for actionaware extraction of key-code frames in developers' work.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE 2019
EditorsTevfik Bultan, Jon Whittle
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages12
ISBN (Electronic)9781728108698
ISBN (Print)9781728108704
Publication statusPublished - 2019
EventInternational Conference on Software Engineering 2019 - Fairmont The Queen Elizabeth Hotel, Montreal, Canada
Duration: 25 May 201931 May 2019
Conference number: 41st (Proceedings)


ConferenceInternational Conference on Software Engineering 2019
Abbreviated titleICSE 2019
Internet address


  • Action Recognition
  • Deep learning
  • Programming Screencast

Cite this