Read it, don't watch it: Captioning bug recordings automatically

Sidong Feng; Mulong Xie; Yinxing Xue; Chunyang Chen

doi:10.1109/ICSE48619.2023.00197

Read it, don't watch it: Captioning bug recordings automatically

Sidong Feng
, Mulong Xie
, Yinxing Xue
, Chunyang Chen

Department of Software Systems & Cybersecurity

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

11 Citations (Scopus)

Abstract

Screen recordings of mobile applications are easy to capture and include a wealth of information, making them a popular mechanism for users to inform developers of the problems encountered in the bug reports. However, watching the bug recordings and efficiently understanding the semantics of user actions can be time-consuming and tedious for developers. Inspired by the conception of the video subtitle in movie industry, we present a lightweight approach CAPdroid to caption bug recordings automatically. CAPdroid is a purely image-based and non-intrusive approach by using image processing and convolutional deep learning models to segment bug recordings, infer user action attributes, and generate subtitle descriptions. The automated experiments demonstrate the good performance of CAPdroid in inferring user actions from the recordings, and a user study confirms the usefulness of our generated step descriptions in assisting developers with bug replay.

Original language	English
Title of host publication	Proceedings - 2023 IEEE/ACM 45th International Conference on Software Engineering, ICSE 2023
Editors	Lori Pollock, Massimiliano Di Penta
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	2349-2361
Number of pages	13
ISBN (Electronic)	9781665457019
ISBN (Print)	9781665457026
DOIs	https://doi.org/10.1109/ICSE48619.2023.00197
Publication status	Published - 2023
Event	International Conference on Software Engineering 2023 - Melbourne, Australia Duration: 14 May 2023 → 20 May 2023 Conference number: 45th https://ieeexplore.ieee.org/xpl/conhome/10172484/proceeding (Proceedings) https://conf.researchr.org/home/icse-2023 (Website) https://ieeexplore.ieee.org/xpl/conhome/10449397/proceeding (ICSE-FoSE 2023 Proceedings) https://ieeexplore.ieee.org/xpl/conhome/10172483/proceeding (ICSE-SEET 2023 Proceedings) https://conf.researchr.org/track/icse-2023/icse-2023-NIER? (ICSE-NIER 2023 Proceedings)

Conference

Conference	International Conference on Software Engineering 2023
Abbreviated title	ICSE 2023
Country/Territory	Australia
City	Melbourne
Period	14/05/23 → 20/05/23
Internet address	https://ieeexplore.ieee.org/xpl/conhome/10172484/proceeding (Proceedings) https://conf.researchr.org/home/icse-2023 (Website) https://ieeexplore.ieee.org/xpl/conhome/10449397/proceeding (ICSE-FoSE 2023 Proceedings) https://ieeexplore.ieee.org/xpl/conhome/10172483/proceeding (ICSE-SEET 2023 Proceedings) https://conf.researchr.org/track/icse-2023/icse-2023-NIER? (ICSE-NIER 2023 Proceedings)

Keywords

android app
bug recording
video captioning

Access to Document

10.1109/ICSE48619.2023.00197

Cite this

@inproceedings{7c0aa56644654eec88509be56eb457cf,

title = "Read it, don't watch it: Captioning bug recordings automatically",

abstract = "Screen recordings of mobile applications are easy to capture and include a wealth of information, making them a popular mechanism for users to inform developers of the problems encountered in the bug reports. However, watching the bug recordings and efficiently understanding the semantics of user actions can be time-consuming and tedious for developers. Inspired by the conception of the video subtitle in movie industry, we present a lightweight approach CAPdroid to caption bug recordings automatically. CAPdroid is a purely image-based and non-intrusive approach by using image processing and convolutional deep learning models to segment bug recordings, infer user action attributes, and generate subtitle descriptions. The automated experiments demonstrate the good performance of CAPdroid in inferring user actions from the recordings, and a user study confirms the usefulness of our generated step descriptions in assisting developers with bug replay.",

keywords = "android app, bug recording, video captioning",

author = "Sidong Feng and Mulong Xie and Yinxing Xue and Chunyang Chen",

note = "Funding Information: This work is partially supported by the Monash FIT RSP fund. Dr Yinxing Xue is supported by the National Nature Science Foundation, China under Grant No. 61972373, and the Basic Research Program of Jiangsu Province China under Grant No. BK20201192. We thank the inspirational discussion about this work with Dr Liming Nie who is supported by grants from the National Natural Science Foundation of China under Grant No. 61972359. Publisher Copyright: {\textcopyright} 2023 IEEE.; International Conference on Software Engineering 2023, ICSE 2023 ; Conference date: 14-05-2023 Through 20-05-2023",

year = "2023",

doi = "10.1109/ICSE48619.2023.00197",

language = "English",

isbn = "9781665457026",

pages = "2349--2361",

editor = "Lori Pollock and \{Di Penta\}, Massimiliano",

booktitle = "Proceedings - 2023 IEEE/ACM 45th International Conference on Software Engineering, ICSE 2023",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

address = "United States of America",

url = "https://ieeexplore.ieee.org/xpl/conhome/10172484/proceeding, https://conf.researchr.org/home/icse-2023, https://ieeexplore.ieee.org/xpl/conhome/10449397/proceeding, https://ieeexplore.ieee.org/xpl/conhome/10172483/proceeding, https://conf.researchr.org/track/icse-2023/icse-2023-NIER?",

}

Feng, S, Xie, M, Xue, Y & Chen, C 2023, Read it, don't watch it: Captioning bug recordings automatically. in L Pollock & M Di Penta (eds), Proceedings - 2023 IEEE/ACM 45th International Conference on Software Engineering, ICSE 2023. IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 2349-2361, International Conference on Software Engineering 2023, Melbourne, Victoria, Australia, 14/05/23. https://doi.org/10.1109/ICSE48619.2023.00197

Read it, don't watch it: Captioning bug recordings automatically. / Feng, Sidong; Xie, Mulong; Xue, Yinxing et al.
Proceedings - 2023 IEEE/ACM 45th International Conference on Software Engineering, ICSE 2023. ed. / Lori Pollock; Massimiliano Di Penta. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2023. p. 2349-2361.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Read it, don't watch it

T2 - International Conference on Software Engineering 2023

AU - Feng, Sidong

AU - Xie, Mulong

AU - Xue, Yinxing

AU - Chen, Chunyang

N1 - Conference code: 45th

PY - 2023

Y1 - 2023

N2 - Screen recordings of mobile applications are easy to capture and include a wealth of information, making them a popular mechanism for users to inform developers of the problems encountered in the bug reports. However, watching the bug recordings and efficiently understanding the semantics of user actions can be time-consuming and tedious for developers. Inspired by the conception of the video subtitle in movie industry, we present a lightweight approach CAPdroid to caption bug recordings automatically. CAPdroid is a purely image-based and non-intrusive approach by using image processing and convolutional deep learning models to segment bug recordings, infer user action attributes, and generate subtitle descriptions. The automated experiments demonstrate the good performance of CAPdroid in inferring user actions from the recordings, and a user study confirms the usefulness of our generated step descriptions in assisting developers with bug replay.

AB - Screen recordings of mobile applications are easy to capture and include a wealth of information, making them a popular mechanism for users to inform developers of the problems encountered in the bug reports. However, watching the bug recordings and efficiently understanding the semantics of user actions can be time-consuming and tedious for developers. Inspired by the conception of the video subtitle in movie industry, we present a lightweight approach CAPdroid to caption bug recordings automatically. CAPdroid is a purely image-based and non-intrusive approach by using image processing and convolutional deep learning models to segment bug recordings, infer user action attributes, and generate subtitle descriptions. The automated experiments demonstrate the good performance of CAPdroid in inferring user actions from the recordings, and a user study confirms the usefulness of our generated step descriptions in assisting developers with bug replay.

KW - android app

KW - bug recording

KW - video captioning

UR - https://www.scopus.com/pages/publications/85168849749

U2 - 10.1109/ICSE48619.2023.00197

DO - 10.1109/ICSE48619.2023.00197

M3 - Conference Paper

AN - SCOPUS:85168849749

SN - 9781665457026

SP - 2349

EP - 2361

BT - Proceedings - 2023 IEEE/ACM 45th International Conference on Software Engineering, ICSE 2023

A2 - Pollock, Lori

A2 - Di Penta, Massimiliano

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

Y2 - 14 May 2023 through 20 May 2023

ER -

Read it, don't watch it: Captioning bug recordings automatically

Abstract

Conference

Keywords

Access to Document

Other files and links

Cite this