A fine-grained data set and analysis of tangling in bug fixing commits

Steffen Herbold; Alexander Trautsch; Benjamin Ledel; Alireza Aghamohammadi; Taher A. Ghaleb; Kuljit Kaur Chahal; Tim Bossenmaier; Bhaveet Nagaria; Philip Makedonski; Matin Nili Ahmadabadi; Kristof Szabados; Helge Spieker; Matej Madeja; Nathaniel Hoy; Valentina Lenarduzzi; Shangwen Wang; Gema Rodríguez-Pérez; Ricardo Colomo-Palacios; Roberto Verdecchia; Paramvir Singh; Yihao Qin; Debasish Chakroborti; Willard Davis; Vijay Walunj; Hongjun Wu; Diego Marcilio; Omar Alam; Abdullah Aldaeej; Idan Amit; Burak Turhan; Simon Eismann; Anna Katharina Wickert; Ivano Malavolta; Matúš Sulír; Fatemeh Fard; Austin Z. Henley; Stratos Kourtzanidis; Eray Tuzun; Christoph Treude; Simin Maleki Shamasbi; Ivan Pashchenko; Marvin Wyrich; James Davis; Alexander Serebrenik; Ella Albrecht; Ethem Utku Aktas; Daniel Strüber; Johannes Erbel

doi:10.1007/s10664-021-10083-5

A fine-grained data set and analysis of tangling in bug fixing commits

Steffen Herbold, Alexander Trautsch, Benjamin Ledel, Alireza Aghamohammadi, Taher A. Ghaleb, Kuljit Kaur Chahal, Tim Bossenmaier, Bhaveet Nagaria, Philip Makedonski, Matin Nili Ahmadabadi, Kristof Szabados, Helge Spieker, Matej Madeja, Nathaniel Hoy, Valentina Lenarduzzi, Shangwen Wang, Gema Rodríguez-Pérez, Ricardo Colomo-Palacios, Roberto Verdecchia, Paramvir SinghYihao Qin, Debasish Chakroborti, Willard Davis, Vijay Walunj, Hongjun Wu, Diego Marcilio, Omar Alam, Abdullah Aldaeej, Idan Amit, Burak Turhan, Simon Eismann, Anna Katharina Wickert, Ivano Malavolta, Matúš Sulír, Fatemeh Fard, Austin Z. Henley, Stratos Kourtzanidis, Eray Tuzun, Christoph Treude, Simin Maleki Shamasbi, Ivan Pashchenko, Marvin Wyrich, James Davis, Alexander Serebrenik, Ella Albrecht, Ethem Utku Aktas, Daniel Strüber, Johannes Erbel

Department of Software Systems & Cybersecurity

Research output: Contribution to journal › Article › Research › peer-review

18 Citations (Scopus)

Abstract

Context: Tangled commits are changes to software that address multiple concerns at once. For researchers interested in bugs, tangled commits mean that they actually study not only bugs, but also other concerns irrelevant for the study of bugs. Objective: We want to improve our understanding of the prevalence of tangling and the types of changes that are tangled within bug fixing commits. Methods: We use a crowd sourcing approach for manual labeling to validate which changes contribute to bug fixes for each line in bug fixing commits. Each line is labeled by four participants. If at least three participants agree on the same label, we have consensus. Results: We estimate that between 17% and 32% of all changes in bug fixing commits modify the source code to fix the underlying problem. However, when we only consider changes to the production code files this ratio increases to 66% to 87%. We find that about 11% of lines are hard to label leading to active disagreements between participants. Due to confirmed tangling and the uncertainty in our data, we estimate that 3% to 47% of data is noisy without manual untangling, depending on the use case. Conclusion: Tangled commits have a high prevalence in bug fixes and can lead to a large amount of noise in the data. Prior research indicates that this noise may alter results. As researchers, we should be skeptics and assume that unvalidated data is likely very noisy, until proven otherwise.

Original language	English
Article number	125
Number of pages	49
Journal	Empirical Software Engineering
Volume	27
Issue number	6
DOIs	https://doi.org/10.1007/s10664-021-10083-5
Publication status	Published - Nov 2022

Keywords

Bug fix
Manual validation
Registered report
Research turk
Tangled changes
Tangled commits

Access to Document

10.1007/s10664-021-10083-5

393650158-oaFinal published version, 2.12 MB

Cite this

Herbold, S., Trautsch, A., Ledel, B., Aghamohammadi, A., Ghaleb, T. A., Chahal, K. K., Bossenmaier, T., Nagaria, B., Makedonski, P., Ahmadabadi, M. N., Szabados, K., Spieker, H., Madeja, M., Hoy, N., Lenarduzzi, V., Wang, S., Rodríguez-Pérez, G., Colomo-Palacios, R., Verdecchia, R., ... Erbel, J. (2022). A fine-grained data set and analysis of tangling in bug fixing commits. Empirical Software Engineering, 27(6), Article 125. https://doi.org/10.1007/s10664-021-10083-5

@article{13faa64642cb4645b68bb87433faa96e,

title = "A fine-grained data set and analysis of tangling in bug fixing commits",

abstract = "Context: Tangled commits are changes to software that address multiple concerns at once. For researchers interested in bugs, tangled commits mean that they actually study not only bugs, but also other concerns irrelevant for the study of bugs. Objective: We want to improve our understanding of the prevalence of tangling and the types of changes that are tangled within bug fixing commits. Methods: We use a crowd sourcing approach for manual labeling to validate which changes contribute to bug fixes for each line in bug fixing commits. Each line is labeled by four participants. If at least three participants agree on the same label, we have consensus. Results: We estimate that between 17% and 32% of all changes in bug fixing commits modify the source code to fix the underlying problem. However, when we only consider changes to the production code files this ratio increases to 66% to 87%. We find that about 11% of lines are hard to label leading to active disagreements between participants. Due to confirmed tangling and the uncertainty in our data, we estimate that 3% to 47% of data is noisy without manual untangling, depending on the use case. Conclusion: Tangled commits have a high prevalence in bug fixes and can lead to a large amount of noise in the data. Prior research indicates that this noise may alter results. As researchers, we should be skeptics and assume that unvalidated data is likely very noisy, until proven otherwise.",

keywords = "Bug fix, Manual validation, Registered report, Research turk, Tangled changes, Tangled commits",

author = "Steffen Herbold and Alexander Trautsch and Benjamin Ledel and Alireza Aghamohammadi and Ghaleb, {Taher A.} and Chahal, {Kuljit Kaur} and Tim Bossenmaier and Bhaveet Nagaria and Philip Makedonski and Ahmadabadi, {Matin Nili} and Kristof Szabados and Helge Spieker and Matej Madeja and Nathaniel Hoy and Valentina Lenarduzzi and Shangwen Wang and Gema Rodr{\'i}guez-P{\'e}rez and Ricardo Colomo-Palacios and Roberto Verdecchia and Paramvir Singh and Yihao Qin and Debasish Chakroborti and Willard Davis and Vijay Walunj and Hongjun Wu and Diego Marcilio and Omar Alam and Abdullah Aldaeej and Idan Amit and Burak Turhan and Simon Eismann and Wickert, {Anna Katharina} and Ivano Malavolta and Mat{\'u}{\v s} Sul{\'i}r and Fatemeh Fard and Henley, {Austin Z.} and Stratos Kourtzanidis and Eray Tuzun and Christoph Treude and Shamasbi, {Simin Maleki} and Ivan Pashchenko and Marvin Wyrich and James Davis and Alexander Serebrenik and Ella Albrecht and Aktas, {Ethem Utku} and Daniel Str{\"u}ber and Johannes Erbel",

note = "Funding Information: Alexander Trautsch and Benjamin Ledel and the development of the infrastructure required for this research project were funded by DFG Grant 402774445. Ivan Pashchenko was partially funded by the H2020 AssureMOSS project (Grant No. 952647). We thank the cloud team of the GWDG ( https://gwdg.de ) that helped us migrate the VisualSHARK to a stronger machine within a day, to meet the demands of this project once many participants started to label at the same time. Publisher Copyright: {\textcopyright} 2022, The Author(s).",

year = "2022",

month = nov,

doi = "10.1007/s10664-021-10083-5",

language = "English",

volume = "27",

journal = "Empirical Software Engineering",

issn = "1382-3256",

publisher = "Springer",

number = "6",

}

Herbold, S, Trautsch, A, Ledel, B, Aghamohammadi, A, Ghaleb, TA, Chahal, KK, Bossenmaier, T, Nagaria, B, Makedonski, P, Ahmadabadi, MN, Szabados, K, Spieker, H, Madeja, M, Hoy, N, Lenarduzzi, V, Wang, S, Rodríguez-Pérez, G, Colomo-Palacios, R, Verdecchia, R, Singh, P, Qin, Y, Chakroborti, D, Davis, W, Walunj, V, Wu, H, Marcilio, D, Alam, O, Aldaeej, A, Amit, I, Turhan, B, Eismann, S, Wickert, AK, Malavolta, I, Sulír, M, Fard, F, Henley, AZ, Kourtzanidis, S, Tuzun, E, Treude, C, Shamasbi, SM, Pashchenko, I, Wyrich, M, Davis, J, Serebrenik, A, Albrecht, E, Aktas, EU, Strüber, D & Erbel, J 2022, 'A fine-grained data set and analysis of tangling in bug fixing commits', Empirical Software Engineering, vol. 27, no. 6, 125. https://doi.org/10.1007/s10664-021-10083-5

TY - JOUR

T1 - A fine-grained data set and analysis of tangling in bug fixing commits

AU - Herbold, Steffen

AU - Trautsch, Alexander

AU - Ledel, Benjamin

AU - Aghamohammadi, Alireza

AU - Ghaleb, Taher A.

AU - Chahal, Kuljit Kaur

AU - Bossenmaier, Tim

AU - Nagaria, Bhaveet

AU - Makedonski, Philip

AU - Ahmadabadi, Matin Nili

AU - Szabados, Kristof

AU - Spieker, Helge

AU - Madeja, Matej

AU - Hoy, Nathaniel

AU - Lenarduzzi, Valentina

AU - Wang, Shangwen

AU - Rodríguez-Pérez, Gema

AU - Colomo-Palacios, Ricardo

AU - Verdecchia, Roberto

AU - Singh, Paramvir

AU - Qin, Yihao

AU - Chakroborti, Debasish

AU - Davis, Willard

AU - Walunj, Vijay

AU - Wu, Hongjun

AU - Marcilio, Diego

AU - Alam, Omar

AU - Aldaeej, Abdullah

AU - Amit, Idan

AU - Turhan, Burak

AU - Eismann, Simon

AU - Wickert, Anna Katharina

AU - Malavolta, Ivano

AU - Sulír, Matúš

AU - Fard, Fatemeh

AU - Henley, Austin Z.

AU - Kourtzanidis, Stratos

AU - Tuzun, Eray

AU - Treude, Christoph

AU - Shamasbi, Simin Maleki

AU - Pashchenko, Ivan

AU - Wyrich, Marvin

AU - Davis, James

AU - Serebrenik, Alexander

AU - Albrecht, Ella

AU - Aktas, Ethem Utku

AU - Strüber, Daniel

AU - Erbel, Johannes

N1 - Funding Information: Alexander Trautsch and Benjamin Ledel and the development of the infrastructure required for this research project were funded by DFG Grant 402774445. Ivan Pashchenko was partially funded by the H2020 AssureMOSS project (Grant No. 952647). We thank the cloud team of the GWDG ( https://gwdg.de ) that helped us migrate the VisualSHARK to a stronger machine within a day, to meet the demands of this project once many participants started to label at the same time. Publisher Copyright: © 2022, The Author(s).

PY - 2022/11

Y1 - 2022/11

N2 - Context: Tangled commits are changes to software that address multiple concerns at once. For researchers interested in bugs, tangled commits mean that they actually study not only bugs, but also other concerns irrelevant for the study of bugs. Objective: We want to improve our understanding of the prevalence of tangling and the types of changes that are tangled within bug fixing commits. Methods: We use a crowd sourcing approach for manual labeling to validate which changes contribute to bug fixes for each line in bug fixing commits. Each line is labeled by four participants. If at least three participants agree on the same label, we have consensus. Results: We estimate that between 17% and 32% of all changes in bug fixing commits modify the source code to fix the underlying problem. However, when we only consider changes to the production code files this ratio increases to 66% to 87%. We find that about 11% of lines are hard to label leading to active disagreements between participants. Due to confirmed tangling and the uncertainty in our data, we estimate that 3% to 47% of data is noisy without manual untangling, depending on the use case. Conclusion: Tangled commits have a high prevalence in bug fixes and can lead to a large amount of noise in the data. Prior research indicates that this noise may alter results. As researchers, we should be skeptics and assume that unvalidated data is likely very noisy, until proven otherwise.

AB - Context: Tangled commits are changes to software that address multiple concerns at once. For researchers interested in bugs, tangled commits mean that they actually study not only bugs, but also other concerns irrelevant for the study of bugs. Objective: We want to improve our understanding of the prevalence of tangling and the types of changes that are tangled within bug fixing commits. Methods: We use a crowd sourcing approach for manual labeling to validate which changes contribute to bug fixes for each line in bug fixing commits. Each line is labeled by four participants. If at least three participants agree on the same label, we have consensus. Results: We estimate that between 17% and 32% of all changes in bug fixing commits modify the source code to fix the underlying problem. However, when we only consider changes to the production code files this ratio increases to 66% to 87%. We find that about 11% of lines are hard to label leading to active disagreements between participants. Due to confirmed tangling and the uncertainty in our data, we estimate that 3% to 47% of data is noisy without manual untangling, depending on the use case. Conclusion: Tangled commits have a high prevalence in bug fixes and can lead to a large amount of noise in the data. Prior research indicates that this noise may alter results. As researchers, we should be skeptics and assume that unvalidated data is likely very noisy, until proven otherwise.

KW - Bug fix

KW - Manual validation

KW - Registered report

KW - Research turk

KW - Tangled changes

KW - Tangled commits

UR - http://www.scopus.com/inward/record.url?scp=85133342067&partnerID=8YFLogxK

U2 - 10.1007/s10664-021-10083-5

DO - 10.1007/s10664-021-10083-5

M3 - Article

AN - SCOPUS:85133342067

SN - 1382-3256

VL - 27

JO - Empirical Software Engineering

JF - Empirical Software Engineering

IS - 6

M1 - 125

ER -

A fine-grained data set and analysis of tangling in bug fixing commits

Abstract

Keywords

Access to Document

Other files and links

Cite this