Evaluating representation learning of code changes for predicting patch correctness in program repair

Haoye Tian, Kui Liu, Abdoul Kader Kabore, Anil Koyuncu, Li Li, Jacques Klein, Tegawende F. Bissyande

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

7 Citations (Scopus)


A large body of the literature of automated program repair develops approaches where patches are generated to be validated against an oracle (e.g., a test suite). Because such an oracle can be imperfect, the generated patches, although validated by the oracle, may actually be incorrect. While the state of the art explore research directions that require dynamic information or that rely on manually-crafted heuristics, we study the benefit of learning code representations in order to learn deep features that may encode the properties of patch correctness. Our empirical work mainly investigates different representation learning approaches for code changes to derive embeddings that are amenable to similarity computations. We report on findings based on embeddings produced by pre-trained and re-trained neural networks. Experimental results demonstrate the potential of embeddings to empower learning algorithms in reasoning about patch correctness: a machine learning predictor with BERT transformer-based embeddings associated with logistic regression yielded an AUC value of about 0.8 in the prediction of patch correctness on a deduplicated dataset of 1000 labeled patches. Our investigations show that learned representations can lead to reasonable performance when comparing against the state-of-the-art, PATCH-SIM, which relies on dynamic information. These representations may further be complementary to features that were carefully (manually) engineered in the literature.

Original languageEnglish
Title of host publicationProceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020
EditorsClaire Le Goues, David Lo
Place of PublicationNew York NY USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages12
ISBN (Electronic)9781450367684
Publication statusPublished - 2020
EventAutomated Software Engineering Conference 2020 - Virtual, Melbourne, Australia
Duration: 21 Sep 202025 Sep 2020
Conference number: 35th
https://dl.acm.org/doi/proceedings/10.1145/3324884 (Proceedings)
https://conf.researchr.org/home/ase-2020 (Website)
https://dl.acm.org/doi/proceedings/10.1145/3417113 (Proceedings)


ConferenceAutomated Software Engineering Conference 2020
Abbreviated titleASE 2020
Internet address


  • Distributed Representation Learning
  • Embeddings
  • Machine learning
  • Patch Correctness
  • Program Repair

Cite this