Towards accurate knowledge transfer between transformer-based models for code summarization

Chaochen Shi, Yong Xiang, Jiangshan Yu, Longxiang Gao

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

1 Citation (Scopus)

Abstract

Automatic code summarization generates high-level natural language descriptions of code snippets, which can benefit software maintenance and code comprehension. Recently, Transformer-based models achieved state-of-the-art performance on code summarization tasks. However, there are data gaps in neural model training for some programming languages. To fill this gap, we propose a novel transfer learning approach to accurately transfer knowledge between Transformer-based models. We train a discriminator to identify which heads of the multi-head attention module should be transferred. On this basis, we define a transfer strategy of parameter matrices. We evaluated the proposed transfer learning approach on four state-of-the-art Transformer-based code summarization models. Experimental results show that models with transferred knowledge outperform original models up to 10.70% in BLEU, 5.36% in ROUGE-L, and 4.34% in METEOR.

Original languageEnglish
Title of host publicationSEKE 2022 - Proceedings of the 34th International Conference on Software Engineering and Knowledge Engineering
EditorsRong Peng, Carlos Eduardo Pantoja, Pankaj Kamthan
Place of PublicationSkokie IL USA
PublisherKnowledge Systems Institute
Pages91-94
Number of pages4
ISBN (Electronic)1891706543, 9781891706547
DOIs
Publication statusPublished - 2022
EventInternational Conference on Software Engineering and Knowledge Engineering 2022 - Pittsburgh, United States of America
Duration: 1 Jul 202210 Jul 2022
Conference number: 34th
http://ksiresearch.org/seke/seke22.html (Website)

Publication series

NameProceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE
PublisherKnowledge Systems Institute Graduate School
ISSN (Print)2325-9000
ISSN (Electronic)2325-9086

Conference

ConferenceInternational Conference on Software Engineering and Knowledge Engineering 2022
Abbreviated titleSEKE 2022
Country/TerritoryUnited States of America
CityPittsburgh
Period1/07/2210/07/22
Internet address

Keywords

  • Code Summarization
  • Transfer Learning

Cite this