Robust early-learning: hindering the memorization of noisy labels

Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, Yi Chang

Research output: Chapter in Book/Report/Conference proceedingConference PaperOther

84 Citations (Scopus)

Abstract

The memorization effects of deep networks show that they will first memorize training data with clean labels and then those with noisy labels. The early stopping method therefore can be exploited for learning with noisy labels. However, the side effect brought by noisy labels will influence the memorization of clean labels before early stopping. In this paper, motivated by the lottery ticket hypothesis which shows that only partial parameters are important for generalization, we find that only partial parameters are important for fitting clean labels and generalize well, which we term as critical parameters; while the other parameters tend to fit noisy labels and cannot generalize well, which we term as non-critical parameters. Based on this, we propose robust early-learning to reduce the side effect of noisy labels before early stopping and thus enhance the memorization of clean labels. Specifically, in each iteration, we divide all parameters into the critical and non-critical ones, and then perform different update rules for different types of parameters. Extensive experiments on benchmark-simulated and real-world label-noise datasets demonstrate the superiority of the proposed method over the state-of-the-art label-noise learning methods.

Original languageEnglish
Title of host publicationProceedings of 2021 International Conference on Learning Representations
EditorsAlice Oh, Naila Murray, Ivan Titov
Place of PublicationUSA
PublisherInternational Conference on Learning Representations (ICLR)
Pages1-15
Number of pages15
Publication statusPublished - 2021
EventInternational Conference on Learning Representations 2021 - Virtual/Online, Vienna, Austria
Duration: 3 May 20217 May 2022
https://iclr.cc/Conferences/2021

Conference

ConferenceInternational Conference on Learning Representations 2021
Abbreviated titleICLR 2021
Country/TerritoryAustria
CityVienna
Period3/05/217/05/22
Internet address

Cite this