Models teaching models: improving model accuracy with slingshot learning

Lachlan O'Neill, Nandini Anantharama, Satya Borgohain, Simon D. Angus

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

One significant obstacle to the successful application of machine learning to real-world data is that of labeling: it is often prohibitively expensive to pay an ethical amount for the human labor required to label a dataset successfully. Human-in-the-loop techniques such as active learning can reduce the cost, but the required human time is still significant and many fixed costs remain. Another option is to employ pre-trained transformer models as labelers at scale, which can yield reasonable accuracy and significant cost savings. However, such models can still be expensive to use due to their high computational requirements, and the opaque nature of these models is not always suitable in applied social science and public use contexts. We propose a novel semi-supervised method, named Slingshot Learning, in which we iteratively and selectively augment a small human-labeled dataset with labels from a high-quality "teacher" model to slingshot the performance of a "student" model in a cost-efficient manner. This reduces the accuracy trade-off required to use these simpler algorithms without disrupting their benefits, such as lower compute requirements, better interpretability, and faster inference. We define and discuss the slingshot learning algorithm and demonstrate its effectiveness on several benchmark tasks, using ALBERT to teach a simple Naive Bayes binary classifier. We experimentally demonstrate that Slingshot learning effectively decreases the performance gap between the teacher and student models. We also analyze its performance in several scenarios and compare different variants of the algorithm.

Original languageEnglish
Title of host publicationEACL 2023
Subtitle of host publicationthe 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference
EditorsIsabelle Augenstein, Andreas Vlachos
Place of PublicationStroudsburg PA USA
PublisherAssociation for Computational Linguistics (ACL)
Pages3233-3247
Number of pages15
ISBN (Electronic)9781959429449
Publication statusPublished - 2023
EventEuropean Association of Computational Linguistics Conference 2023 - Dubrovnik, Croatia
Duration: 2 May 20236 May 2023
Conference number: 17th
https://2023.eacl.org/ (Website)
https://aclanthology.org/volumes/2023.eacl-main/ (Proceedings)

Conference

ConferenceEuropean Association of Computational Linguistics Conference 2023
Abbreviated titleEACL 2023
Country/TerritoryCroatia
CityDubrovnik
Period2/05/236/05/23
Internet address

Cite this