Rewire-then-probe: a contrastive recipe for probing biomedical knowledge of pre-trained language models

Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, Nigel Collier

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


Knowledge probing is crucial for understanding the knowledge transfer mechanism behind the pre-trained language models (PLMs). Despite the growing progress of probing knowledge for PLMs in the general domain, specialised areas such as the biomedical domain are vastly under-explored. To facilitate this, we release a well-curated biomedical knowledge probing benchmark, MedLAMA, constructed based on the Unified Medical Language System (UMLS) Metathesaurus. We test a wide spectrum of state-of-the-art PLMs and probing approaches on our benchmark, reaching at most 3% of acc@10. While highlighting various sources of domain-specific challenges that amount to this underwhelming performance, we illustrate that the underlying PLMs have a higher potential for probing tasks. To achieve this, we propose Contrastive-Probe, a novel self-supervised contrastive probing approach, that adjusts the underlying PLMs without using any probing data. While Contrastive-Probe pushes the acc@10 to 28%, the performance gap still remains notable. Our human expert evaluation suggests that the probing performance of our Contrastive-Probe is still under-estimated as UMLS still does not include the full spectrum of factual knowledge. We hope MedLAMA and Contrastive-Probe facilitate further developments of more suited probing techniques for this domain. Our code and dataset are publicly available at
Original languageEnglish
Title of host publicationACL 2022, The 60th Annual Meeting of the Association for Computational Linguistics
Subtitle of host publicationProceedings of the Conference, Vol. 1 (Long Papers)
EditorsDanilo Croce, Ryan Cotterell, Jordan Zhang
Place of PublicationStroudsburg PA USA
PublisherAssociation for Computational Linguistics (ACL)
Number of pages13
ISBN (Electronic)9781955917216
Publication statusPublished - 2022
EventAnnual Meeting of the Association for Computational Linguistics 2022 - Dublin, Ireland
Duration: 22 May 202227 May 2022
Conference number: 60th (Proceedings - Short) (Proceedings - Long) (Website)


ConferenceAnnual Meeting of the Association for Computational Linguistics 2022
Abbreviated titleACL 2022
Internet address

Cite this