On the effectiveness of query weighting for adapting rank learners to new unlabelled collections

Pengfei Li, Mark Sanderson, Mark Carman, Falk Scholer

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    7 Citations (Scopus)

    Abstract

    Query-level instance weighting is a technique for unsuper-vised transfer ranking, which aims to train a ranker on a source collection so that it also performs effectively on a target collection, even if no judgement information exists for the latter. Past work has shown that this approach can be used to significantly improve effectiveness; in this work, the approach is re-examined on a wide set of publicly available L2R test collections with more advanced learning to rank algorithms. Different query-level weighting strategies are examined against two transfer ranking frameworks: AdaRank and a new weighted LambdaMART algorithm. Our experimental results show that the effectiveness of different weighting strategies, including those shown in past work, vary under different transferring environments. In particular, (i) Kullback-Leibler based density-ratio estimation tends to outperform a classification-based approach and (ii) aggregating document-level weights into query-level weights is likely superior to direct estimation using a query-level representation. The Nemenyi statistical test, applied across multiple datasets, indicates that most weighting transfer learning methods do not significantly outperform baselines, although there is potential for the further development of such techniques.

    Original languageEnglish
    Title of host publicationProceedings of the 25th ACM International Conference on Information and Knowledge Management (CIKM 2016)
    Subtitle of host publicationOctober 24-28, 2016, Indianapolis, IN, USA
    EditorsKavita Ganesan, Chase Geigle, Xia Ning
    Place of PublicationNew York, New York
    PublisherAssociation for Computing Machinery (ACM)
    Pages1413-1422
    Number of pages10
    ISBN (Electronic)9781450340731
    DOIs
    Publication statusPublished - 24 Oct 2016
    EventACM International Conference on Information and Knowledge Management 2016 - Indianapolis, United States of America
    Duration: 24 Oct 201628 Oct 2016
    Conference number: 25th
    https://dl.acm.org/doi/proceedings/10.1145/2983323

    Conference

    ConferenceACM International Conference on Information and Knowledge Management 2016
    Abbreviated titleCIKM 2016
    CountryUnited States of America
    CityIndianapolis
    Period24/10/1628/10/16
    Internet address

    Keywords

    • Information retrieval
    • Learning to rank
    • Ranking adaptation

    Cite this