Effort-based information retrieval evaluation with varied evaluation depth and topic sizes

Prabha Rajagopal, Sri Devi Ravana

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


The information retrieval accessed globally is a vital productivity boost for most organization. However, the outcome of information retrieval system evaluation does not agree with the real user‟s satisfaction. Information retrieval systems retrieving low effort documents are preferred by users as effort factor influences user satisfaction. However, there lacks evaluation on deeper depth and reduced topic sizes with the incorporation of effort in evaluating the information retrieval systems. Therefore, this study aims to measure the effectiveness of evaluating information retrieval systems using effort-based relevance judgment with varied evaluation depth, and reduced topic sizes using effort-based relevance judgment. The evaluation depth is varied from 10, 100 and 1000, and the correlation coefficient between the baseline and those generated using effort-based relevance judgment is evaluated. Topic sizes smaller than 50 were also considered and measured to observe changes in correlation coefficient due to effort. As a result, standard evaluation depth, and reduced topic sizes are sufficient for evaluation using effort-based relevance judgment.

Original languageEnglish
Title of host publicationICBIM 2019 - 2019 The 3rd International Conference on Business and Information Management
EditorsHarri Ketamo, Pedro Nueno, Uma Kumar
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages5
ISBN (Electronic)9781450372329
Publication statusPublished - 2019
Externally publishedYes
EventInternational Conference on Business and Information Management 2019 - Paris, France
Duration: 12 Sep 201914 Sep 2019
Conference number: 3rd
https://dl.acm.org/doi/proceedings/10.1145/3361785 (Proceedings)


ConferenceInternational Conference on Business and Information Management 2019
Abbreviated titleICBIM 2019
Internet address


  • Effort
  • Information retrieval evaluation
  • Relevance judgment
  • Topic size

Cite this