NDLI: Choices in batch information retrieval evaluation

Please wait, while we are loading the content...

Economic models of search

Using eye tracking for evaluating web search interfaces

Efficient top-k retrieval with signatures

An enterprise search paradigm based on extended query auto-completion: do we still need search and navigation?

Classifying microblogs for disasters

ADCS reaches adulthood: an analysis of the conference and its community over the last eighteen years

Merging algorithms for enterprise search

Power walk: revisiting the random surfer

Exploring the magic of WAND

Integrated instance- and class-based generative modeling for text classification

Choices in batch information retrieval evaluation

Conditional collocation in Japanese

Visual summarisation of text for surveillance and situational awareness in hospitals

Quality biased thread retrieval using the voting model

Malformed UTF-8 and spam

Crisis management knowledge from social media

Towards information retrieval evaluation with reduced and only positive judgements

Managing short postings lists

Choices in batch information retrieval evaluation

Content Provider	ACM Digital Library
Author	Thomas, Paul Moffat, Alistair Scholer, Falk
Abstract	Web search tools are used on a daily basis by billions of people. The commercial providers of these services spend large amounts of money measuring their own effectiveness and benchmarking against their competitors; nothing less than their corporate survival is at stake. Techniques for offline or "batch" evaluation of search quality have received considerable attention, spanning ways of constructing relevance judgments; ways of using them to generate numeric scores; and ways of inferring system "superiority" from sets of such scores. Our purpose in this paper is consider these mechanisms as a chain of inter-dependent activities, in order to explore some of the ramifications of alternative components. By disaggregating the different activities, and asking what the ultimate objective of the measurement process is, we provide new insights into evaluation approaches, and are able to suggest new combinations that might prove fruitful avenues for exploration. Our observations are examined with reference to data collected from a user study covering 34 users undertaking a total of six search tasks each, using two systems of markedly different quality. We hope to encourage broader awareness of the many factors that go into an evaluation of search effectiveness, and of the implications of these choices, and encourage researchers to carefully report all aspects of the evaluation process when describing their system performance experiments.
Starting Page	74
Ending Page	81
Page Count	8
File Format	PDF
ISBN	9781450325240
DOI	10.1145/2537734.2537745
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2013-12-05
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Evaluation Effectiveness Relevance judgment Information retrieval
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in