NDLI: IR evaluation methods for retrieving highly relevant documents

Content Provider	ACM Digital Library
Author	Järvelin, Kalervo Kekäläinen, Jaana
Abstract	This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modem large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In- Query I) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.
Starting Page	243
Ending Page	250
Page Count	8
File Format	PDF
ISSN	01635840
DOI	10.1145/3130348.3130374
Journal	ACM SIGIR Forum (SIGF)
Volume Number	51
Issue Number	2
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	1978-08-01
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Content Type	Text
Resource Type	Article
Subject	Management Information Systems Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

IR evaluation methods for retrieving highly relevant documents

Ir evaluation methods for retrieving highly relevant documents (2000).

Retrieving highly dynamic widely distributed information

IR research:systems, interaction, evaluation and theories

Some ideas for estimating the number of relevant documents

Workshop on mathematical/formal methods in information retrieval (MF/IR 2000)

Recommended reading for IR research students

ACM SIGIR workshop on mathematical/formal methods in information retrieval MF/IR 2005

Presentation schemes for component analysis in IR experiments

IR evaluation methods for retrieving highly relevant documents

Similar Documents

IR evaluation methods for retrieving highly relevant documents

Ir evaluation methods for retrieving highly relevant documents (2000).

Retrieving highly dynamic widely distributed information

IR research:systems, interaction, evaluation and theories

Some ideas for estimating the number of relevant documents

Workshop on mathematical/formal methods in information retrieval (MF/IR 2000)

Recommended reading for IR research students

ACM SIGIR workshop on mathematical/formal methods in information retrieval MF/IR 2005

Presentation schemes for component analysis in IR experiments

IR evaluation methods for retrieving highly relevant documents