NDLI: A Comparative Analysis of Interleaving Methods for Aggregated Search

Content Provider	ACM Digital Library
Author	Zhou, Ke Rijke, Maarten De Schuth, Anne Chuklin, Aleksandr
Copyright Year	2015
Abstract	A result page of a modern search engine often goes beyond a simple list of “10 blue links.” Many specific user needs (e.g., News, Image, Video) are addressed by so-called aggregated or vertical search solutions: specially presented documents, often retrieved from specific sources, that stand out from the regular organic Web search results. When it comes to evaluating ranking systems, such complex result layouts raise their own challenges. This is especially true for so-called interleaving methods that have arisen as an important type of online evaluation: by mixing results from two different result pages, interleaving can easily break the desired Web layout in which vertical documents are grouped together, and hence hurt the user experience. We conduct an analysis of different interleaving methods as applied to aggregated search engine result pages. Apart from conventional interleaving methods, we propose two vertical-aware methods: one derived from the widely used Team-Draft Interleaving method by adjusting it in such a way that it respects vertical document groupings, and another based on the recently introduced Optimized Interleaving framework. We show that our proposed methods are better at preserving the user experience than existing interleaving methods while still performing well as a tool for comparing ranking systems. For evaluating our proposed vertical-aware interleaving methods, we use real-world click data as well as simulated clicks and simulated ranking systems.
Starting Page	1
Ending Page	38
Page Count	38
File Format	PDF
ISSN	10468188
e-ISSN	15582868
DOI	10.1145/2668120
Volume Number	33
Issue Number	2
Journal	ACM Transactions on Information Systems (TOIS)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2015-02-17
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Information retrieval Aggregated search Clicks Interleaved comparison Interleaving Online evaluation
Content Type	Text
Resource Type	Article
Subject	Computer Science Applications Information Systems Business, Management and Accounting

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Fidelity, Soundness, and Efficiency of Interleaved Comparison Methods

Large-scale validation and analysis of interleaved search evaluation

The Effects of Aggregated Search Coherence on Search Behavior

Locality-Based pruning methods for web search

Search improvement via automatic query reformulation

Concept-Based Information Retrieval Using Explicit Semantic Analysis

Lexical ambiguity and information retrieval

Aggregation Methods for Proximity-Based Opinion Retrieval

Repeatable evaluation of search services in dynamic environments

A Comparative Analysis of Interleaving Methods for Aggregated Search

Similar Documents

Fidelity, Soundness, and Efficiency of Interleaved Comparison Methods

Large-scale validation and analysis of interleaved search evaluation

The Effects of Aggregated Search Coherence on Search Behavior

Locality-Based pruning methods for web search

Search improvement via automatic query reformulation

Concept-Based Information Retrieval Using Explicit Semantic Analysis

Lexical ambiguity and information retrieval

Aggregation Methods for Proximity-Based Opinion Retrieval

Repeatable evaluation of search services in dynamic environments

A Comparative Analysis of Interleaving Methods for Aggregated Search