NDLI: A pipelined architecture for distributed text query evaluation

Content Provider	Springer Nature Link
Author	Moffat, Alistair Webber, William Zobel, Justin Baeza Yates, Ricardo
Copyright Year	2006
Abstract	Two principal query-evaluation methodologies have been described for cluster-based implementation of distributed information retrieval systems: document partitioning and term partitioning. In a document-partitioned system, each of the processors hosts a subset of the documents in the collection, and executes every query against its local sub-collection. In a term-partitioned system, each of the processors hosts a subset of the inverted lists that make up the index of the collection, and serves them to a central machine as they are required for query evaluation.In this paper we introduce a pipelined query-evaluation methodology, based on a term-partitioned index, in which partially evaluated queries are passed amongst the set of processors that host the query terms. This arrangement retains the disk read benefits of term partitioning, but more effectively shares the computational load. We compare the three methodologies experimentally, and show that term distribution is inefficient and scales poorly. The new pipelined approach offers efficient memory utilization and efficient use of disk accesses, but suffers from problems with load balancing between nodes. Until these problems are resolved, document partitioning remains the preferred method.
Starting Page	205
Ending Page	231
Page Count	27
File Format	PDF
ISSN	13864564
Journal	Information Retrieval
Volume Number	10
Issue Number	3
e-ISSN	15737659
Language	English
Publisher	Kluwer Academic Publishers
Publisher Date	2006-10-05
Publisher Place	Dordrecht
Access Restriction	Subscribed
Subject Keyword	Distributed retrieval Text searching Index representations Data Structures, Cryptology and Information Theory Management of Computing and Information Systems
Content Type	Text
Resource Type	Article
Subject	Library and Information Sciences Information Systems

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Mobile Agents for Distributed and Heterogeneous Information Retrieval

Matching Index Expressions for Information Retrieval

User evaluation of ontology as query construction tool

Dipe-D: A Tool for Knowledge-Based Query Formulation in Information Retrieval

Information Retrieval with a Hybrid Automatic Query Expansion and Data Fusion Procedure

A Task-Oriented Non-Interactive Evaluation Methodology for Information Retrieval Systems

An Evaluation of Automatically Constructed Hypertexts for Information Retrieval

Within-Document Retrieval: A User-Centred Evaluation of Relevance Profiling

A Bayesian Framework for XML Information Retrieval: Searching and Learning with the INEX Collection

A pipelined architecture for distributed text query evaluation

Similar Documents

Mobile Agents for Distributed and Heterogeneous Information Retrieval

Matching Index Expressions for Information Retrieval

User evaluation of ontology as query construction tool

Dipe-D: A Tool for Knowledge-Based Query Formulation in Information Retrieval

Information Retrieval with a Hybrid Automatic Query Expansion and Data Fusion Procedure

A Task-Oriented Non-Interactive Evaluation Methodology for Information Retrieval Systems

An Evaluation of Automatically Constructed Hypertexts for Information Retrieval

Within-Document Retrieval: A User-Centred Evaluation of Relevance Profiling

A Bayesian Framework for XML Information Retrieval: Searching and Learning with the INEX Collection

A pipelined architecture for distributed text query evaluation