NDLI: Improved file synchronization techniques for maintaining large replicated collections over slow networks

Content Provider	IEEE Xplore Digital Library
Author	Suel, T. Noel, P. Trendafilov, D.
Copyright Year	2004
Description	Author affiliation: CIS Dept., Polytech. Univ. Brooklyn, NY, USA (Suel, T.; Noel, P.; Trendafilov, D.)
Abstract	We study the problem of maintaining large replicated collections of files or documents in a distributed environment with limited bandwidth. This problem arises in a number of important applications, such as synchronization of data between accounts or devices, content distribution and Web caching networks, Web site mirroring, storage networks, and large scale Web search and mining. At the core of the problem lies the following challenge, called the file synchronization problem: given two versions of a file on different machines, say an outdated and a current one, how can we update the outdated version with minimum communication cost, by exploiting the significant similarity between the versions? While a popular open source tool for this problem called rsync is used in hundreds of thousands of installations, there have been only very few attempts to improve upon this tool in practice. We propose a framework for remote file synchronization and describe several new techniques that result in significant bandwidth savings. Our focus is on applications where very large collections have to be maintained over slow connections. We show that a prototype implementation of our framework and techniques achieves significant improvements over rsync. As an example application, we focus on the efficient synchronization of very large Web page collections for the purpose of search, mining, and content distribution.
Sponsorship	Microsoft Res. bea IBM MITRE Sun Microsystems
Starting Page	153
Ending Page	164
File Size	626980
Page Count	12
File Format	PDF
ISBN	0769520650
ISSN	10636382
DOI	10.1109/ICDE.2004.1319992
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2004-04-02
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Bandwidth Web search Web pages Large-scale systems Computational Intelligence Society Costs Prototypes Engineering profession Internet Search engines
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Improved File Synchronization Techniques for Maintaining Large Replicated Collections over Slow Networks (2004)

Efficient query evaluation on large textual collections in a peer-to-peer environment

Compressing the graph structure of the Web

Optimized Inverted List Assignment in Distributed Search Engine Architectures

A natural language processing based Internet agent

Improved single-round protocols for remote file synchronization

Semantic Similarity Based Focused Crawling

Lexical postcorrection of OCR-results:the web as a dynamic secondary dictionary?

Change aware distributed file system for a distributed search engine

Improved file synchronization techniques for maintaining large replicated collections over slow networks

Similar Documents

Improved File Synchronization Techniques for Maintaining Large Replicated Collections over Slow Networks (2004)

Efficient query evaluation on large textual collections in a peer-to-peer environment

Compressing the graph structure of the Web

Optimized Inverted List Assignment in Distributed Search Engine Architectures

A natural language processing based Internet agent

Improved single-round protocols for remote file synchronization

Semantic Similarity Based Focused Crawling

Lexical postcorrection of OCR-results:the web as a dynamic secondary dictionary?

Change aware distributed file system for a distributed search engine

Improved file synchronization techniques for maintaining large replicated collections over slow networks