NDLI: Low-Overhead Fault Tolerance for High-Throughput Data Processing Systems

Content Provider	IEEE Xplore Digital Library
Author	Martin, A. Knauth, T. Creutz, S. Becker, D. Weigert, S. Fetzer, C. Brito, A.
Copyright Year	2011
Abstract	The MapReduce programming paradigm proved to be a useful approach for building highly scalable data processing systems. One important reason for its success is simplicity, including the fault tolerance mechanisms. However, this simplicity comes at a price: efficiency. MapReduce's fault tolerance scheme stores too much intermediate information on disk. This inefficiency negatively affects job completion time. Furthermore, this inefficiency in particular forbids the application of MapReduce in near real-time scenarios where jobs need to produce results quickly. In this paper, we discuss an alternative fault tolerance scheme that is inspired by virtual synchrony. The key feature of our approach is a low-overhead deterministic execution. Deterministic execution reduces the amount of persistently stored information. In addition, because persisting intermediate results are no longer required for fault tolerance, we use more efficient communication techniques that considerably improve job completion time and throughput. Our contribution is twofold: (i) we enable the use of MapReduce for jobs ranging from seconds to a few tens of seconds, satisfying these deadlines even in the case of failures, (ii) we considerably reduce the fault tolerance overhead and as such the overhead of MapReduce in general. Our modifications are transparent to the application.
Starting Page	689
Ending Page	699
File Size	242106
Page Count	11
File Format	PDF
ISBN	9781612843841
ISSN	10636927
DOI	10.1109/ICDCS.2011.29
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2011-06-20
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Fault tolerance Fault tolerant systems Synchronization Programming Computer crashes Aggregates Monitoring
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Hardware and Architecture Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Fused Data Structures for Handling Multiple Faults in Distributed Systems

Minimizing Latency in Fault-Tolerant Distributed Stream Processing Systems

AJIRA: A Lightweight Distributed Middleware for MapReduce and Stream Processing

Fault-tolerant external clock synchronization

Using program analysis to identify and compensate for nondeterminism in fault-tolerant, replicated systems

Fault-Tolerant Consensus in Unknown and Anonymous Networks

Atomic broadcast in asynchronous crash-recovery distributed systems

Fault-tolerant Total Order Multicast to asynchronous groups

Load balancing schemes for high-throughput distributed fault-tolerant servers

Low-Overhead Fault Tolerance for High-Throughput Data Processing Systems

Similar Documents

Fused Data Structures for Handling Multiple Faults in Distributed Systems

Minimizing Latency in Fault-Tolerant Distributed Stream Processing Systems

AJIRA: A Lightweight Distributed Middleware for MapReduce and Stream Processing

Fault-tolerant external clock synchronization

Using program analysis to identify and compensate for nondeterminism in fault-tolerant, replicated systems

Fault-Tolerant Consensus in Unknown and Anonymous Networks

Atomic broadcast in asynchronous crash-recovery distributed systems

Fault-tolerant Total Order Multicast to asynchronous groups

Load balancing schemes for high-throughput distributed fault-tolerant servers

Low-Overhead Fault Tolerance for High-Throughput Data Processing Systems