NDLI: A Hybrid Approach to High Availability in Stream Processing Systems

Content Provider	IEEE Xplore Digital Library
Author	Zhe Zhang Yu Gu Fan Ye Hao Yang Minkyong Kim Hui Lei Zhen Liu
Copyright Year	2010
Abstract	Stream processing is widely used by today’s applications such as financial data analysis and disaster response. In distributed stream processing systems, machine fail-stop events are handled by either active standby or passive standby. However, existing high availability (HA) schemes have not sufficiently addressed the situation when a machine becomes temporarily unavailable due to data rate spikes, intensive analysis or job sharing, which happens frequently but lasts for short time. It is not clear how well active and passive standby fare against such transient unavailability. In this paper, we first critically examine the suitability of active and passive standby against transient unavailability in a real testbed environment. We find that both approaches have advantages and drawbacks, but neither is ideal to provide fast recovery at low overhead as required to handle transient unavailability. Based on the insights gained, we propose a novel hybrid HA method that switches between active and passive standby modes depending on the occurrence of failure events. It presents a desirable tradeoff that is different from existing HA approaches: low overhead during normal conditions and fast recovery upon transient or permanent failure events. We have implemented our hybrid method and compared it with existing HA designs with comprehensive evaluation. The results show that our hybrid method can reduce two-thirds of the recovery time compared to passive standby and 80% message overhead compared to active standby, allowing applications to enjoy uninterrupted processing without paying a high premium.
Starting Page	138
Ending Page	148
File Size	449811
Page Count	11
File Format	PDF
ISBN	9781424472611
ISSN	10636927
e-ISBN	9781424472628
DOI	10.1109/ICDCS.2010.81
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2010-06-21
Publisher Place	Italy
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Availability Telecommunication traffic Delay Distributed computing Streaming media Laboratories Computer science Data analysis Testing Switches stream processing systems hybrid method high availability
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Hardware and Architecture Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Balanced high availability in layered distributed computing systems

A Case Study of Traffic Locality in Internet P2P Live Streaming Systems

PRINS: Optimizing Performance of Reliable Internet Storages

Minimizing Latency in Fault-Tolerant Distributed Stream Processing Systems

Parallel and distributed computing for cybersecurity

Video multicast over the Internet

Sword: Scalable and Flexible Workload Generator for Distributed Data Processing Systems

Toward Predictive Failure Management for Distributed Stream Processing Systems

Aggressive release consistency for software distributed shared memory

A Hybrid Approach to High Availability in Stream Processing Systems

Similar Documents

Balanced high availability in layered distributed computing systems

A Case Study of Traffic Locality in Internet P2P Live Streaming Systems

PRINS: Optimizing Performance of Reliable Internet Storages

Minimizing Latency in Fault-Tolerant Distributed Stream Processing Systems

Parallel and distributed computing for cybersecurity

Video multicast over the Internet

Sword: Scalable and Flexible Workload Generator for Distributed Data Processing Systems

Toward Predictive Failure Management for Distributed Stream Processing Systems

Aggressive release consistency for software distributed shared memory

A Hybrid Approach to High Availability in Stream Processing Systems