NDLI: Timely Long Tail Identification through Agent Based Monitoring and Analytics

Content Provider	IEEE Xplore Digital Library
Author	Garraghan, P. Xue Ouyang Townend, P. Jie Xu
Copyright Year	2015
Description	Author affiliation: Sch. of Comput., Univ. of Leeds, Leeds, UK (Garraghan, P.; Xue Ouyang; Townend, P.; Jie Xu)
Abstract	The increasing complexity and scale of distributed systems has resulted in the manifestation of emergent behavior which substantially affects overall system performance. A significant emergent property is that of the "Long Tail", whereby a small proportion of task stragglers significantly impact job execution completion times. To mitigate such behavior, straggling tasks occurring within the system need to be accurately identified in a timely manner. However, current approaches focus on mitigation rather than identification, which typically identify stragglers too late in the execution lifecycle. This paper presents a method and tool to identify Long Tail behavior within distributed systems in a timely manner, through a combination of online and offline analytics. This is achieved through historical analysis to profile and model task execution patterns, which then inform online analytic agents that monitor task execution at runtime. Furthermore, we provide an empirical analysis of two large-scale production Cloud data enters that demonstrate the challenge of data skew within modern distributed systems, this analysis shows that approximately 5% of task stragglers caused by data skew impact 50% of the total jobs for batch processes. Our results demonstrate that our approach is capable of identifying task stragglers less than 11% into their execution lifecycle with 98% accuracy, signifying significant improvement over current state-of-the-art practice and enables far more effective mitigation strategies in large-scale distributed systems worldwide.
Starting Page	19
Ending Page	26
File Size	460719
Page Count	8
File Format	PDF
ISBN	9781479987818
ISSN	15550885
DOI	10.1109/ISORC.2015.39
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2015-04-13
Publisher Place	New Zealand
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Analytical models Runtime Monitoring Production Quality of service Cloud computing Servers Long Tail Stragglers Distributed Systems Data analysis agent based datacenter
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Project Daytona: Data Analytics as a Cloud Service

Analysing cloud simulation results using big data analytics model

Describing Cloud Use Case in MetaMORP(h)OSY

Asynchronous Service Analysis of Cloud DVR DataCenter

Event-Based Multi-level Service Monitoring

SMART: An Application Framework for Real Time Big Data Analysis on Heterogeneous Cloud Environments

A monitoring data set for evaluating QoS-aware service-based systems

Predicting service metrics for cluster-based services using real-time analytics

Real-Time QoS Monitoring for Cloud-Based Big Data Analytics Applications in Mobile Environments

Timely Long Tail Identification through Agent Based Monitoring and Analytics

Similar Documents

Project Daytona: Data Analytics as a Cloud Service

Analysing cloud simulation results using big data analytics model

Describing Cloud Use Case in MetaMORP(h)OSY

Asynchronous Service Analysis of Cloud DVR DataCenter

Event-Based Multi-level Service Monitoring

SMART: An Application Framework for Real Time Big Data Analysis on Heterogeneous Cloud Environments

A monitoring data set for evaluating QoS-aware service-based systems

Predicting service metrics for cluster-based services using real-time analytics

Real-Time QoS Monitoring for Cloud-Based Big Data Analytics Applications in Mobile Environments

Timely Long Tail Identification through Agent Based Monitoring and Analytics