NDLI: Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Please wait, while we are loading the content...

Tiera: towards flexible multi-tiered cloud storage instances

G-DUR: a middleware for assembling, analyzing, and improving transactional protocols

LVD: lean virtual disks

Trustworthy geographically fenced hybrid clouds

StackSync: bringing elasticity to dropbox-like file synchronization

Slider: incremental sliding window analytics

Fast, expressive top-k matching

HyRec: leveraging browsers for scalable recommenders

Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Thematic event processing

Affinity-aware checkpoint restart

Mutable checkpoint-restart: automating live update for generic server programs

Consensus inside

A framework for the dynamic evolution of highly-available dataflow programs

Building global and scalable systems with atomic multicast

Strong consistency in cache augmented SQL systems

SOUP: an online social network by the people, for the people

SenSocial: a middleware for integrating online social networks and mobile sensing data streams

PLEROMA: a SDN-based high performance publish/subscribe middleware

CrowdWiFi: efficient crowdsensing of roadside WiFi networks

Publish/subscribe network designs for multiplayer games

Stage-aware anomaly detection through tracking log points

Archie: a speculative replicated transactional system

Mitigating interference in cloud services by middleware reconfiguration

CAMP: a cost adaptive multi-queue eviction policy for key-value stores

Stay-Away, protecting sensitive applications from performance interference

Virtual machine consolidation in the wild

Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Content Provider	ACM Digital Library
Author	Guo, Yanfei Cheng, Dazhao Rao, Jia Zhou, Xiaobo
Abstract	The deployment of MapReduce in datacenters and clouds present several challenges in achieving good job performance. Compared to in-house dedicated clusters, datacenters and clouds often exhibit significant hardware and performance heterogeneity due to continuous server replacement and multi-tenant interferences. As most Mapreduce implementations assume homogeneous clusters, heterogeneity can cause significant load imbalance in task execution, leading to poor performance and low cluster utilizations. Despite existing optimizations on task scheduling and load balancing, MapReduce still performs poorly on heterogeneous clusters. In this paper, we find that the homogeneous configuration of tasks on heterogeneous nodes can be an important source of load imbalance and thus cause poor performance. Tasks should be customized with different settings to match the capabilities of heterogeneous nodes. To this end, we propose an adaptive task tuning approach, Ant, that automatically finds the optimal settings for individual tasks running on different nodes. Ant works best for large jobs with multiple rounds of map task execution. It first configures tasks with randomly selected configurations and gradually improves tasks settings by reproducing the settings from best performing tasks and discarding poor performing configurations. To accelerate task tuning and avoid trapping in local optimum, Ant uses genetic functions during task configuration. Experimental results on a heterogeneous cluster and a virtual cluster with varying hardware capabilities show that Ant improves the average job completion time by 23%, 11%, and 16% compared to stock Hadoop, customized Hadoop with industry recommendations, and a profiling-based configuration approach, respectively.
Starting Page	97
Ending Page	108
Page Count	12
File Format	PDF
ISBN	9781450327855
DOI	10.1145/2663165.2666089
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2014-12-08
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Adaptive task tuning Heterogeneous clusters Improving mapreduce performance
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Improving mapreduce performance through data placement in heterogeneous hadoop clusters.

Performance Modeling of MapReduce Jobs in Heterogeneous Cloud Environments

On Improving Fault Tolerance for Heterogeneous Hadoop MapReduce Clusters

Improving MapReduce Performance by Data Prefetching in Heterogeneous or Shared Environments

A usage-aware scheduler for improving MapReduce performance in heterogeneous environments

A review of adaptive approaches to MapReduce scheduling in heterogeneous environments

A Throughput Driven Task Scheduler for Improving MapReduce Performance in Job-Intensive Environments

Task fusion: improving utilization of multi-user clusters

Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Similar Documents

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Improving mapreduce performance through data placement in heterogeneous hadoop clusters.

Performance Modeling of MapReduce Jobs in Heterogeneous Cloud Environments

On Improving Fault Tolerance for Heterogeneous Hadoop MapReduce Clusters

Improving MapReduce Performance by Data Prefetching in Heterogeneous or Shared Environments

A usage-aware scheduler for improving MapReduce performance in heterogeneous environments

A review of adaptive approaches to MapReduce scheduling in heterogeneous environments

A Throughput Driven Task Scheduler for Improving MapReduce Performance in Job-Intensive Environments

Task fusion: improving utilization of multi-user clusters

Improving MapReduce performance in heterogeneous environments with adaptive task tuning