NDLI: DREAMS: Dynamic resource allocation for MapReduce with data skew

Content Provider	IEEE Xplore Digital Library
Author	Zhihong Liu Qi Zhang Zhani, M.F. Boutaba, R. Yaping Liu Zhenghu Gong
Copyright Year	2015
Description	Author affiliation: Sci. & Technol. on Parallel & Distrib. Process. Lab., Nat. Univ. of Defense Technol., Changsha, China (Yaping Liu) \|\| Coll. of Comput., Nat. Univ. of Defense Technol., Changsha, China (Zhihong Liu; Zhenghu Gong) \|\| David R. Cheriton Sch. of Comput. Sci., Univ. of Waterloo, Waterloo, ON, Canada (Qi Zhang; Zhani, M.F.; Boutaba, R.)
Abstract	MapReduce has become a popular model for large-scale data processing in recent years. However, existing MapRe-duce schedulers still suffer from an issue known as partitioning skew, where the output of map tasks is unevenly distributed among reduce tasks. In this paper, we present DREAMS, a framework that provides run-time partitioning skew mitigation. Unlike previous approaches that try to balance the workload of reducers by repartitioning the intermediate data assigned to each reduce task, in DREAMS we cope with partitioning skew by adjusting task run-time resource allocation. We show that our approach allows DREAMS to eliminate the overhead of data repartitioning. Through experiments using both real and synthetic workloads running on a 11-node virtual virtualised Hadoop cluster, we show that DREAMS can effectively mitigate negative impact of partitioning skew, thereby improving job performance by up to 20.3%.
Starting Page	18
Ending Page	26
File Size	1371221
Page Count	9
File Format	PDF
ISBN	9781479982417
DOI	10.1109/INM.2015.7140272
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2015-05-11
Publisher Place	Canada
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Resource management Containers Predictive models Mathematical model Monitoring Biomedical monitoring Yarn
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

DREAMS: Dynamic Resource Allocation for MapReduce with Data Skew

Improving utilization through dynamic VM resource allocation in hybrid cloud environment

Telemetry for Elastic Data (TED): Middleware for MapReduce Job Metering and Rating

SMapReduce: Optimising Resource Allocation by Managing Working Slots at Runtime

Provision of Disk I/O Guarantee for MapReduce Applications

Elastic MapReduce Execution

EHadoop: Network I/O Aware Scheduler for Elastic MapReduce Cluster

Chisel: A Resource Savvy Approach for Handling Skew in MapReduce Applications

GERBIL: MPI+YARN

DREAMS: Dynamic resource allocation for MapReduce with data skew

Similar Documents

DREAMS: Dynamic Resource Allocation for MapReduce with Data Skew

Improving utilization through dynamic VM resource allocation in hybrid cloud environment

Telemetry for Elastic Data (TED): Middleware for MapReduce Job Metering and Rating

SMapReduce: Optimising Resource Allocation by Managing Working Slots at Runtime

Provision of Disk I/O Guarantee for MapReduce Applications

Elastic MapReduce Execution

EHadoop: Network I/O Aware Scheduler for Elastic MapReduce Cluster

Chisel: A Resource Savvy Approach for Handling Skew in MapReduce Applications

GERBIL: MPI+YARN

DREAMS: Dynamic resource allocation for MapReduce with data skew