NDLI: Maestro: Replica-Aware Map Scheduling for MapReduce

Content Provider	IEEE Xplore Digital Library
Author	Ibrahim, S. Hai Jin Lu Lu Bingsheng He Antoniu, G. Song Wu
Copyright Year	2012
Abstract	MapReduce has emerged as a leading programming model for data-intensive computing. Many recent research efforts have focused on improving the performance of the distributed frameworks supporting this model. Many optimizations are network-oriented and most of them mainly address the data shuffling stage of MapReduce. Our studies with Hadoop demonstrate that, apart from the shuffling phase, another source of excessive network traffic is the high number of map task executions which process remote data. That leads to an excessive number of useless speculative executions of map tasks and to an unbalanced execution of map tasks across different machines. All these factors produce a noticeable performance degradation. We propose a novel scheduling algorithm for map tasks, named Maestro, to improve the overall performance of the MapReduce computation. Maestro schedules the map tasks in two waves: first, it fills the empty slots of each data node based on the number of hosted map tasks and on the replication scheme for their input data, second, runtime scheduling takes into account the probability of scheduling a map task on a given machine depending on the replicas of the task's input data. These two waves lead to a higher locality in the execution of map tasks and to a more balanced intermediate data distribution for the shuffling phase. In our experiments on a 100-node cluster, Maestro achieves around 95% local map executions, reduces speculative map tasks by 80% and results in an improvement of up to 34% in the execution time.
Starting Page	435
Ending Page	442
File Size	477041
Page Count	8
File Format	PDF
ISBN	9781467313957
DOI	10.1109/CCGrid.2012.122
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2012-05-13
Publisher Place	Canada
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Benchmark testing Runtime Processor scheduling Educational institutions Scheduling Distributed databases Schedules scheduling MapReduce Hadoop cloud computing replication
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Maestro: Replica-Aware Map Scheduling for MapReduce

Locality-Aware Reduce Task Scheduling for MapReduce

TaskTracker Aware Scheduling for Hadoop MapReduce

Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic

Maestro: Replica-Aware Map Scheduling for MapReduce

An effective data locality aware task scheduling method for MapReduce framework in heterogeneous environments

An improved schedule of MapReduce programming environment in cloud computing

MC2: Map Concurrency Characterization for MapReduce on the Cloud

MapReduce in the Clouds for Science

Maestro: Replica-Aware Map Scheduling for MapReduce

Similar Documents

Maestro: Replica-Aware Map Scheduling for MapReduce

Locality-Aware Reduce Task Scheduling for MapReduce

TaskTracker Aware Scheduling for Hadoop MapReduce

Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic

Maestro: Replica-Aware Map Scheduling for MapReduce

An effective data locality aware task scheduling method for MapReduce framework in heterogeneous environments

An improved schedule of MapReduce programming environment in cloud computing

MC2: Map Concurrency Characterization for MapReduce on the Cloud

MapReduce in the Clouds for Science

Maestro: Replica-Aware Map Scheduling for MapReduce