NDLI: A Hybrid Scheduling Algorithm for Data Intensive Workloads in a MapReduce Environment

Content Provider	IEEE Xplore Digital Library
Author	Phuong Nguyen Simon, T. Halem, M. Chapman, D. Quang Le
Copyright Year	2012
Abstract	The specific choice of workload task schedulers for Hadoop MapReduce applications can have a dramatic effect on job workload latency. The Hadoop Fair Scheduler (FairS) assigns resources to jobs such that all jobs get, on average, an equal share of resources over time. Thus, it addresses the problem with a FIFO scheduler when short jobs have to wait for long running jobs to complete. We show that even for the FairS, jobs are still forced to wait significantly when the MapReduce system assigns equal sharing of resources due to dependencies between Map, Shuffle, Sort, Reduce phases. We propose a Hybrid Scheduler (HybS) algorithm based on dynamic priority in order to reduce the latency for variable length concurrent jobs, while maintaining data locality. The dynamic priorities can accommodate multiple task lengths, job sizes, and job waiting times by applying a greedy fractional knapsack algorithm for job task processor assignment. The estimated runtime of Map and Reduce tasks are provided to the HybS dynamic priorities from the historical Hadoop log files. In addition to dynamic priority, we implement a reordering of task processor assignment to account for data availability to automatically maintain the benefits of data locality in this environment. We evaluate our approach by running concurrent workloads consisting of the Word-count and Terasort benchmarks, and a satellite scientific data processing workload and developing a simulator. Our evaluation shows the HybS system improves the average response time for the workloads approximately 2.1x faster over the Hadoop FairS with a standard deviation of 1.4x.
Starting Page	161
Ending Page	167
File Size	656961
Page Count	7
File Format	PDF
ISBN	9781467344326
DOI	10.1109/UCC.2012.32
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2012-11-05
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Dynamic priority Runtime Scheduling algorithms Heuristic algorithms Scheduler Hadoop Benchmark testing Dynamic scheduling Workflow Scheduling Time factors MapReduce
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A usage-aware scheduler for improving MapReduce performance in heterogeneous environments

Design Dynamic Data Allocation Scheduler to Improve MapReduce Performance in Heterogeneous Clouds

A heuristic fault tolerant MapReduce framework for minimizing makespan in Hybrid Cloud Environment

Hybrid algorithms for dynamic schedulability testing

HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers

A Heterogeneity-aware Data Distribution and Rebalance Method in Hadoop Cluster

A review of research on MapReduce scheduling algorithms in Hadoop

A MapReduce Task Scheduling Algorithm for Deadline-Constraint in Homogeneous Environment

Dynamic Processing Slots Scheduling for I/O Intensive Jobs of Hadoop MapReduce

A Hybrid Scheduling Algorithm for Data Intensive Workloads in a MapReduce Environment

Similar Documents

A usage-aware scheduler for improving MapReduce performance in heterogeneous environments

Design Dynamic Data Allocation Scheduler to Improve MapReduce Performance in Heterogeneous Clouds

A heuristic fault tolerant MapReduce framework for minimizing makespan in Hybrid Cloud Environment

Hybrid algorithms for dynamic schedulability testing

HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers

A Heterogeneity-aware Data Distribution and Rebalance Method in Hadoop Cluster

A review of research on MapReduce scheduling algorithms in Hadoop

A MapReduce Task Scheduling Algorithm for Deadline-Constraint in Homogeneous Environment

Dynamic Processing Slots Scheduling for I/O Intensive Jobs of Hadoop MapReduce

A Hybrid Scheduling Algorithm for Data Intensive Workloads in a MapReduce Environment