NDLI: Towards Machine Learning-Based Auto-tuning of MapReduce

Content Provider	IEEE Xplore Digital Library
Author	Yigitbasi, N. Willke, T.L. Guangdeng Liao Epema, D.
Copyright Year	2013
Description	Author affiliation: Intel Labs., Hillsboro, OR, USA (Yigitbasi, N.; Willke, T.L.; Guangdeng Liao) \|\| Delft Univ. of Technol., Delft, Netherlands (Epema, D.)
Abstract	MapReduce, which is the de facto programming model for large-scale distributed data processing, and its most popular implementation Hadoop have enjoyed widespread adoption in industry during the past few years. Unfortunately, from a performance point of view getting the most out of Hadoop is still a big challenge due to the large number of configuration parameters. Currently these parameters are tuned manually by trial and error, which is ineffective due to the large parameter space and the complex interactions among the parameters. Even worse, the parameters have to be re-tuned for different MapReduce applications and clusters. To make the parameter tuning process more effective, in this paper we explore machine learning-based performance models that we use to auto-tune the configuration parameters. To this end, we first evaluate several machine learning models with diverse MapReduce applications and cluster configurations, and we show that support vector regression model (SVR) has good accuracy and is also computationally efficient. We further assess our auto-tuning approach, which uses the SVR performance model, against the Starfish auto tuner, which uses a cost-based performance model. Our findings reveal that our auto-tuning approach can provide comparable or in some cases better performance improvements than Starfish with a smaller number of parameters. Finally, we propose and discuss a complete and practical end-to-end auto-tuning flow that combines our machine learning-based performance models with smart search algorithms for the effective training of the models and the effective exploration of the parameter space.
Sponsorship	IEEE
Starting Page	11
Ending Page	20
File Size	396195
Page Count	10
File Format	PDF
ISBN	9780769551029
ISSN	15267539
DOI	10.1109/MASCOTS.2013.9
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2013-08-14
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Benchmark testing Computational modeling Data models Tuning Accuracy Training Training data big data distributed systems performance modeling hadoop
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Performance Assurance Model for HiveQL on Large Data Volume

vHadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Tuple MapReduce: Beyond Classic MapReduce

Performance evaluation and tuning for MapReduce computing in Hadoop distributed file system

Machine learning approaches on map reduce for Big Data analytics

Data chaos: An entropy based MapReduce framework for scalable learning

Fast Modeling of Analytics Workloads for Big Data Services

Machine Learning Based Auto-Tuning for Enhanced OpenCL Performance Portability

An experimental approach towards big data for analyzing memory utilization on a hadoop cluster using HDFS and MapReduce

Towards Machine Learning-Based Auto-tuning of MapReduce

Similar Documents

Performance Assurance Model for HiveQL on Large Data Volume

vHadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Tuple MapReduce: Beyond Classic MapReduce

Performance evaluation and tuning for MapReduce computing in Hadoop distributed file system

Machine learning approaches on map reduce for Big Data analytics

Data chaos: An entropy based MapReduce framework for scalable learning

Fast Modeling of Analytics Workloads for Big Data Services

Machine Learning Based Auto-Tuning for Enhanced OpenCL Performance Portability

An experimental approach towards big data for analyzing memory utilization on a hadoop cluster using HDFS and MapReduce

Towards Machine Learning-Based Auto-tuning of MapReduce