NDLI: Breaking the boundary for whole-system performance optimization of big data

Content Provider	IEEE Xplore Digital Library
Author	Yan Li Kun Wang Qi Guo Xin Li Xiaochen Zhang Guancheng Chen Tao Liu Jian Li
Copyright Year	2013
Description	Author affiliation: IBM Res. - Austin, Austin, TX, USA (Jian Li) \|\| City Univ. of New York, New York, NY, USA (Xiaochen Zhang) \|\| IBM Res. - China, China (Yan Li; Kun Wang; Qi Guo; Xin Li; Guancheng Chen; Tao Liu)
Abstract	MapReduce plays an critical role in finding insights in Big Data. The performance optimization of MapReduce programs is challenging because it requires a comprehensive understanding of the whole system including both hardware layers (processors, storages, networks and etc), and software stacks (operating systems, JVM, runtime, applications and etc). However, most of the existing performance tuning and optimization are based on empirical and heuristic attempts. It remains a blank on how to build a systematical framework which breaks the boundary of multiple layers for performance optimization. In this paper, we propose a performance evaluation framework by correlating performance metrics from different layers, which provides insights to efficiently pinpoint the performance issue. This framework is composed of a series of predefined patterns. Each pattern indicates one or more potential issues. The behavior of a MapReduce program is mapped to the corresponding resource utilization. The framework provides a holistic approach which allows users at different levels of experience to conduct MapReduce program performance optimization. We use Terasort benchmark running on a 10-node Power7R2 cluster as a real case to show how this framework improves the performance. By this framework, we finally get the Terasort result improved from 47 mins to less than 8 mins. In addition to the best practice on performance tuning, several key findings are summarized as valuable workload analysis for JVM, MapReduce runtime and application design.
Starting Page	126
Ending Page	131
File Size	848036
Page Count	6
File Format	PDF
e-ISBN	9781479912353
DOI	10.1109/ISLPED.2013.6629278
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2013-09-04
Publisher Place	China
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Optimization Tuning Hardware Runtime Java Indexes Software
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Breaking the boundary for whole-system performance optimization of big data

Correlation-based performance analysis for full-system MapReduce optimization

Some remarks on self-tuning logical database design

Assessing the Impact of Optimization in Java Virtual Machines (2005)

Runtime protecting system for java applications with dynamic data flow analyzing

MPI tuning with Intel/spl copy/ Trace Analyzer and Intel/spl copy/ Trace Collector

Feasibility of high level compiler optimizations in online synthesis

Relative factors in performance analysis of java virtual.

Runtime Data Analysis for Java Programs (2003)

Breaking the boundary for whole-system performance optimization of big data

Similar Documents

Breaking the boundary for whole-system performance optimization of big data

Correlation-based performance analysis for full-system MapReduce optimization

Some remarks on self-tuning logical database design

Assessing the Impact of Optimization in Java Virtual Machines (2005)

Runtime protecting system for java applications with dynamic data flow analyzing

MPI tuning with Intel/spl copy/ Trace Analyzer and Intel/spl copy/ Trace Collector

Feasibility of high level compiler optimizations in online synthesis

Relative factors in performance analysis of java virtual.

Runtime Data Analysis for Java Programs (2003)

Breaking the boundary for whole-system performance optimization of big data