NDLI: Workload characterization on a production Hadoop cluster: A case study on Taobao

Content Provider	IEEE Xplore Digital Library
Author	Zujie Ren Xianghua Xu Jian Wan Weisong Shi Min Zhou
Copyright Year	2012
Description	Author affiliation: School of Computer Science and Technology, Hangzhou Dianzi University, China (Zujie Ren; Xianghua Xu; Jian Wan) \|\| Taobao, Inc., Hangzhou, China (Min Zhou) \|\| Department of Computer Science, Wayne State University, Detroit, USA (Weisong Shi)
Abstract	MapReduce is becoming the state-of-the-art computing paradigm for processing large-scale datasets on a large cluster with tens or thousands of nodes. It has been widely used in various fields such as e-commerce, Web search, social networks, and scientific computation. Understanding the characteristics of MapReduce workloads is the key to achieving better configuration decisions and improving the system throughput. However, workload characterization of MapReduce, especially in a large-scale production environment, has not been well studied yet. To gain insight on MapReduce workloads, we collected a two-week workload trace from a 2,000-node Hadoop cluster at Taobao, which is the biggest online e-commerce enterprise in Asia, ranked $14^{th}$ in the world as reported by Alexa. The workload trace covered 912,157 jobs, logged from Dec. 4 to Dec. 20, 2011. We characterized the workload at the granularity of job and task, respectively and concluded with a set of interesting observations. The results of workload characterization are representative and generally consistent with data platforms for e-commerce websites, which can help other researchers and engineers understand the performance and job characteristics of Hadoop in their production environments. In addition, we use these job analysis statistics to derive several implications for potential performance optimization solutions.
Starting Page	3
Ending Page	13
File Size	688617
Page Count	11
File Format	PDF
ISBN	9781467345316
e-ISBN	9781467345323
DOI	10.1109/IISWC.2012.6402895
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2012-11-04
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Production Log-normal distribution Synchronization Educational institutions Google Resource management MapReduce workload characterization Hadoop
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Workload Analysis, Implications, and Optimization on a Production Hadoop Cluster: A Case Study on Taobao

Workload characterization on a production hadoop cluster: a case study on taobao.

1workload analysis, implications and optimization on a production hadoop cluster: a case.

Using a Tunable Knob for Reducing Makespan of MapReduce Jobs in a Hadoop Cluster

Three Pieces of the MapReduce Workload Management Puzzle ∗

Study of Hadoop-MapReduce on Google N-Gram Datasets

An approach for MapReduce based log analysis using Hadoop

WaxElephant: A Realistic Hadoop Simulator for Parameters Tuning and Scalability Analysis

Performance analysis of MapReduce programs on Hadoop cluster

Workload characterization on a production Hadoop cluster: A case study on Taobao

Similar Documents

Workload Analysis, Implications, and Optimization on a Production Hadoop Cluster: A Case Study on Taobao

Workload characterization on a production hadoop cluster: a case study on taobao.

1workload analysis, implications and optimization on a production hadoop cluster: a case.

Using a Tunable Knob for Reducing Makespan of MapReduce Jobs in a Hadoop Cluster

Three Pieces of the MapReduce Workload Management Puzzle ∗

Study of Hadoop-MapReduce on Google N-Gram Datasets

An approach for MapReduce based log analysis using Hadoop

WaxElephant: A Realistic Hadoop Simulator for Parameters Tuning and Scalability Analysis

Performance analysis of MapReduce programs on Hadoop cluster

Workload characterization on a production Hadoop cluster: A case study on Taobao