NDLI: Exploiting Analytics Shipping with Virtualized MapReduce on HPC Backend Storage Servers

Content Provider	IEEE Xplore Digital Library
Author	Cong Xu Goldstone, R. Zhuo Liu Hui Chen Neitzel, B. Weikuan Yu
Copyright Year	1990
Abstract	Large-scale scientific applications on High-Performance Computing (HPC) systems are generating a colossal amount of data that need to be analyzed in a timely manner for new knowledge, but are too costly to transfer due to their sheer size. Many HPC systems have catered to in situ analytics solutions that can analyze temporary datasets as they are generated, i.e., without storing to long-term storage media. However, there is still an open question on how to conduct efficient analytics of permanent datasets that have been stored to the backend persistent storage because of their long-term value. To fill the void, we exploit the analytics shipping model for fast analysis of large-scale scientific datasets on HPC backend storage servers. Through an efficient integration of MapReduce and the popular Lustre storage system, we have developed a Virtualized Analytics Shipping (VAS) framework that can ship MapReduce programs to Lustre storage servers. The VAS framework includes three component techniques: (a) virtualized analytics shipping with fast network and disk I/O; (b) stripe-aligned data distribution and task scheduling and (c) pipelined intermediate data merging and reducing. The first technique provides necessary isolation between MapReduce analytics and Lustre I/O services. The second and third techniques optimize MapReduce on Lustre and avoid explicit shuffling. Our performance evaluation demonstrates that VAS offers an exemplary implementation of analytics shipping and delivers fast and virtualized MapReduce programs on backend Lustre storage servers.
Sponsorship	IEEE Computer Society
Starting Page	185
Ending Page	196
Page Count	12
File Size	1566454
File Format	PDF
ISSN	10459219
Volume Number	27
Issue Number	1
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2016-01-01
Publisher Place	U.S.A.
Access Restriction	One Nation One Subscription (ONOS)
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Servers Yarn Bandwidth Interference Analytical models Merging Computational modeling Lustre Analytics Shipping Hadoop MapReduce HPC lustre Analytics shipping hadoop mapreduce
Content Type	Text
Resource Type	Article
Subject	Signal Processing Computational Theory and Mathematics Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

High-Performance Design of YARN MapReduce on Modern HPC Clusters with Lustre and RDMA

NOHAA: A NOvel Framework for HPC Analytics over Windows Azure

YARNsim: Simulating Hadoop YARN

Mammoth: Gearing Hadoop Towards Memory-Intensive MapReduce Applications

Virtual Shuffling for Efficient Data Movement in MapReduce

Enhancing MapReduce via Asynchronous Data Processing

Hadoop Performance Modeling for Job Estimation and Resource Provisioning

Supporting HPC Analytics Applications with Access Patterns Using Data Restructuring and Data-Centric Scheduling Techniques in MapReduce

Design and Evaluation of Network-Levitated Merge for Hadoop Acceleration

Exploiting Analytics Shipping with Virtualized MapReduce on HPC Backend Storage Servers

Similar Documents

High-Performance Design of YARN MapReduce on Modern HPC Clusters with Lustre and RDMA

NOHAA: A NOvel Framework for HPC Analytics over Windows Azure

YARNsim: Simulating Hadoop YARN

Mammoth: Gearing Hadoop Towards Memory-Intensive MapReduce Applications

Virtual Shuffling for Efficient Data Movement in MapReduce

Enhancing MapReduce via Asynchronous Data Processing

Hadoop Performance Modeling for Job Estimation and Resource Provisioning

Supporting HPC Analytics Applications with Access Patterns Using Data Restructuring and Data-Centric Scheduling Techniques in MapReduce

Design and Evaluation of Network-Levitated Merge for Hadoop Acceleration

Exploiting Analytics Shipping with Virtualized MapReduce on HPC Backend Storage Servers