NDLI: Design and Evaluation of Network-Levitated Merge for Hadoop Acceleration

Content Provider	IEEE Xplore Digital Library
Author	Weikuan Yu Yandong Wang Xinyu Que
Copyright Year	1990
Abstract	Hadoop is a popular open source implementation of the MapReduce programming model for cloud computing. However, it faces a number of issues to achieve the best performance from the underlying systems. These include a serialization barrier that delays the reduce phase, repetitive merges, and disk accesses, and the lack of portability to different interconnects. To keep up with the increasing volume of data sets, Hadoop also requires efficient I/O capability from the underlying computer systems to process and analyze data. We describe Hadoop-A, an acceleration framework that optimizes Hadoop with plug-in components for fast data movement, overcoming the existing limitations. A novel network-levitated merge algorithm is introduced to merge data without repetition and disk access. In addition, a full pipeline is designed to overlap the shuffle, merge, and reduce phases. Our experimental results show that Hadoop-A significantly speeds up data movement in MapReduce and doubles the throughput of Hadoop. In addition, Hadoop-A significantly reduces disk accesses caused by intermediate data.
Sponsorship	IEEE Computer Society
Starting Page	602
Ending Page	611
Page Count	10
File Size	1822288
File Format	PDF
ISSN	10459219
Volume Number	25
Issue Number	3
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-01-01
Publisher Place	U.S.A.
Access Restriction	One Nation One Subscription (ONOS)
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Pipelines Protocols IP networks Servers Acceleration Data processing Algorithm design and analysis cloud computing Hadoop MapReduce network-levitated merge Hadoop acceleration
Content Type	Text
Resource Type	Article
Subject	Signal Processing Computational Theory and Mathematics Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Processing Cassandra Datasets with Hadoop-Streaming Based Approaches

Exploiting Analytics Shipping with Virtualized MapReduce on HPC Backend Storage Servers

Analyzing Massive Machine Maintenance Data in a Computing Cloud

Hadoop Performance Modeling for Job Estimation and Resource Provisioning

Distributed Selfish Replication

Enabling Public Auditability and Data Dynamics for Storage Security in Cloud Computing

KASR: A Keyword-Aware Service Recommendation Method on MapReduce for Big Data Applications

Implementation of Distributed Floor Control Protocols on Overlay Networks

Overlay Networks with Linear Capacity Constraints

Design and Evaluation of Network-Levitated Merge for Hadoop Acceleration

Similar Documents

Processing Cassandra Datasets with Hadoop-Streaming Based Approaches

Exploiting Analytics Shipping with Virtualized MapReduce on HPC Backend Storage Servers

Analyzing Massive Machine Maintenance Data in a Computing Cloud

Hadoop Performance Modeling for Job Estimation and Resource Provisioning

Distributed Selfish Replication

Enabling Public Auditability and Data Dynamics for Storage Security in Cloud Computing

KASR: A Keyword-Aware Service Recommendation Method on MapReduce for Big Data Applications

Implementation of Distributed Floor Control Protocols on Overlay Networks

Overlay Networks with Linear Capacity Constraints

Design and Evaluation of Network-Levitated Merge for Hadoop Acceleration