NDLI: Improving HDFS write performance using efficient replica placement

Content Provider	IEEE Xplore Digital Library
Author	Patel Neha, M. Patel Narendra, M. Hasan, M.I. Shah Parth, D. Patel Mayur, M.
Copyright Year	2014
Description	Author affiliation: CSPIT-CHARUSAT, Anand, India (Patel Neha, M.; Shah Parth, D.; Patel Mayur, M.) \|\| BVM Eng., Anand, India (Patel Narendra, M.; Hasan, M.I.)
Abstract	In last half decade, there is a tremendous growth in the network applications; we are experiencing an information explosion era and for that large amount of distributed data being stored and managed. Distributed file system is designed to handle these types of data. Major design issues in DFS are scalability, fault tolerance, flexibility and availability. The most prevalent DFS to deal with these challenges is the Hadoop Distributed File System (HDFS) which is a variant of the Google File System (GFS). Apache Hadoop able to solve current issues of Big Data by simplifying the implementation of data intensive and exceptionally parallel distributed applications. HDFS handles fault tolerance using Data Replication. HDFS replicates each data block on different datanode for reliability and availability. The existing implementation of HDFS in Hadoop performs replication in a pipelined manner which takes much time for replication. Here proposed system is an alternative parallel approach for efficient replica placement in HDFS to improve throughput. The experimentation has been performed to compare its performance with existing pipelined replication approach, which improve HDFS write throughput up to 10% testified by the TestDFSIO benchmark. This paper also depicts the analysis on the basis of different HDFS configuration parameter like file block size and replication factor which affects HDFS write performance in both approaches.
Starting Page	36
Ending Page	39
File Size	720385
Page Count	4
File Format	PDF
e-ISBN	9781479942367
DOI	10.1109/CONFLUENCE.2014.6949234
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-09-25
Publisher Place	India
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Fault tolerance File systems Replication factor (R.F) Pipelines Fault tolerant systems Distributed databases Pipelined Writing Hadoop Distributed File System (HDFS) Throughput Parallel
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

High Performance and Fault Tolerant Distributed File System for Big Data Storage and Processing Using Hadoop

Improving HDFS performance using local caching system

A Study of Replica Reconstruction Schemes for Multi-rack HDFS Clusters

Performance Evaluation of Read and Write Operations in Hadoop Distributed File System

HDFS+: Concurrent Writes Improvements for HDFS

Fault Tolerant Erasure Coded Replication for HDFS Based Cloud Storage

Aurora: Adaptive Block Replication in Distributed File Systems

A review on hadoop — HDFS infrastructure extensions

hatS: A Heterogeneity-Aware Tiered Storage for Hadoop

Improving HDFS write performance using efficient replica placement

Similar Documents

High Performance and Fault Tolerant Distributed File System for Big Data Storage and Processing Using Hadoop

Improving HDFS performance using local caching system

A Study of Replica Reconstruction Schemes for Multi-rack HDFS Clusters

Performance Evaluation of Read and Write Operations in Hadoop Distributed File System

HDFS+: Concurrent Writes Improvements for HDFS

Fault Tolerant Erasure Coded Replication for HDFS Based Cloud Storage

Aurora: Adaptive Block Replication in Distributed File Systems

A review on hadoop — HDFS infrastructure extensions

hatS: A Heterogeneity-Aware Tiered Storage for Hadoop

Improving HDFS write performance using efficient replica placement