NDLI: FINGER: A novel erasure coding scheme using fine granularity blocks to improve Hadoop write and update performance

Content Provider	IEEE Xplore Digital Library
Author	Subedi, P. Ping Huang Young, B. Xubin He
Copyright Year	2015
Description	Author affiliation: Electr. & Comput. Eng., Virginia Commonwealth Univ., Richmond, VA, USA (Subedi, P.; Ping Huang; Young, B.; Xubin He)
Abstract	With the explosive increase of the data by volume in various fields of science, engineering, information services, etc., data-intensive computing has gained significant interest in recent years. Various challenges ranging from efficient peta-scale data management to the adoption of highly scalable cloud computing have become a norm for data center administrators. Highly scalable architectures such as Hadoop, BlobSeer and MapR are used in large data centers for efficient data management, and employ 3-way replication for fault tolerance or data availability. One means of reducing storage overhead of replication in data-centers is erasure coding. However, HDFS-RAID (erasure-coded Hadoop) uses large block sizes and does not support update operations. Therefore, changing any file-block content requires recreating the whole file, which effectively reduces the overall write and update performance of the system. We propose FINe Grained ERasure coding scheme (FINGER) for the erasure-coded Hadoop FileSystem, which improves both write and update performance without sacrificing the read performance. The main idea is to chunk the large block size (64 or 128 MB) into smaller chunks; the chunk layout is designed to mitigate extra reads when performing erasure coding on a large block update and maintains the same metdata size as HDFS-RAID. We implement the update operation in Hadoop and conduct testbed experiments to demonstrate that FINGER improves the write and update performance by 38.20% and 8.6% w.r.t. 3-way replication and by 8.08% and up to 5.68×w.r.t HDFS-RAID respectively.
Starting Page	255
Ending Page	264
File Size	642834
Page Count	10
File Format	PDF
e-ISBN	9781467378918
DOI	10.1109/NAS.2015.7255194
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2015-08-06
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Cloud computing Erasure-coded storage system Fingers Layout Switches Hadoop Metadata Encoding Decoding Reliability In-place Update Multi-threading
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Secure multi-key file-sharing for cloud storage with erasure coding

SEARS: Space efficient and reliable storage system in the cloud

BEC: A reliable and efficient mechanism for cloud storage service

EC-FRM: An Erasure Coding Framework to Speed Up Reads for Erasure Coded Cloud Storage Systems

Aggregation Decoding for Multi-failure Recovery in Erasure-Coded Storage

Dominoes: Speculative Repair in Erasure-Coded Hadoop System

Fault Tolerant Erasure Coded Replication for HDFS Based Cloud Storage

Making mapreduce scheduling effective in erasure-coded storage clusters

HDFS+: Erasure Coding Based Hadoop Distributed File System

FINGER: A novel erasure coding scheme using fine granularity blocks to improve Hadoop write and update performance

Similar Documents

Secure multi-key file-sharing for cloud storage with erasure coding

SEARS: Space efficient and reliable storage system in the cloud

BEC: A reliable and efficient mechanism for cloud storage service

EC-FRM: An Erasure Coding Framework to Speed Up Reads for Erasure Coded Cloud Storage Systems

Aggregation Decoding for Multi-failure Recovery in Erasure-Coded Storage

Dominoes: Speculative Repair in Erasure-Coded Hadoop System

Fault Tolerant Erasure Coded Replication for HDFS Based Cloud Storage

Making mapreduce scheduling effective in erasure-coded storage clusters

HDFS+: Erasure Coding Based Hadoop Distributed File System

FINGER: A novel erasure coding scheme using fine granularity blocks to improve Hadoop write and update performance