NDLI: A Novel Optimization Method to Improve De-duplication Storage System Performance

Content Provider	IEEE Xplore Digital Library
Author	Chuanyi Liu Yibo Xue Dapeng Ju Dongsheng Wang
Copyright Year	2009
Abstract	Data De-duplication has become a commodity component in data-intensive storage systems. But compared with other traditional storage paradigms, de-duplication system achieves elimination of data duplications or redundancies at the cost of bringing several additional layers or function components into the I/O path, and these additional components are either CPU-intensive or I/O intensive, largely hindering the overall system performance. Direct against the above potential system bottlenecks, this paper quantitatively analyzes the overhead of each main component introduced by de-duplication, and then proposes two performance optimization methods. The one is parallel calculation of content aware chunk identifiers, which fully utilizes the parallelism both inter and intra chunks by using a certain task partition and chunk content distribution algorithm. Experiments demonstrate that it can improve up to 150% of the system throughput, and at the same time much better utilize the multiprocessor resources. The other one is storage pipelining, which overlaps the CPU-bound, I/O-bound and network communication tasks. Through a dedicated five-stage storage pipeline design for file archival operations, experimental results show that the system throughput can increase up to 25% according to our workloads.
Starting Page	228
Ending Page	235
File Size	591195
Page Count	8
File Format	PDF
ISBN	9781424457885
ISSN	15219097
e-ISBN	9781424457892
DOI	10.1109/ICPADS.2009.103
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2009-12-08
Publisher Place	China
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Optimization methods System performance Throughput Cost function Pipeline processing Cryptography Computer science Laboratories Information science Performance analysis Parallel Hash Data De-duplication Performance Optimization Storage Pipeline
Content Type	Text
Resource Type	Article
Subject	Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Evaluation of a hash-compress-encrypt pipeline for storage system applications

A novel class of serial-parallel redundant signed-digit multipliers

A Parallel and Pipelined Architecture for Accelerating Fingerprint Computation in High Throughput Data Storages

A Program Behavior Study of Block Cryptography Algorithms on GPGPU

Multi-weighted tree based query optimization method for parallel relational database systems

Optimal Placement of Pipeline Applications on Grid

Design for high throughput SHA-1 hash function on FPGA

AES crypto chip utilizing high-speed parallel pipelined architecture

Celerity Hardware Implementation of the AES with Data Parallel and Pipelining Architecture inside the Round Function

A Novel Optimization Method to Improve De-duplication Storage System Performance

Similar Documents

Evaluation of a hash-compress-encrypt pipeline for storage system applications

A novel class of serial-parallel redundant signed-digit multipliers

A Parallel and Pipelined Architecture for Accelerating Fingerprint Computation in High Throughput Data Storages

A Program Behavior Study of Block Cryptography Algorithms on GPGPU

Multi-weighted tree based query optimization method for parallel relational database systems

Optimal Placement of Pipeline Applications on Grid

Design for high throughput SHA-1 hash function on FPGA

AES crypto chip utilizing high-speed parallel pipelined architecture

Celerity Hardware Implementation of the AES with Data Parallel and Pipelining Architecture inside the Round Function

A Novel Optimization Method to Improve De-duplication Storage System Performance