NDLI: MDHT: a multi-level-indexed DHT algorithm to extra-large-scale data retrieval on HDFS/Hadoop architecture

Content Provider	Springer Nature Link
Author	Tang, Yu Fan, Aihua Wang, Yingjie Yao, Yuanzhe
Copyright Year	2014
Abstract	Corresponding to the storing and fast searching needs of an extra-large scale of energy monitoring and statistics data, we propose a multi-level-indexed distributed hash table (mDHT) algorithm and complete a MapReduce implementation of the algorithm on the open-standard HDFS/Hbase platform. Such an approach uses a columnar storage structure for energy consumption data storage and creates a hashed index table to provide a quick search and retrieval method for extra-large-scale data processing systems. Such a hashed indexing scheme is implemented on a 3-node Hadoop cluster, and the simulation experiments at a scale up to 48 million data records indicate that, when the data volume reaches the scale of 12 million to 48 millions, the proposed mDHT algorithm presents an outstanding performance in data writing operation, compared to that of traditional SQL Server implementation. Even compared to the single-indexed DHT (sDHT) application, the mDHT solution outperforms by reducing the data retrieval time by 24.5–48.6 %. The multi-level-indexed DHT algorithm presented in this paper contributes a key technique to developing a fast search engine to the extra-large scale of data on the cloud storage architecture.
Starting Page	1835
Ending Page	1844
Page Count	10
File Format	PDF
ISSN	16174909
Journal	Personal Technologies
Volume Number	18
Issue Number	8
e-ISSN	16174917
Language	English
Publisher	Springer London
Publisher Date	2014-08-09
Publisher Place	London
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Data retrieval Distributed hash table Cloud storage Multi-level index Algorithm User Interfaces and Human Computer Interaction Computer Science Personal Computing
Content Type	Text
Resource Type	Article
Subject	Management Science and Operations Research Library and Information Sciences Computer Science Applications Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

RPR: recommendation for passengers by roads based on cloud computing and taxis traces data

A video cloud platform combing online and offline cloud computing technologies

A read-only distributed hash table (2007).

Automating the Hadoop configuration for easy setup in resilient cloud systems

Towards a scalable HDFS architecture

Data Currency in Replicated DHTs

FARM: file annotation and retrieval on mobile devices

Multi-space Distributed Hash Tables for Multiple Transport Domains (2003)

A Random Structure for Optimum Cache Size Distributed hash table (DHT) Peer-to-Peer design

MDHT: a multi-level-indexed DHT algorithm to extra-large-scale data retrieval on HDFS/Hadoop architecture

Similar Documents

RPR: recommendation for passengers by roads based on cloud computing and taxis traces data

A video cloud platform combing online and offline cloud computing technologies

A read-only distributed hash table (2007).

Automating the Hadoop configuration for easy setup in resilient cloud systems

Towards a scalable HDFS architecture

Data Currency in Replicated DHTs

FARM: file annotation and retrieval on mobile devices

Multi-space Distributed Hash Tables for Multiple Transport Domains (2003)

A Random Structure for Optimum Cache Size Distributed hash table (DHT) Peer-to-Peer design

MDHT: a multi-level-indexed DHT algorithm to extra-large-scale data retrieval on HDFS/Hadoop architecture