NDLI: Leveraging hadoop framework to develop duplication detector and analysis using Mapreduce, Hive and Pig

Content Provider	IEEE Xplore Digital Library
Author	Sethi, P. Kumar, P.
Copyright Year	2014
Description	Author affiliation: Dept. of Comput. Sci. Eng. & Inf. Technol., Jaypee Inst. of Inf. Technol., Noida, India (Sethi, P.; Kumar, P.)
Abstract	The burgeoning volume of torrential data continues to grow exponentially in this very age of the Internet of Things. As this torrent of digital datasets continue to outgrow in datacenters, the focus needs to be shifted to stored data reduction methods and that too pertaining to NoSQL databases as traditional structured storage systems continuously tend to face challenges in providing the required storage, throughputs and computational power requirements necessary to capture, store, manage and analyze the deluge of data. Deduplication systems, thus designed, retain a single copy of redundant data on disk to save disk space, but what if we want to keep certain copies intentionally and need wishful elimination. This paper leverages Hadoop framework to design and develop a duplication detection system that detects multiple copies of the same data right at the file level itself and that too before transmission. Thereafter, various datasets are tuned for better performance and analysed using MapReduce, Hive and Pig.
Sponsorship	IEEE Comput. Soc.
Starting Page	454
Ending Page	460
File Size	587767
Page Count	7
File Format	PDF
ISBN	9781479951727
e-ISBN	9781479951734
DOI	10.1109/IC3.2014.6897216
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-08-07
Publisher Place	India
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Hive NoSQL Random access memory Hadoop Deduplication HDFS MongoDB Cryptography Pig MapReduce HBase
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Chapter 11 – MapReduce , Hadoop , HDFS , Hbase , MongoDB , Apache HIVE , and Related

Color and Texture Feature Extraction Using Apache Hadoop Framework

Performance evaluation of a MongoDB and hadoop platform for scientific data analysis

Modified Fuzzy K-mean clustering using MapReduce in Hadoop and cloud

Enhancing performance of Hadoop and MapReduce for scientific data using NoSQL database

SETiNS: Storage efficiency techniques in No-SQL database for Cloud based design

Security maturity in NoSQL databases - are they secure enough to haul the modern IT applications?

Implementation and evaluation of scalable data structure over HBase

ATLAS Data Management Accounting with Hadoop Pig and HBase

Leveraging hadoop framework to develop duplication detector and analysis using Mapreduce, Hive and Pig

Similar Documents

Chapter 11 – MapReduce , Hadoop , HDFS , Hbase , MongoDB , Apache HIVE , and Related

Color and Texture Feature Extraction Using Apache Hadoop Framework

Performance evaluation of a MongoDB and hadoop platform for scientific data analysis

Modified Fuzzy K-mean clustering using MapReduce in Hadoop and cloud

Enhancing performance of Hadoop and MapReduce for scientific data using NoSQL database

SETiNS: Storage efficiency techniques in No-SQL database for Cloud based design

Security maturity in NoSQL databases - are they secure enough to haul the modern IT applications?

Implementation and evaluation of scalable data structure over HBase

￼ATLAS Data Management Accounting with Hadoop Pig and HBase

Leveraging hadoop framework to develop duplication detector and analysis using Mapreduce, Hive and Pig

ATLAS Data Management Accounting with Hadoop Pig and HBase