NDLI: Federated MapReduce to Transparently Run Applications on Multicluster Environment

Content Provider	IEEE Xplore Digital Library
Author	Chun-Yu Wang Tzu-Li Tai Shu Jui-Shing Jyh-Biau Chang Ce-Kuen Shieh
Copyright Year	2014
Description	Author affiliation: Dept. of Digital Applic., Univ. of Kang Ning, Tainan, Taiwan (Jyh-Biau Chang) \|\| Dept. of Electr. Eng., Nat. Cheng Kung Univ., Tainan, Taiwan (Chun-Yu Wang; Tzu-Li Tai; Shu Jui-Shing; Ce-Kuen Shieh)
Abstract	In the Cloud era, data is generated everywhere, how to efficiently analyze those "Big Data" that have properties such as large volume, fast generation, and variety, are most critical issues. MapReduce is a simplified distributed parallel data processing model. It has been widely applied in many areas such as web indexing, clustering and classification. However, when it confronted the sensitive data, such as network log or mails, which are distributed among independent organizations, these data must keep privacy and cannot be aggregated for centralized analyzing. We propose Federated MapReduce (Fed-MR), a framework aimed at analyzing geometrically distributed data among independent organizations while avoiding data movement. In contrast to previous works, Fed-MR retains the simplicity of MapReduce programming eto provide a transparent way to run original MapReduce jobs across multiple clusters without any extra programming burden. Fed-MR also integrates multiple clusters in different locations to form hierarchical Top-Region relationships. Experiments, compared to a single cluster with the same number of worker nodes, had shown that the computation time was only increased by an average of 30% in WordCount and 10% in Grep. Therefore, Fed-MR has reasonable overheads in performance for analyzing data across Internet-connected clusters while no additional Global Reduce function was required as in traditional hierarchical MapReduce frameworks.
Starting Page	296
Ending Page	303
File Size	4080575
Page Count	8
File Format	PDF
e-ISBN	9781479950577
DOI	10.1109/BigData.Congress.2014.50
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-06-27
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Cloud computing File systems Distributed databases Collaboration Hadoop Programming Hierarchical MapReduce Multicluster Data models
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

FedLoop: Looping on Federated MapReduce

The Application of MapReduce in the Cloud Computing

Performance analysis of MapReduce programs on Hadoop cluster

Costs of a federated and hybrid cloud environment aimed at MapReduce video transcoding

Multicluster Hadoop Distributed File System

NORA: Network Oriented Resource Allocation for Data Intensive Applications in the Cloud Environment

Research on Hadoop Cloud Computing Model and its Applications

Implementation of MapReduce-based image conversion module in cloud computing environment

Application Performance Analysis of Distributed File Systems under Cloud Computing Environment

Federated MapReduce to Transparently Run Applications on Multicluster Environment

Similar Documents

FedLoop: Looping on Federated MapReduce

The Application of MapReduce in the Cloud Computing

Performance analysis of MapReduce programs on Hadoop cluster

Costs of a federated and hybrid cloud environment aimed at MapReduce video transcoding

Multicluster Hadoop Distributed File System

NORA: Network Oriented Resource Allocation for Data Intensive Applications in the Cloud Environment

Research on Hadoop Cloud Computing Model and its Applications

Implementation of MapReduce-based image conversion module in cloud computing environment

Application Performance Analysis of Distributed File Systems under Cloud Computing Environment

Federated MapReduce to Transparently Run Applications on Multicluster Environment