NDLI: Provisioning and Evaluating Multi-domain Networked Clouds for Hadoop-based Applications

Content Provider	IEEE Xplore Digital Library
Author	Mandal, A. Yufeng Xin Baldine, I. Ruth, P. Heerman, C. Chase, J. Orlikowski, V. Yumerefendi, A.
Copyright Year	2011
Abstract	This paper presents the design, implementation, and evaluation of a new system for on-demand provisioning of Hadoop clusters across multiple cloud domains. The Hadoop clusters are created "on-demand" and are composed of virtual machines from multiple cloud sites linked with bandwidth-provisioned network pipes. The prototype uses an existing federated cloud control framework called Open Resource Control Architecture (ORCA), which orchestrates the leasing and configuration of virtual infrastructure from multiple autonomous cloud sites and network providers. ORCA enables computational and network resources from multiple clouds and network substrates to be aggregated into a single virtual "slice" of resources, built to order for the needs of the application. The experiments examine various provisioning alternatives by evaluating the performance of representative Hadoop benchmarks and applications on resource topologies with varying bandwidths. The evaluations examine conditions in which multi-cloud Hadoop deployments pose significant advantages or disadvantages during Map/Reduce/Shuffle operations. Further, the experiments compare multi-cloud Hadoop deployments with single-cloud deployments and investigate Hadoop Distributed File System (HDFS) performance under varying network configurations. The results show that networked clouds make cross-cloud Hadoop deployment feasible with high bandwidth network links between clouds. As expected, performance for some benchmarks degrades rapidly with constrained inter-cloud bandwidth. MapReduce shuffle patterns and certain Hadoop Distributed File System (HDFS) operations that span the constrained links are particularly sensitive to network performance. Hadoop's topology-awareness feature can mitigate these penalties to a modest degree in these hybrid bandwidth scenarios. Additional observations show that contention among co-located virtual machines is a source of irregular performance for Hadoop applications on virtual cloud infrastructure.
Starting Page	690
Ending Page	697
File Size	173418
Page Count	8
File Format	PDF
ISBN	9781467300902
DOI	10.1109/CloudCom.2011.107
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2011-11-29
Publisher Place	Greece
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Performance analysis of Hadoop/MapReduce Network topology Prototypes Bandwidth Network provisioning Benchmark testing Educational institutions Virtual machining Topology Multi-cloud provisioning Networked clouds
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Elastic provisioning of virtual Hadoop clusters in OpenStack-based Clouds

Reducing the Power Consumption of Servers with Bandwidth Consideration

vHadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Initial Findings for Provisioning Variation in Cloud Computing

Can MPI Benefit Hadoop and MapReduce Applications?

Resilin: Elastic MapReduce over Multiple Clouds

Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic

Storage Management in Virtualized Cloud Environment

A deep investigation into network performance in virtual machine based cloud environments

Provisioning and Evaluating Multi-domain Networked Clouds for Hadoop-based Applications

Similar Documents

Elastic provisioning of virtual Hadoop clusters in OpenStack-based Clouds

Reducing the Power Consumption of Servers with Bandwidth Consideration

vHadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Initial Findings for Provisioning Variation in Cloud Computing

Can MPI Benefit Hadoop and MapReduce Applications?

Resilin: Elastic MapReduce over Multiple Clouds

Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic

Storage Management in Virtualized Cloud Environment

A deep investigation into network performance in virtual machine based cloud environments

Provisioning and Evaluating Multi-domain Networked Clouds for Hadoop-based Applications