NDLI: HotROD: Managing grid storage with on-demand replication

Content Provider	IEEE Xplore Digital Library
Author	Rao, S. Reed, B. Silberstein, A.
Copyright Year	2013
Description	Author affiliation: Microsoft Corp., Redmond, WA, USA (Rao, S.) \|\| Osmeta Inc., USA (Reed, B.) \|\| Trifacta Inc., USA (Silberstein, A.)
Abstract	Enterprises (such as, Yahoo!, LinkedIn, Facebook) operate their own compute/storage infrastructure, which is effectively a “private cloud”. The private cloud consists of multiple clusters, each of which is managed independently. With HDFS, whenever data is stored in the cluster, it is replicated within the cluster for availability. Unfortunately, for datasets shared across the enterprise, this leads to the problem of over-replication within the private cloud. An analysis of Yahoo!'s HDFS usage suggests that the disk space consumed due to replication of shared datasets is substantial (viz., to the tune of PB's of storage). New data sets are typically popular and requested by many processing jobs in (different) clusters. This demand is satisfied by copying the dataset to each of the clusters. As data sets age, however, they get used less and become cold. We then have the opposite problem of having data overreplicated across clusters: each cluster has enough replicas to recover from data loss locally, and the sum total of replicas is high. We address both the problems of initially replicating data and cross cluster recovery in a private cloud setting using the same technique: on-demand replication, which we refer to as Hot Replication On-Demand (HotROD). By making files visible across HDFS clusters, we let a cluster pull in remote replicas as needed, both for initial replication and later recovery. We implemented HotROD as an extension to a standard HDFS installation.
Starting Page	243
Ending Page	249
File Size	219492
Page Count	7
File Format	PDF
ISBN	9781467353038
e-ISBN	9781467353045
e-ISBN	9781467353021
DOI	10.1109/ICDEW.2013.6547458
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2013-04-08
Publisher Place	Australia
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Cloud computing LinkedIn Databases Standards organizations Organizations Servers Facebook
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

HotROD: Managing Grid Storage With On-Demand Replication

Methods for maintaining security on cloud of stored data

Energy-efficient data replication in cloud computing datacenters

Models for efficient data replication in cloud computing datacenters

Dual direction load balancing and partial replication storage of cloud DaaS

Research on the strategy of FLDC replication dynamically created in cloud storage

Reliable and Available Data Replication Planning for Cloud Storage

A cloud service integration platform for web applications

A survey on cloud computing

HotROD: Managing grid storage with on-demand replication

Similar Documents

HotROD: Managing Grid Storage With On-Demand Replication

Methods for maintaining security on cloud of stored data

Energy-efficient data replication in cloud computing datacenters

Models for efficient data replication in cloud computing datacenters

Dual direction load balancing and partial replication storage of cloud DaaS

Research on the strategy of FLDC replication dynamically created in cloud storage

Reliable and Available Data Replication Planning for Cloud Storage

A cloud service integration platform for web applications

A survey on cloud computing

HotROD: Managing grid storage with on-demand replication