NDLI: Hadoop on Named Data Networking: Experience and Results

Please wait, while we are loading the content...

Delay Scalings and Mean-Field Limits in Networked Systems

A Low-Complexity Approach to Distributed Cooperative Caching with Geographic Constraints

Hieroglyph: Locally-Sufficient Graph Processing via Compute-Sync-Merge

Investigation of the 2016 Linux TCP Stack Vulnerability at Scale

Fluid-Model-Based Car Routing for Modern Ridesharing Systems

Online Optimization for Markets and the Cloud: Theory and Practice

Stein's Method for Mean-Field Approximations in Light and Heavy Traffic Regimes

Understanding Reduced-Voltage Operation in Modern DRAM Devices: Experimental Characterization, Analysis, and Mechanisms

Hadoop on Named Data Networking: Experience and Results

On Gradient-Based Optimization: Accelerated, Distributed, Asynchronous and Stochastic

Portfolio-driven Resource Management for Transient Cloud Servers

Matrix Factorization at the Frontier of Non-convex Optimizations: Abstract for SIGMETRICS 2017 Rising Star Award Talk

Outward Influence and Cascade Size Estimation in Billion-scale Networks

A Case Study in Power Substation Network Dynamics

Routing Money, Not Packets: A Tutorial on Internet Economics

Optimal Service Elasticity in Large-Scale Distributed Systems

A Simple Yet Effective Balanced Edge Partition Model for Parallel Computing

Characterizing and Modeling Patching Practices of Industrial Control Systems

Pseudo-Separation for Assessment of Structural Vulnerability of a Network

Expected Values Estimated via Mean-Field Approximation are 1/N-Accurate: Extended Abstract

Exploiting Data Longevity for Enhancing the Lifetime of Flash-based Storage Class Memory

Using Burstable Instances in the Public Cloud: Why, When and How?

Optimal Posted Prices for Online Cloud Resource Allocation

Accelerating Performance Inference over Closed Systems by Asymptotic Methods

Persistent Spread Measurement for Big Network Data Based on Register Intersection

Queue-Proportional Sampling: A Better Approach to Crossbar Scheduling for Input-Queued Switches

Overcommitment in Cloud Services Bin packing with Chance Constraints

Security Game with Non-additive Utilities and Multiple Attacker Resources

On the Capacity Requirement for Arbitrary End-to-End Deadline and Reliability Guarantees in Multi-hop Networks

Analysis of a Stochastic Model of Replication in Large Distributed Storage Systems: A Mean-Field Approach

Design-Induced Latency Variation in Modern DRAM Chips: Characterization, Analysis, and Latency Reduction Mechanisms

Dandelion: Redesigning the Bitcoin Network for Anonymity

On Optimal Two-Sided Pricing of Congested Networks

Quality and Cost of Deterministic Network Calculus: Design and Evaluation of an Accurate and Fast Analysis

Deconstructing the Energy Consumption of the Mobile Page Load

Optimizing Speculative Execution of Deadline-Sensitive Jobs in Cloud

A Spot Capacity Market to Increase Power Infrastructure Utilization in Multi-Tenant Data Centers

Hour-Ahead Offering Strategies in Electricity Market for Power Producers with Storage and Intermittent Supply

Why "Some" Like It Hot Too: Thermal Attack on Data Centers

Incentivizing Reliable Demand Response with Customers' Uncertainties and Capacity Planning

A Study on Performance and Power Efficiency of Dense Non-Volatile Caches in Multi-Core Systems

Scheduling Coflows in Datacenter Networks: Improved Bound for Total Weighted Completion Time

Characterizing 3D Floating Gate NAND Flash

ECF: An MPTCP Path Scheduler to Manage Heterogeneous Paths

Simplex Queues for Hot-Data Download

An Empirical Analysis of Facebook's Free Basics

Multipath TCP on a VANET: A Performance Study

A Fast, Small, and Dynamic Forwarding Information Base

HFTraC: High-Frequency Traffic Control

Adaptive TTL-Based Caching for Content Delivery

Hadoop on Named Data Networking: Experience and Results

Content Provider	ACM Digital Library
Author	Gibbens, Mathias Zhang, Beichuan Gniady, Chris Ye, Lei
Abstract	In today's data centers, clusters of servers are arranged to perform various tasks in a massively distributed manner: handling web requests, processing scientific data, and running simulations of real-world problems. These clusters are very complex, and require a significant amount of planning and administration to ensure that they perform to their maximum potential. Planning and configuration can be a long and complicated process; once completed it is hard to completely re-architect an existing cluster. In addition to planning the physical hardware, the software must also be properly configured to run on a cluster. Information such as which server is in which rack and the total network bandwidth between rows of racks constrain the placement of jobs scheduled to run on a cluster. Some software may be able to use hints provided by a user about where to schedule jobs, while others may simply place them randomly and hope for the best. Every cluster has at least one bottleneck that constrains the overall performance to less than the optimal that may be achieved on paper. One common bottleneck is the speed of the network: communication between servers in a rack may be unable to saturate their network connections, but traffic flowing between racks or rows in a data center can easily overwhelm the interconnect switches. Various network topologies have been proposed to help mitigate this problem by providing multiple paths between points in the network, but they all suffer from the same fundamental problem: it is cost-prohibitive to build a network that can provide concurrent full network bandwidth between all servers. Researchers have been working on developing new network protocols that can make more efficient use of existing network hardware through a blurring of the line between network layer and applications. One of the most well-known examples of this is Named Data Networking (NDN), a data-centric network architecture that has been in development for several years. While NDN has received significant attention for wide-area Internet, a detailed understanding of NDN benefits and challenges in the data center environment has been lacking. The Named Data Networking architecture retrieves content by names rather than connecting to specific hosts. It provides benefits such as highly efficient and resilient content distribution, which fit well to data-intensive distributed computing. This paper presents and discusses our experience in modifying Apache Hadoop, a popular MapReduce framework, to operate on an NDN network. Through this first-of-its-kind implementation process, we demonstrate the feasibility of running an existing, large, and complex piece of distributed software commonly seen in data centers over NDN. We show advantages such as simplified network code and reduced network traffic, which are beneficial in a data center environment. There are also challenges faced by NDN that are being addressed by the community, which can be magnified under data center traffic. Through detailed evaluation, we show a reduction of 16% for overall data transmission between Hadoop nodes while writing data with default replication settings. Preliminary results also show promise for in-network caching of repeated reads in distributed applications. We show that while overall performance is currently slower under NDN, there are challenges and opportunities for further NDN improvements.
Starting Page	55
Ending Page	55
Page Count	1
File Format	PDF
ISBN	9781450350327
DOI	10.1145/3078505.3078508
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2017-06-05
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Data centers Named data networking Emerging technologies Hadoop Large-scale systems
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Toward terabyte-scale caching with SSD in a named data networking router

Named Data Networking for Genomics Data Management and Integrated Workflows

Hadoop-HBase for large-scale data

Large Scale Distributed Data Science using Apache Spark

Named data networking on a router: forwarding at 20gbps and beyond

Schematizing Trust in Named Data Networking

Receiver Mobility in Vehicular Named Data Networking

Large scale cross-media data retrieval based on Hadoop

Sharing mHealth Data via Named Data Networking

Hadoop on Named Data Networking: Experience and Results

Similar Documents

Toward terabyte-scale caching with SSD in a named data networking router

Named Data Networking for Genomics Data Management and Integrated Workflows

Hadoop-HBase for large-scale data

Large Scale Distributed Data Science using Apache Spark

Named data networking on a router: forwarding at 20gbps and beyond

Schematizing Trust in Named Data Networking

Receiver Mobility in Vehicular Named Data Networking

Large scale cross-media data retrieval based on Hadoop

Sharing mHealth Data via Named Data Networking

Hadoop on Named Data Networking: Experience and Results