NDLI: Towards Comprehensive Traffic Forecasting in Cloud Computing: Design and Application

Content Provider	ACM Digital Library
Author	Ma, Zhiqiang Yanhui Geng Zhiqiang Ma Wei Bai Hao Wang Yangming Zhao Bai, Wei Gu, Lin Wang, Guohui Zhao, Yangming Lin Gu Chen, Kai Geng, Yanhui Peng, Yang Yang Peng Guohui Wang Wang, Hao Kai Chen
Description	Author Affiliation: Facebook, USA(University of electronic science and technology of china, China (Zhao, Yangming; Huawei noah's ark lab., Hong Kong, China (Yanhui Geng); Lin Gu); Zhiqiang Ma; Wei Bai; Kai Chen; Hong kong univ. of sci. & technol., Hong Kong, China (Yang Peng; Facebook, New York, NY, USA (Guohui Wang); Huawei noah's ark lab, Hong Kong (Geng, Yanhui); Shanghai jiao tong university, China (Wang, Hao); Gu, Lin); Ma, Zhiqiang; Bai, Wei; Chen, Kai; Hong kong university of science and technology, Hong Kong (Peng, Yang; Shanghai jiao tong univ., Shanghai, China (Hao Wang); Univ. of electron. sci. & technol. of china, Chengdu, China (Yangming Zhao); Wang, Guohui))
Abstract	In this paper, we present our effort towards comprehensive traffic forecasting for big data applications using external, light-weighted file system monitoring. Our idea is motivated by the key observations that rich traffic demand information already exists in the log and meta-data files of many big data applications, and that such information can be readily extracted through run-time file system monitoring. As the first step, we use Hadoop as a concrete example to explore our methodology and develop a system called HadoopWatch to predict traffic demands of Hadoop applications. We further implement HadoopWatch in a small-scale testbed with 10 physical servers and 30 virtual machines. Our experiments over a series of MapReduce applications demonstrate that HadoopWatch can forecast the traffic demand with almost 100% accuracy and time advance. Furthermore, it makes no modification on the Hadoop framework, and introduces little overhead to the application performance. Finally, to showcase the utility of accurate traffic prediction made by HadoopWatch, we design and implement a simple HadoopWatch-enabled network optimization module into the HadoopWatch controller, and with realistic Hadoop job benchmarks we find that even a simple algorithm can leverage the forecasting results provided by HadoopWatch to significantly improve the Hadoop job completion time by up to 14.72%.
Starting Page	2210
Ending Page	2222
Page Count	13
File Format	PDF
ISSN	10636692
DOI	10.1109/TNET.2015.2458892
Volume Number	24
Issue Number	4
Journal	IEEE/ACM Transactions on Networking (TON)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2016-08-01
Access Restriction	One Nation One Subscription (ONOS)
Content Type	Text
Resource Type	Article
Subject	Electrical and Electronic Engineering Computer Networks and Communications Software Computer Science Applications

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Cloud Computing Roundtable

Migration towards cloud-assisted live media streaming

Towards cloud-ready transport networks

Software defined networking opportunities for transport

Design of a fuzzy traffic controller for ATM networks

An Online Auction Framework for Dynamic Resource Provisioning in Cloud Computing

Multimedia communications for users

A Survey of Mobile Cloud Computing Applications: Perspectives and Challenges

Design and implementation series VIII: cloud computing and IMS-switched wireless sensor networks

Towards Comprehensive Traffic Forecasting in Cloud Computing: Design and Application

Similar Documents

Cloud Computing Roundtable

Migration towards cloud-assisted live media streaming

Towards cloud-ready transport networks

Software defined networking opportunities for transport

Design of a fuzzy traffic controller for ATM networks

An Online Auction Framework for Dynamic Resource Provisioning in Cloud Computing

Multimedia communications for users

A Survey of Mobile Cloud Computing Applications: Perspectives and Challenges

Design and implementation series VIII: cloud computing and IMS-switched wireless sensor networks

Towards Comprehensive Traffic Forecasting in Cloud Computing: Design and Application