NDLI: T-Storm: Traffic-Aware Online Scheduling in Storm

Content Provider	IEEE Xplore Digital Library
Author	Jielong Xu Zhenhua Chen Jian Tang Sen Su
Copyright Year	2014
Description	Author affiliation: Dept. of Electr. Eng. & Comput. Sci., Syracuse Univ., Syracuse, NY, USA (Jielong Xu; Zhenhua Chen; Jian Tang) \|\| State Key Lab. of Networking & Switching, Beijing Univ. of Posts & Telecommun., Beijing, China (Sen Su)
Abstract	Storm has emerged as a promising computation platform for stream data processing. In this paper, we first show inefficiencies of the current practice of Storm scheduling and challenges associated with applying traffic-aware online scheduling in Storm via experimental results and analysis. Motivated by our observations, we design and implement a new stream data processing system based on Storm, namely, T-Storm. Compared to Storm, T-Storm has the following desirable features: 1) based on runtime states, it accelerates data processing by leveraging effective traffic-aware scheduling for assigning/re-assigning tasks dynamically, which minimizes inter-node and inter-process traffic while ensuring no worker nodes are overloaded, 2) it enables fine-grained control over worker node consolidation such that T-Storm can achieve better performance with even fewer worker nodes, 3) it allows hot-swapping of scheduling algorithms and adjustment of scheduling parameters on the fly, and 4) it is transparent to Storm users (i.e., Storm applications can be ported to run on T-Storm without any changes). We conducted real experiments in a cluster using well-known data processing applications for performance evaluation. Extensive experimental results show that compared to Storm (with the default scheduler), T-Storm can achieve over 84% and 27% speedup on lightly and heavily loaded topologies respectively (in terms of average processing time) with 30% less number of worker nodes.
Starting Page	535
Ending Page	544
File Size	4945576
Page Count	10
File Format	PDF
ISSN	10636927
e-ISBN	9781479951697
DOI	10.1109/ICDCS.2014.61
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-06-30
Publisher Place	Spain
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Storms Topology Fasteners Scheduling Data processing Monitoring Schedules Resource Management Big Data Stream Data Processing Storm
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Hardware and Architecture Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Scalable Traffic-Aware Virtual Machine Management for Cloud Data Centers

On QoS-aware scheduling of data stream applications over fog computing infrastructures

Longer Is Better: Exploiting Path Diversity in Data Center Networks

An Application-Aware Scheduling Policy for Real-Time Traffic

Online stream processing of machine-to-machine communications traffic: A platform comparison

R-Storm: Resource-Aware Scheduling in Storm

Scheduling Decisions in Stream Processing on Heterogeneous Clusters

PRISM: Fine-Grained Resource-Aware Scheduling for MapReduce

Elastic Stream Processing with Latency Guarantees

T-Storm: Traffic-Aware Online Scheduling in Storm

Similar Documents

Scalable Traffic-Aware Virtual Machine Management for Cloud Data Centers

On QoS-aware scheduling of data stream applications over fog computing infrastructures

Longer Is Better: Exploiting Path Diversity in Data Center Networks

An Application-Aware Scheduling Policy for Real-Time Traffic

Online stream processing of machine-to-machine communications traffic: A platform comparison

R-Storm: Resource-Aware Scheduling in Storm

Scheduling Decisions in Stream Processing on Heterogeneous Clusters

PRISM: Fine-Grained Resource-Aware Scheduling for MapReduce

Elastic Stream Processing with Latency Guarantees

T-Storm: Traffic-Aware Online Scheduling in Storm