NDLI: Dynamic load balancing for I/O-intensive applications on clusters

Please wait, while we are loading the content...

Dynamic load balancing for I/O-intensive applications on clusters

Content Provider	ACM Digital Library
Author	Manzanares, Adam Ruan, Xiaojun Qin, Xiao Jiang, Hong Yin, Shu
Copyright Year	2009
Abstract	Load balancing for clusters has been investigated extensively, mainly focusing on the effective usage of global CPU and memory resources. However, previous CPU- or memory-centric load balancing schemes suffer significant performance drop under I/O-intensive workloads due to the imbalance of I/O load. To solve this problem, we propose two simple yet effective I/O-aware load-balancing schemes for two types of clusters: (1) homogeneous clusters where nodes are identical and (2) heterogeneous clusters, which are comprised of a variety of nodes with different performance characteristics in computing power, memory capacity, and disk speed. In addition to assigning I/O-intensive sequential and parallel jobs to nodes with light I/O loads, the proposed schemes judiciously take into account both CPU and memory load sharing in the system. Therefore, our schemes are able to maintain high performance for a wide spectrum of workloads. We develop analytic models to study mean slowdowns, task arrival, and transfer processes in system levels. Using a set of real I/O-intensive parallel applications and synthetic parallel jobs with various I/O characteristics, we show that our proposed schemes consistently improve the performance over existing non-I/O-aware load-balancing schemes, including CPU- and Memory-aware schemes and a PBS-like batch scheduler for parallel and sequential jobs, for a diverse set of workload conditions. Importantly, this performance improvement becomes much more pronounced when the applications are I/O-intensive. For example, the proposed approaches deliver 23.6--88.0 % performance improvements for I/O-intensive applications such as LU decomposition, Sparse Cholesky, Titan, Parallel text searching, and Data Mining. When I/O load is low or well balanced, the proposed schemes are capable of maintaining the same level of performance as the existing non-I/O-aware schemes.
Starting Page	1
Ending Page	38
Page Count	38
File Format	PDF
ISSN	15533077
e-ISSN	15533093
DOI	10.1145/1629075.1629078
Volume Number	5
Issue Number	3
Journal	ACM Transactions on Storage (TOS)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2009-11-30
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	I/O-intensive applications Load balancing Clusters Heterogeneity Storage systems
Content Type	Text
Resource Type	Article
Subject	Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Dynamic load balancing for I/O-intensive applications on clusters (2009)

Dynamic load balancing for I/O-intensive tasks on heterogeneous clusters

Dynamic Load Balancing for I/O-Intensive Tasks on Heterogeneous Clusters (2003)

Hardware Fault Recovery for I/O Intensive Applications

An intelligent dynamic load balancer for workstation clusters

Dynamic Load Balancing for I/O-Intensive Applications on Clusters (2009)

Prediction Based Dynamic Load Balancing Techniques in Heterogeneous Clusters

Tunable randomization for load management in shared-disk clusters

Dynamic load balancing for I/O-intensive applications on clusters (2009)

Dynamic load balancing for I/O-intensive applications on clusters

Similar Documents

Dynamic load balancing for I/O-intensive applications on clusters (2009)

Dynamic load balancing for I/O-intensive tasks on heterogeneous clusters

Dynamic Load Balancing for I/O-Intensive Tasks on Heterogeneous Clusters (2003)

Hardware Fault Recovery for I/O Intensive Applications

An intelligent dynamic load balancer for workstation clusters

Dynamic Load Balancing for I/O-Intensive Applications on Clusters (2009)

Prediction Based Dynamic Load Balancing Techniques in Heterogeneous Clusters

Tunable randomization for load management in shared-disk clusters

Dynamic load balancing for I/O-intensive applications on clusters (2009)

Dynamic load balancing for I/O-intensive applications on clusters