NDLI: Making work queue cluster-friendly for data intensive scientific applications

Content Provider	IEEE Xplore Digital Library
Author	Albrecht, M. Rajan, D. Thain, D.
Copyright Year	2013
Description	Author affiliation: Dept. of Comput. Sci. & Eng., Univ. of Notre Dame, Notre Dame, IN, USA (Albrecht, M.; Rajan, D.; Thain, D.)
Abstract	Researchers with large-scale data-intensive applications often wish to scale up applications to run on multiple clusters, employing a middleware layer for resource management across clusters. However, at the very largest scales, such middleware is often “unfriendly” to individual clusters, which are usually designed to support communication within the cluster, not outside of it. To address this problem we have modified the Work Queue master-worker application framework to support a hierarchical configuration that more closely matches the physical architecture of existing clusters. Using a synthetic application we explore the properties of the system and evaluate its performance under multiple configurations, with varying worker reliability, network capabilities, and data requirements. We show that by matching the software and hardware architectures more closely we can gain both a modest improvement in runtime and a dramatic reduction in network footprint at the master. We then run a scalable molecular dynamics application (AWE) to examine the impact of hierarchy on performance, cost and efficiency for real scientific applications and see a 96% reduction in network footprint, making it much more palatable to system operators and opening the possibility of increasing the application scale by another order of magnitude or more.
Sponsorship	IEEE Comput. Soc.
Starting Page	1
Ending Page	8
File Size	354500
Page Count	8
File Format	PDF
ISBN	9781479908981
DOI	10.1109/CLUSTER.2013.6702628
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2013-09-23
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Bandwidth Computer architecture Data transfer Resource management Reliability Runtime Hierarchical systems
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Making work queue cluster-friendly for data intensive scientific applications.

Intelligent architecture for automatic resource allocation in computer clusters

Scientific Computing Autonomic Reliability Framework

Research of Grid Platform Oriented to Intensive Data Processing

A Runtime Resource-aware Architecture for Service-oriented Embedded Systems

A Data Placement Strategy for Data-Intensive Scientific Workflows in Cloud

The Trident Scientific Workflow Workbench

Protocols for wide-area data-intensive applications: Design and performance issues

GridMate: A Portable Simulation Environment for Large-Scale Adaptive Scientific Applications

Making work queue cluster-friendly for data intensive scientific applications

Similar Documents

Making work queue cluster-friendly for data intensive scientific applications.

Intelligent architecture for automatic resource allocation in computer clusters

Scientific Computing Autonomic Reliability Framework

Research of Grid Platform Oriented to Intensive Data Processing

A Runtime Resource-aware Architecture for Service-oriented Embedded Systems

A Data Placement Strategy for Data-Intensive Scientific Workflows in Cloud

The Trident Scientific Workflow Workbench

Protocols for wide-area data-intensive applications: Design and performance issues

GridMate: A Portable Simulation Environment for Large-Scale Adaptive Scientific Applications

Making work queue cluster-friendly for data intensive scientific applications