NDLI: Middleware support for many-task computing

Please wait, while we are loading the content...

Middleware support for many-task computing

Content Provider	Springer Nature Link
Author	Raicu, Ioan Foster, Ian Wilde, Mike Zhang, Zhao Iskra, Kamil Beckman, Peter Zhao, Yong Szalay, Alex Choudhary, Alok Little, Philip Moretti, Christopher Chaudhary, Amitabh Thain, Douglas
Copyright Year	2010
Abstract	Many-task computing aims to bridge the gap between two computing paradigms, high throughput computing and high performance computing. Many-task computing denotes high-performance computations comprising multiple distinct activities, coupled via file system operations. The aggregate number of tasks, quantity of computing, and volumes of data may be extremely large. Traditional techniques found in production systems in the scientific community to support many-task computing do not scale to today’s largest systems, due to issues in local resource manager scalability and granularity, efficient utilization of the raw hardware, long wait queue times, and shared/parallel file system contention and scalability. To address these limitations, we adopted a “top-down” approach to building a middleware called Falkon, to support the most demanding many-task computing applications at the largest scales. Falkon (Fast and Light-weight tasK executiON framework) integrates (1) multi-level scheduling to enable dynamic resource provisioning and minimize wait queue times, (2) a streamlined task dispatcher able to achieve orders-of-magnitude higher task dispatch rates than conventional schedulers, and (3) data diffusion which performs data caching and uses a data-aware scheduler to co-locate computational and storage resources. Micro-benchmarks have shown Falkon to achieve over 15K+ tasks/s throughputs, scale to hundreds of thousands of processors and to millions of queued tasks, and execute billions of tasks per day. Data diffusion has also shown to improve applications scalability and performance, with its ability to achieve hundreds of Gb/s I/O rates on modest sized clusters, with Tb/s I/O rates on the horizon. Falkon has shown orders of magnitude improvements in performance and scalability than traditional approaches to resource management across many diverse workloads and applications at scales of billions of tasks on hundreds of thousands of processors across clusters, specialized systems, Grids, and supercomputers. Falkon’s performance and scalability have enabled a new class of applications called Many-Task Computing to operate at previously so-believed impossible scales with high efficiency.
Starting Page	291
Ending Page	314
Page Count	24
File Format	PDF
ISSN	13867857
Journal	Cluster Computing
Volume Number	13
Issue Number	3
e-ISSN	15737543
Language	English
Publisher	Springer US
Publisher Date	2010-04-16
Publisher Place	Boston
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Many-task computing High-throughput Computing High-performance computing Petascale Loosely-coupled applications Data-intensive distributed computing Falkon Swift Computer Communication Networks Operating Systems Processor Architectures
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

On the role of application and resource characterizations in heterogeneous distributed computing systems

MOMTH: multi-objective scheduling algorithm of many tasks in Hadoop

Loosely coupled coordinated management in virtualized data centers

FastMPJ: a scalable and efficient Java message-passing library

Supporting high‐performance I/O in QoS‐enabled ORB middleware

Sky Computing: exploring the aggregated Cloud resources

Malleable applications for scalable high performance computing

AutoMate: Enabling Autonomic Applications on the Grid

Managing security in high‐performance distributed computations

Middleware support for many-task computing

Similar Documents

On the role of application and resource characterizations in heterogeneous distributed computing systems

MOMTH: multi-objective scheduling algorithm of many tasks in Hadoop

Loosely coupled coordinated management in virtualized data centers

FastMPJ: a scalable and efficient Java message-passing library

Supporting high‐performance I/O in QoS‐enabled ORB middleware

Sky Computing: exploring the aggregated Cloud resources

Malleable applications for scalable high performance computing

AutoMate: Enabling Autonomic Applications on the Grid

Managing security in high‐performance distributed computations

Middleware support for many-task computing