NDLI: Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics

Content Provider	IEEE Xplore Digital Library
Author	Dayal, J. Bratcher, D. Eisenhauer, G. Schwan, K. Wolf, M. Xuechen Zhang Abbasi, H. Klasky, S. Podhorszki, N.
Copyright Year	2014
Description	Author affiliation: Oak Ridge Nat. Labs., Oak Ridge, TN, USA (Abbasi, H.; Klasky, S.; Podhorszki, N.) \|\| Georgia Inst. of Technol., Atlanta, GA, USA (Dayal, J.; Bratcher, D.; Eisenhauer, G.; Schwan, K.; Wolf, M.; Xuechen Zhang)
Abstract	As high-end systems move toward exascale sizes, a new model of scientific inquiry being developed is one in which online data analytics run concurrently with the high end simulations producing data outputs. Goals are to gain rapid insights into the ongoing scientific processes, assess their scientific validity, and/or initiate corrective or supplementary actions by launching additional computations when needed. The Flexpath system presented in this paper addresses the fundamental problem of how to structure and efficiently implement the communications between high end simulations and concurrently running online data analytics, the latter comprised of componentized dynamic services and service pipelines. Using a type-based publish/subscribe approach, Flexpath encourages diversity by permitting analytics services to differ in their computational and scaling characteristics and even in their internal execution models. Flex path uses direct and MxN connections between interacting services to reduce data movements, to allow for runtime connectivity changes to accommodate component arrivals/departures, and to support the multiple underlying communication protocols used for analytics workflows in which simulation outputs are processed by analytics services residing on the same nodes where they are generated, on the same machine, and/or on attached or remote analytics engines. This paper describes the design and implementation of Flexpath, and evaluates it with two widely used scientific applications and their associated data analytics methods.
Starting Page	246
Ending Page	255
File Size	1422766
Page Count	10
File Format	PDF
ISBN	9781479927845
DOI	10.1109/CCGrid.2014.104
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-05-26
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Data models Analytical models Runtime Arrays Pipelines Subscriptions Computational modeling Code Coupling Publish/Subscribe Data Staging Data Analytics in-Situ Scalable I/O
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

I/O Containers: Managing the Data Analytics and Visualization Pipelines of High End Codes

Flexpath: Type-Based Publish/Subscribe System for Large-scale Science Analytics

SODA: Science-Driven Orchestration of Data Analytics

FlexIO: I/O Middleware for Location-Flexible Scientific Data Analytics

A scalable messaging system for accelerating discovery from large scale scientific simulations

Sub-2-Sub: Self-Organizing Content-Based Publish Subscribe for Dynamic Large Scale Collaborative Networks (2006)

Runtime System for I/O Staging in Support of In-Situ Processing of Extreme Scale Data

Andes: A Highly Scalable Persistent Messaging System

A formal framework for software development using publish/Subscribe architecture

Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics

Similar Documents

I/O Containers: Managing the Data Analytics and Visualization Pipelines of High End Codes

Flexpath: Type-Based Publish/Subscribe System for Large-scale Science Analytics

SODA: Science-Driven Orchestration of Data Analytics

FlexIO: I/O Middleware for Location-Flexible Scientific Data Analytics

A scalable messaging system for accelerating discovery from large scale scientific simulations

Sub-2-Sub: Self-Organizing Content-Based Publish Subscribe for Dynamic Large Scale Collaborative Networks (2006)

Runtime System for I/O Staging in Support of In-Situ Processing of Extreme Scale Data

Andes: A Highly Scalable Persistent Messaging System

A formal framework for software development using publish/Subscribe architecture

Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics