NDLI: Combining in-situ and in-transit processing to enable extreme-scale scientific analysis

Content Provider	IEEE Xplore Digital Library
Author	Bennett, J.C. Abbasi, H. Bremer, P.-T. Grout, R. Gyulassy, A. Tong Jin Klasky, S. Kolla, H. Parashar, M. Pascucci, V. Pebay, P. Thompson, D. Hongfeng Yu Fan Zhang Chen, J.
Copyright Year	2012
Abstract	With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations to a set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution of various analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. We demonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.
Starting Page	1
Ending Page	9
File Size	2483341
Page Count	9
File Format	PDF
ISBN	9781467308052
ISSN	21674337
e-ISBN	9781467308069
DOI	10.1109/SC.2012.31
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2012-11-10
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Computational modeling Analytical models Data models Data visualization Algorithm design and analysis Data transfer Processor scheduling
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Combining in-situ and in-transit processing to enable extreme-scale scientific analysis

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis.

An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis

A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations

Analytical modeling and optimization for affinity based thread scheduling on multicore systems

Performance evaluation of on-line scheduling algorithms for imprecise computation

Improving parallelism in I/O systems

The design and implement of ultra-scale data parallel in-situ visualization system

Combining in-situ and in-transit processing to enable extreme-scale scientific analysis

Similar Documents

Combining in-situ and in-transit processing to enable extreme-scale scientific analysis

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis.

An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis

A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations

Analytical modeling and optimization for affinity based thread scheduling on multicore systems

Performance evaluation of on-line scheduling algorithms for imprecise computation

Improving parallelism in I/O systems

The design and implement of ultra-scale data parallel in-situ visualization system

Combining in-situ and in-transit processing to enable extreme-scale scientific analysis