NDLI: Scalable massively parallel I/O to task-local files

Content Provider	IEEE Xplore Digital Library
Author	Frings, W. Wolf, F. Petkov, V.
Copyright Year	2009
Description	Author affiliation: Julich Supercomput. Centre, Julich, Germany (Frings, W.) \|\| Tech. Univ. Munchen, Munich, Germany (Petkov, V.) \|\| RWTH Aachen Univ., Aachen, Germany (Wolf, F.)
Abstract	Parallel applications often store data in multiple task-local files, for example, to remember checkpoints, to circumvent memory limitations, or to record performance data. When operating at very large processor configurations, such applications often experience scalability limitations when the simultaneous creation of thousands of files causes metadataserver contention or simply when large file counts complicate file management or operations on those files even destabilize the file system. SIONlib is a parallel I/O library that addresses this problem by transparently mapping a large number of task-local files onto a small number of physical files via internal metadata handling and block alignment to ensure high performance. While requiring only minimal source code changes, SIONlib significantly reduces file creation overhead and simplifies file handling without penalizing read and write performance. We evaluate SIONlib's efficiency with up to 288 K tasks and report significant performance improvements in two application scenarios.
Starting Page	1
Ending Page	11
File Size	280084
Page Count	11
File Format	PDF
ISBN	9781605587448
DOI	10.1145/1654059.1654077
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2009-11-14
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Association for Computing Machinery, Inc. (ACM)
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Scalable massively parallel I/O to task-local files

Scalable I/O for parallel access to task-local files with SIONlib

Autotuned parallel I/O for highly scalable biosequence analysis

Towards scalable I/O architecture for exascale systems

Scalable massively parallel I/O to task-local files (2009)

PDS/PIO: lightweight libraries for collective parallel I/O

Scalable parallel I/O alternatives for massively parallel partitioned solver systems

Poster: a scalable two-phase parallel I/O library with application to a large scale subsurface simulator

Parallel I/O Subsystems in Massively Parallel Supercomputers (1995)

Scalable massively parallel I/O to task-local files

Similar Documents

Scalable massively parallel I/O to task-local files

Scalable I/O for parallel access to task-local files with SIONlib

Autotuned parallel I/O for highly scalable biosequence analysis

Towards scalable I/O architecture for exascale systems

Scalable massively parallel I/O to task-local files (2009)

PDS/PIO: lightweight libraries for collective parallel I/O

Scalable parallel I/O alternatives for massively parallel partitioned solver systems

Poster: a scalable two-phase parallel I/O library with application to a large scale subsurface simulator

Parallel I/O Subsystems in Massively Parallel Supercomputers (1995)

Scalable massively parallel I/O to task-local files