NDLI: Storage utilization in the long tail of science

Please wait, while we are loading the content...

Accelerating our understanding of supernova explosion mechanism via simulations and visualizations with GenASiS

Using Mozilla badges to certify XSEDE users and promote training

XSEDE value added, cost avoidance, and return on investment

In-core volume rendering for Cartesian grid fluid dynamics simulations

Paleoscape model of coastal South Africa during modern human origins: progress in scaling and coupling climate, vegetation, and agent-based models on XSEDE

Extending access to HPC skills through a blended online course

Overview of XSEDE-PRACE collaborative projects in 2014

Bring the NLACE model online using XSEDE and HUBzero

Cyberinfrastructure resources enabling creation of the loblolly pine reference transcriptome

Connecting the non-traditional user-community to the national CyberInfrastructure

Publishing and consuming GLUE v2.0 resource information in XSEDE

GeoHashViz: interactive analytics for mapping spatiotemporal diffusion of Twitter hashtags

NCBI-BLAST programs optimization on XSEDE resources for sustainable aquaculture

ISLET: an isolated, scalable, & lightweight environment for training

Market-based on demand scheduling (MBoDS) in co-operative grid environment

Autotuning OpenACC work distribution via direct search

FlowGate: towards extensible and scalable web-based flow cytometry data analysis

Science gateways for humanities, arts, and social science

A prototype sampling interface for PAPI

The CIPRES workbench: a flexible framework for creating science gateways

A scalable computational approach to political redistricting optimization

On fostering a culture of research cyberinfrastructure grant proposals within a community of service providers in an EPSCoR state

Optimizing codes on the Xeon Phi: a case-study with LAMMPS

Porting scientific libraries to PGAS in XSEDE resources: practice and experience

Discovering the influence of socioeconomic factors on online game behaviors

Incorporating interactive compute environments into web-based training materials using the Cornell job runner service

Jetstream: a self-provisioned, scalable science and engineering cloud environment

Enabling HPC simulation workflows for complex industrial flow problems

Performance examinations of multiple time-stepping algorithms on stampede supercomputer

TAS view of XSEDE users and usage

Bridges: a uniquely flexible HPC resource for new communities and data analytics

A performance predictor for UltraScan supercomputer calculations

Performance assessment of real-time estimation of continuous-time stochastic volatility of financial data on GPUs

Multidisciplinary research and education with open tools: metagenomic analysis of 16S rRNA using Arduino, Android, Mothur and XSEDE

Using data science to understand tape-based archive workloads

CDD: computational discovery desktop

Grouping game players using parallelized k-means on supercomputers

Storage utilization in the long tail of science

Leveraging DiaGrid hub for interactively generating and running parallel programs

The VAT: enhanced video analysis

Advanced user environment design and implementation on integrated multi-architecture supercomputers

Inversion of magnetotelluric data using integral equation approach with variable sensitivity domain: application to earthscope MT data

Integrating apache spark into PBS-Based HPC environments

A SIMD tabu search implementation for solving the quadratic assignment problem with GPU acceleration

Storage utilization in the long tail of science

Content Provider	ACM Digital Library
Author	Tatineni, Mahidhar Wagner, Rick Lockwood, Glenn K.
Abstract	The increasing expansion of computations in non-traditional domain sciences has resulted in an increasing demand for research cyberinfrastructure that is suitable for small- and mid-scale job sizes. The computational aspects of these emerging communities are coming into focus and being addressed through the deployment of several new XSEDE resources that feature easy on-ramps, customizable software environments through virtualization, and interconnects optimized for jobs that only use hundreds or thousands of cores; however, the data storage requirements for these emerging communities remains much less well characterized. To this end, we examined the distribution of file sizes on two of the Lustre file systems within the Data Oasis storage system at the San Diego Supercomputer Center (SDSC). We found that there is a very strong preference for small files among SDSC's users, with 90% of all files being less than 2 MB in size. Furthermore, 50% of all file system capacity is consumed by files under 2 GB in size, and these distributions are consistent on both scratch and projects storage file systems. Because parallel file systems like Lustre and GPFS are optimized for parallel IO to large, widestripe files, these findings suggest that parallel file systems may not be the most suitable storage solutions when designing cyberinfrastructure to meet the needs of emerging communities.
Starting Page	1
Ending Page	8
Page Count	8
File Format	PDF
ISBN	9781450337205
DOI	10.1145/2792745.2792777
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2015-07-26
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Storage Utilization Lustre Parallel file systems
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in