Loading...
Please wait, while we are loading the content...
Similar Documents
Parallel I/O and the metadata wall
| Content Provider | ACM Digital Library |
|---|---|
| Author | Howard, Kristopher Alam, Sadaf R. El-Harake, Hussein N. Stringfellow, Neil Verzelloni, Fabio |
| Abstract | Large HPC installations typically make use of parallel file systems that adhere to POSIX I/O conventions, and that implement a separation of data and metadata in order to maintain high performance. File systems such as GPFS and Lustre have evolved to enable an increase in data bandwidth that is primarily achieved by adding more disk drives behind an increasing number of disk controllers. Improvements in metadata performance cannot be achieved by just deploying a large volume of hardware, as the defining characteristics are the number of simultaneous operations that can be carried out and the latency of those operations. For highly scalable applications using parallel I/O libraries, the speed of metadata operations, such as opening a file on thousands of processes, has the potential to become the major bottleneck to improved I/O performance. This Metadata Wall has the ability to grow such that metadata operations can take much longer than the subsequent data operations, even on systems with very large amounts of I/O data bandwidth. We present results showing the performance of metadata operations with standard disk equipment and with solid state storage hardware, and extrapolate whether we expect the evolution in hardware alone will be sufficient to limit the effects of this I/O Metadata Wall. We also report challenges in making the metadata I/O measurements and subsequent analysis for parallel file systems. |
| Starting Page | 13 |
| Ending Page | 18 |
| Page Count | 6 |
| File Format | |
| ISBN | 9781450311038 |
| DOI | 10.1145/2159352.2159356 |
| Language | English |
| Publisher | Association for Computing Machinery (ACM) |
| Publisher Date | 2011-11-13 |
| Publisher Place | New York |
| Access Restriction | Subscribed |
| Subject Keyword | Metadata benchmarking Solid state disk Parallel file systems |
| Content Type | Text |
| Resource Type | Article |
