Loading...
Please wait, while we are loading the content...
Turbine: a distributed-memory dataflow engine for extreme-scale many-task applications
| Content Provider | ACM Digital Library |
|---|---|
| Author | Lusk, Ewing L. Maheshwari, Ketan Foster, Ian T. Katz, Daniel S. Wilde, Michael Wozniak, Justin M. Armstrong, Timothy G. |
| Abstract | Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper layer of many loosely-coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program. "Many-task" programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tighly-coupled parallelism at the lower level via multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and inter-task data movement is a significant performance challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution, and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems. |
| Starting Page | 1 |
| Ending Page | 12 |
| Page Count | 12 |
| File Format | |
| ISBN | 9781450318761 |
| DOI | 10.1145/2443416.2443421 |
| Language | English |
| Publisher | Association for Computing Machinery (ACM) |
| Publisher Date | 2012-05-20 |
| Publisher Place | New York |
| Access Restriction | Subscribed |
| Subject Keyword | Concurrency Adlb Mpi Turbine Exascale Swift Dataflow |
| Content Type | Text |
| Resource Type | Article |
