Loading...
Please wait, while we are loading the content...
Similar Documents
Achieving Data-Aware Load Balancing through Distributed Queues and Key / Value Stores
| Content Provider | Semantic Scholar |
|---|---|
| Author | Wang, Ke Raicu, Ioan |
| Copyright Year | 2014 |
| Abstract | Load balancing techniques (e.g. work stealing) are important to obtain the best performance for distributed task scheduling system. In work stealing, tasks are randomly migrated from heavy-loaded schedulers to idle ones. However, for data-intensive applications where tasks are dependent and task execution involves processing large amount of data, migrating tasks blindly would compromise the data-locality incurring significant data-transferring overhead. In this work, we propose a data-aware work stealing technique that combines key-value stores and distributed queues enabling it to achieve good load balancing, all while maximizing data-locality. We leverage a distributed key-value store, ZHT, as a meta-data service that stores task dependency and data-locality information. We implement the proposed technique in MATRIX, a distributed task execution fabric. We evaluate the work with all-pairs application structured as direct acyclic graph from biometrics, and compare with Falkon data-diffusion technique. |
| File Format | PDF HTM / HTML |
| Alternate Webpage(s) | http://datasys.cs.iit.edu/reports/2014_GCASR14_paper-data-aware-scheduling.pdf |
| Language | English |
| Access Restriction | Open |
| Content Type | Text |
| Resource Type | Article |
