NDLI: Automating Distributed Partial Aggregation

Please wait, while we are loading the content...

Automating Distributed Partial Aggregation

Reservation-based Scheduling: If You're Late Don't Blame Us!

Transparent, Live Migration of a Software-Defined Network

GentleRain: Cheap and Scalable Causal Consistency with Physical Clocks

Exploiting iterative-ness for parallel ML computations

Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks

What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems

PerfScope: Practical Online Server Performance Bug Inference in Production Cloud Computing Infrastructures

Tales of the Tail: Hardware, OS, and Application-level Sources of Tail Latency

On the Control Plane of a Self-service Cloud Platform

DIMMer: A case for turning off DIMMs in clouds

Hybris: Robust Hybrid Cloud Storage

The Case for Drill-Ready Cloud Computing

Merlin: Application- and Platform-aware Resource Allocation in Consolidated Server Systems

DaRPC: Data Center RPC

Adaptive Stream Processing using Dynamic Batch Sizing

ATAC: Ambient Temperature-Aware Capping for Power Efficient Datacenters

Connected Components in MapReduce and Beyond

Vanguard: Increasing Server Efficiency via Workload Isolation in the Storage I/O Path

Long-term SLOs for reclaimed cloud computing resources

Vagabond: Dynamic Network Endpoint Reconfiguration in Virtualized Environments

Paxos Quorum Leases: Fast Reads Without Sacrificing Writes

C-Hint: An Effective and Reliable Cache Management for RDMA-Accelerated Key-Value Stores

Distributed Autonomous Virtual Resource Management in Datacenters Using Finite-Markov Decision Process

ReproLite: A Lightweight Tool to Quickly Reproduce Hard System Bugs

Wrangler: Predictable and Faster Jobs using Fewer Resources

vPipe: Piped I/O Offloading for Efficient Data Movement in Virtualized Clouds

Dynamic Performance Profiling of Cloud Caches

PriorityMeister: Tail Latency QoS for Shared Networked Storage

Automating Distributed Partial Aggregation

Content Provider	ACM Digital Library
Author	McDirmid, Sean Zhang, Jiaxing Liu, Chang Guo, Zhenyu Zhou, Hucheng Moscibroda, Thomas
Abstract	Partial aggregation is of great importance in many distributed data-parallel systems. Most notably, it is commonly applied by MapReduce programs to optimize I/O by successively aggregating partially reduced results into a final result, as opposed to aggregating all input records at once. In spite of its importance, programmers currently enable partial aggregation by tediously encoding their reduce functionality into separate reduce and combine functions. This is error prone and often leads to missed optimization opportunities. This paper proposes an algorithm that automatically verifies if the original monolithic reduce function of a MapReduce program is eligible for partial aggregation, and if so, synthesizes enabling partial aggregation code. The key insight behind this algorithm is a novel necessary and sufficient condition for when partial aggregation is applicable to a reduce function. This insight provides us with a formal foundation for an automaton, which derives a satisfiability problem that can be fed into a standard SMT solver. By doing so, we transform the problem of synthesis into a program inversion problem, which is however nondeterministic. Although such inversion is hard to solve in general, we observe that most reducers in practical distributed computing contexts can be classified into a few categories for which we can design efficient synthesis algorithms. Finally, we build and evaluate a prototype of our method to demonstrate its feasibility in the SCOPE distributed data-parallel system.
Starting Page	1
Ending Page	12
Page Count	12
File Format	PDF
ISBN	9781450332521
DOI	10.1145/2670979.2670980
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2014-11-03
Publisher Place	New York
Access Restriction	Subscribed
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Automating Distributed Partial Aggregation

Automating Distributed Partial Aggregation

Automating Distributed Partial Aggregation

Automating experimentation on distributed testbeds

Automating software tests with partial oracles in integrated environments

WS-Aggregation: distributed aggregation of web services data

Distributed event aggregation for content-based publish/subscribe systems

Automating compensation in a multidatabase

Automating Information Aggregation Processes in Information Centres

Automating Distributed Partial Aggregation

Similar Documents

Automating Distributed Partial Aggregation

Automating Distributed Partial Aggregation

Automating Distributed Partial Aggregation

Automating experimentation on distributed testbeds

Automating software tests with partial oracles in integrated environments

WS-Aggregation: distributed aggregation of web services data

Distributed event aggregation for content-based publish/subscribe systems

Automating compensation in a multidatabase

Automating Information Aggregation Processes in Information Centres

Automating Distributed Partial Aggregation