NDLI: BBQ: Elastic MapReduce over Cloud Platforms

Please wait, while we are loading the content...

Efficient Event Correlation over Distributed Systems

Optimal Resource Configuration of Complex Services in the Cloud

Adaptive Hybrid Queue Configuration for Supercomputer Systems

Deploying High Throughput Scientific Workflows on Container Schedulers with Makeflow and Mesos

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Mitigating YARN Container Overhead with Input Splits

APHiD: Hierarchical Task Placement to Enable a Tapered Fat Tree Topology for Lower Power and Cost in HPC Networks

CloudSight: A tenant-oriented transparency framework for cross-layer cloud troubleshooting

Preemptive Software Transactional Memory

Unveiling the Interplay Between Global Link Arrangements and Network Management Algorithms on Dragonfly Networks

Joint Optimization of Scaling and Placement of Virtual Network Services

Combating the Bandits in the Cloud: A Moving Target Defense Approach

Security Implications of Blockchain Cloud with Analysis of Block Withholding Attack

Maximum Sustainable Throughput Prediction for Data Stream Processing over Public Clouds

Dynamic Management of In-memory Storage for Efficiently Integrating Compute- and Data-intensive Computing on HPC Systems

Energy Efficient Algorithm for VNF Placement and Chaining

AnalyzeThat: A Programmable Shared-Memory System for an Array of Processing-In-Memory Devices

Pattern-Directed Replication Scheme for Heterogeneous Object-based Storage

Scalable Assembly for Massive Genomic Graphs

Enhancing the rCUDA Remote GPU Virtualization Framework: from a Prototype to a Production Solution

Performance Optimization by Dynamically Altering Cache Replacement Algorithm in CPU-GPU Heterogeneous Multi-Core Architecture

Using the Cloud for parameter estimation problems: comparing Spark vs MPI with a case-study

Scheduling Data Stream Jobs on Distributed Systems with Background Load

Apply Block Index Technique to Scientific Data Analysis and I/O Systems

Improving Resource Efficiency of Container-instance Clusters on Clouds

Analyzing the Parallel I/O Severity of MPI Applications

An Empirical Evaluation of How The Network Impacts The Performance and Energy Efficiency in RAMCloud

Automatic Consolidation of Virtual Machines in On-Premises Cloud Platforms

4CeeD: Real-Time Data Acquisition and Analysis Framework for Material-related Cyber-Physical Environments

Modeling Distributed Platforms from Application Traces for Realistic File Transfer Simulation

Flexible Scheduling of Distributed Analytic Applications

A new on-line method for scheduling independent tasks

Leveraging Renewable Energy in Edge Clouds for Data Stream Analysis in IoT

Parallel Variable Selection for Effective Performance Prediction

Swift-X: Accelerating OpenStack Swift with RDMA for Building an Efficient HPC Cloud

Enabling Distributed Software-Defined Environments Using Dynamic Infrastructure Service Composition

Supporting Fault-Tolerance in Presence of In-Situ Analytics

Application-Agnostic Power Monitoring in Virtualized Environments

Acyclic Partitioning of Large Directed Acyclic Graphs

Chrysaor: Fine-Grained, Fault-Tolerant Cloud-of-Clouds MapReduce

ProvChain: A Blockchain-based Data Provenance Architecture in Cloud Environment with Enhanced Privacy and Availability

WattsKit: Software-Defined Power Monitoring of Distributed Systems

High-Performance Key-Value Store On OpenSHMEM

KPI-agnostic Control for Fine-Grained Vertical Elasticity

Implementation and Evaluation of One-sided PGAS Communication in XcalableACC for Accelerated Clusters

A New File System I/O Mode for Efficient User-level Caching

mD3DOCKxb: An Ultra-Scalable CPU-MIC Coordinated Virtual Screening Framework

Cable-geometric error-prone approach for low-latency interconnection networks

Extending Message Passing Interface Windows to Storage

Analog-Digital Approach in Human Brain Modeling

Cloud Resource Scaling for Big Data Streaming Applications Using A Layered Multi-dimensional Hidden Markov Model

Smart RDF Data storage in Graph Databases

Hybrid Mobile Edge Computing: Unleashing the Full Potential of Edge Computing in Mobile Device Use Cases

Formal modeling and performance evaluation of a run-time rank remapping technique in Broadcast, Allgather and Allreduce MPI collective operations

On the Use of In-Memory Analytics Workflows to Compute eScience Indicators from Large Climate Datasets

A Game-Theoretic Approach for Runtime Capacity Allocation in MapReduce

Efficient Cache Update for In-Memory Cluster Computing with Spark

A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling

CtrlCloud: Performance-Aware Adaptive Control for Shared Resources in Clouds

A Two-Stage Multi-Objective Optimization of Erasure Coding in Overlay Networks

DOTA: Delay Bounded Optimal Cloudlet Deployment and User Association in WMANs

Towards Big Data Analytics across Multiple Clusters

Offloading communication control logic in GPU accelerated applications

CBase: A New Paradigm for Fast Virtual Machine Migration across Data Centers

Advanced Thread Synchronization for Multithreaded MPI Implementations

A Performance Study of UCX over InfiniBand

Towards Energy Budget Control in HPC

An Anomaly Detection Fabric for Clouds Based on Collaborative VM Communities

T-VMI: Trusted Virtual Machine Introspection in Cloud Environments

Predicting Cloud Performance for HPC Applications: a User-oriented Approach

Energy-efficient I/O Thread Schedulers for NVMe SSDs on NUMA

PCSsampler: Sample-based, Private-state Cluster Scheduling

Combining Both a Component Model and a Task-based Model for HPC Applications: a Feasibility Study on Gysela

COPS: Cost Based Object Placement Strategies on Hybrid Storage System for DBaaS Cloud

Scaling a Convolutional Neural Network for classification of Adjective Noun Pairs with TensorFlow on GPU Clusters

Towards Distributed Software-Defined Environments

TuNao: A High-Performance and Energy-Efficient Reconfigurable Accelerator for Graph Processing

Fine-grained Supervision and Restriction of Biomedical Applications in Linux Containers

Cost Model and Analysis of Iterative MapReduce Applications for Hybrid Cloud Bursting

A Level-Wise Load Balanced Scientific Workflow Execution Optimization using NSGA-II

Defining Intercloud Security Framework and Architecture Components for Multi-Cloud Data Intensive Applications

Automatic Adaption of the Sampling Frequency for Detailed Performance Analysis

Exploring Shared State in Key-Value Store for Window-Based Multi-Pattern Streaming Analytics

IT Security and Privacy Standards in Comparison: Improving FedRAMP Authorization for Cloud Service Providers

GPU in-memory processing using Spark for iterative computation

Performance Modelling and Cost Effective Execution for Distributed Graph Processing on Configurable VMs

QoS-Aware Virtual Infrastructures Allocation on SDN-based Clouds

Multi-dimensional admission control and capacity planning for IaaS clouds with multiple service classes

Preliminary Performance Analysis of Multi-rail Fat-tree Networks

Performance Modelling and Verification of Cloud-based Auto-Scaling Policies

On Estimating Minimum Bids for Amazon EC2 Spot Instances

LogAider: A tool for mining potential correlations of HPC log events

Modeling correlation between QoS attributes for trust computation in cloud computing environments

An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications

Optimized Cloud Deployment of Multi-tenant Software Considering Data Protection Concerns

Scaling HDFS to more than 1 million operations per second with HopsFS

Dynamic Resource Management Across Cloud-Edge Resources for Performance-Sensitive Applications

A Live Demo for Showing the Benefits of Applying the Remote GPU Virtualization Technique to Cloud Computing

BIOPET: towards Scalable, Maintainable, User-friendly, Robust and Flexible NGS data analysis pipelines

RAPID: A Fast Data Update Protocol in Erasure Coded Storage Systems for Big Data

Performance Models for Communication in Collective I/O Operations

Evaluation of HPC-Big Data Applications Using Cloud Platforms

A lightweight MapReduce framework for secure processing with SGX

Fine-grained Nested Virtual Machine Performance Analysis Through First Level Hypervisor Tracing

A Robust Tabu Search Heuristic for VM Consolidation under Demand Uncertainty in Virtualized Datacenters

SynAPTIC: Secure And Persistent connecTIvity for Containers

Practical Service Placement Approach for Microservices Architecture

Designing and Modelling Selective Replication for Fault-tolerant HPC Applications

Crowdsourced Data Integrity Verification for Key-Value Stores in the Cloud

Toward a Holistic Framework for Conducting Scientific Evaluations of OpenStack

Massively Parallel Simulations of Spread of Infectious Diseases over Realistic Social Networks

Optimization of checkpoints and execution model for an implementation of OpenMP on distributed memory architectures

Load and Video Performance Patterns of a Cloud Based WebRTC Architecture

Medical Imaging Processing on a Big Data platform using Python: Experiences with Heterogeneous and Homogeneous Architectures

MapReduce-based Algorithms for Managing Big RDF Graphs: State-of-the-Art Analysis, Paradigms, and Future Directions

Reducing Load Imbalance of Virtual Clusters via Reconfiguration and Adaptive Job Scheduling

A Data-driven Approach based on Auto-Regressive Models for Energy-Efficient Clouds

PRIVAaaS: privacy approach for a distributed cloud-based data analytics platforms

Secure cloud storage service for detection of security violations

Lemonade: A scalable and efficient Spark-based platform for data analytics

Optimized MapFile based Storage of Small files in Hadoop

Empirical Mode Decomposition for Modeling of Parallel Applications on Intel Xeon Phi Processors

Evaluating the performance of continuous test-based cloud service certification

Data-Aware Support for Hybrid HPC and Big Data Applications

IBM Research Hybrid Cloud Poster

An Accuracy-Aware Implementation of Two-Point Three-Dimensional Correlation Function using Bin-Recycling Strategy on GPU

Energy model for low-power cluster

A Game Theoretic Method for VM-to-Hypervisor Attacks Detection in Cloud Environment

Using the Jetstream Research Cloud to provide Science Gateway resources

A Big Data Architecture for Automotive Applications: PSA Group Deployment Experience

Near-optimal policies for energy-aware task assignment in server farms

DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions

AURA: Recovering from Transient Failures in Cloud Deployments

BBQ: Elastic MapReduce over Cloud Platforms

Multi-agent recommendation system in Internet of Things

Massive Data Load on Distributed Database Systems over HBase

Mermaid: Integrating Vertex-Centric with Edge-Centric for Real-World Graph Processing

Representing Variant Calling Format as Directed Acyclic Graphs to enable the use of cloud computing for efficient and cost effective genome analysis

Techniques for Handling Error in User-estimated Execution Times During Resource Management on Systems Processing MapReduce Jobs

EffiEye: Application-aware Large Flow Detection in Data Center

BBQ: Elastic MapReduce over Cloud Platforms

Content Provider	ACM Digital Library
Author	Konstantinou, Ioannis Chalvantzis, Nikolaos Koziris, Nectarios
Abstract	Cloud infrastructure services such as Amazon EMR allow users to have access to tailor-made Big Data processing clusters within a few clicks from their web browser, thanks to the elastic property of the cloud. In virtual cloud environments, resource management is desired to be performed in a way which optimizes utilization, thus maximizing the value of the resources acquired. As cloud infrastructures become increasingly popular for Big Data analysis, the execution of programs with respect to user selected performance goals, such as job completion deadlines, remains a challenge. In this work we present BARBECUE (joB AwaRe Big-data Elasticity CloUd managEment System), a system that allows a Hadoop MapReduce virtual cluster to automatically adjust its size to the workload it is required to execute in order to respect individual jobs' completion deadlines without acquiring more resources than the least necessary. To that end, BBQ's Decision Making module uses a Performance Model for MapReduce jobs which can express cluster resources (i.e., YARN Container capacity) and execution time as a function of the number of nodes in the cluster. BBQ leverages the abstraction of YARN, making it feasible for integration with other execution frameworks, such as Spark, with the necessary changes to its pluggable Decision Making module. We also add a new feature to Hadoop MapReduce which can now dynamically, on-the-fly update the number of selected ReduceTasks in cases where the cluster is expanded, so that our system makes full use of the resources it has acquired during the reduce phase of the execution. BBQ uses an adaptation of the hill climbing algorithm to estimate the optimal combination of number of nodes and reduce waves given a known job, its data input and an execution deadline. The attendees will be able to watch the system perform cluster resizes in real-time in order to execute its assigned jobs in time.
Starting Page	766
Ending Page	771
Page Count	6
File Format	PDF
ISBN	9781509066100
DOI	10.1109/CCGRID.2017.140
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2017-05-14
Access Restriction	Subscribed
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in