NDLI: Dynamic Management of In-memory Storage for Efficiently Integrating Compute- and Data-intensive Computing on HPC Systems

Please wait, while we are loading the content...

Efficient Event Correlation over Distributed Systems

Optimal Resource Configuration of Complex Services in the Cloud

Adaptive Hybrid Queue Configuration for Supercomputer Systems

Deploying High Throughput Scientific Workflows on Container Schedulers with Makeflow and Mesos

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Mitigating YARN Container Overhead with Input Splits

APHiD: Hierarchical Task Placement to Enable a Tapered Fat Tree Topology for Lower Power and Cost in HPC Networks

CloudSight: A tenant-oriented transparency framework for cross-layer cloud troubleshooting

Preemptive Software Transactional Memory

Unveiling the Interplay Between Global Link Arrangements and Network Management Algorithms on Dragonfly Networks

Joint Optimization of Scaling and Placement of Virtual Network Services

Combating the Bandits in the Cloud: A Moving Target Defense Approach

Security Implications of Blockchain Cloud with Analysis of Block Withholding Attack

Maximum Sustainable Throughput Prediction for Data Stream Processing over Public Clouds

Dynamic Management of In-memory Storage for Efficiently Integrating Compute- and Data-intensive Computing on HPC Systems

Energy Efficient Algorithm for VNF Placement and Chaining

AnalyzeThat: A Programmable Shared-Memory System for an Array of Processing-In-Memory Devices

Pattern-Directed Replication Scheme for Heterogeneous Object-based Storage

Scalable Assembly for Massive Genomic Graphs

Enhancing the rCUDA Remote GPU Virtualization Framework: from a Prototype to a Production Solution

Performance Optimization by Dynamically Altering Cache Replacement Algorithm in CPU-GPU Heterogeneous Multi-Core Architecture

Using the Cloud for parameter estimation problems: comparing Spark vs MPI with a case-study

Scheduling Data Stream Jobs on Distributed Systems with Background Load

Apply Block Index Technique to Scientific Data Analysis and I/O Systems

Improving Resource Efficiency of Container-instance Clusters on Clouds

Analyzing the Parallel I/O Severity of MPI Applications

An Empirical Evaluation of How The Network Impacts The Performance and Energy Efficiency in RAMCloud

Automatic Consolidation of Virtual Machines in On-Premises Cloud Platforms

4CeeD: Real-Time Data Acquisition and Analysis Framework for Material-related Cyber-Physical Environments

Modeling Distributed Platforms from Application Traces for Realistic File Transfer Simulation

Flexible Scheduling of Distributed Analytic Applications

A new on-line method for scheduling independent tasks

Leveraging Renewable Energy in Edge Clouds for Data Stream Analysis in IoT

Parallel Variable Selection for Effective Performance Prediction

Swift-X: Accelerating OpenStack Swift with RDMA for Building an Efficient HPC Cloud

Enabling Distributed Software-Defined Environments Using Dynamic Infrastructure Service Composition

Supporting Fault-Tolerance in Presence of In-Situ Analytics

Application-Agnostic Power Monitoring in Virtualized Environments

Acyclic Partitioning of Large Directed Acyclic Graphs

Chrysaor: Fine-Grained, Fault-Tolerant Cloud-of-Clouds MapReduce

ProvChain: A Blockchain-based Data Provenance Architecture in Cloud Environment with Enhanced Privacy and Availability

WattsKit: Software-Defined Power Monitoring of Distributed Systems

High-Performance Key-Value Store On OpenSHMEM

KPI-agnostic Control for Fine-Grained Vertical Elasticity

Implementation and Evaluation of One-sided PGAS Communication in XcalableACC for Accelerated Clusters

A New File System I/O Mode for Efficient User-level Caching

mD3DOCKxb: An Ultra-Scalable CPU-MIC Coordinated Virtual Screening Framework

Cable-geometric error-prone approach for low-latency interconnection networks

Extending Message Passing Interface Windows to Storage

Analog-Digital Approach in Human Brain Modeling

Cloud Resource Scaling for Big Data Streaming Applications Using A Layered Multi-dimensional Hidden Markov Model

Smart RDF Data storage in Graph Databases

Hybrid Mobile Edge Computing: Unleashing the Full Potential of Edge Computing in Mobile Device Use Cases

Formal modeling and performance evaluation of a run-time rank remapping technique in Broadcast, Allgather and Allreduce MPI collective operations

On the Use of In-Memory Analytics Workflows to Compute eScience Indicators from Large Climate Datasets

A Game-Theoretic Approach for Runtime Capacity Allocation in MapReduce

Efficient Cache Update for In-Memory Cluster Computing with Spark

A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling

CtrlCloud: Performance-Aware Adaptive Control for Shared Resources in Clouds

A Two-Stage Multi-Objective Optimization of Erasure Coding in Overlay Networks

DOTA: Delay Bounded Optimal Cloudlet Deployment and User Association in WMANs

Towards Big Data Analytics across Multiple Clusters

Offloading communication control logic in GPU accelerated applications

CBase: A New Paradigm for Fast Virtual Machine Migration across Data Centers

Advanced Thread Synchronization for Multithreaded MPI Implementations

A Performance Study of UCX over InfiniBand

Towards Energy Budget Control in HPC

An Anomaly Detection Fabric for Clouds Based on Collaborative VM Communities

T-VMI: Trusted Virtual Machine Introspection in Cloud Environments

Predicting Cloud Performance for HPC Applications: a User-oriented Approach

Energy-efficient I/O Thread Schedulers for NVMe SSDs on NUMA

PCSsampler: Sample-based, Private-state Cluster Scheduling

Combining Both a Component Model and a Task-based Model for HPC Applications: a Feasibility Study on Gysela

COPS: Cost Based Object Placement Strategies on Hybrid Storage System for DBaaS Cloud

Scaling a Convolutional Neural Network for classification of Adjective Noun Pairs with TensorFlow on GPU Clusters

Towards Distributed Software-Defined Environments

TuNao: A High-Performance and Energy-Efficient Reconfigurable Accelerator for Graph Processing

Fine-grained Supervision and Restriction of Biomedical Applications in Linux Containers

Cost Model and Analysis of Iterative MapReduce Applications for Hybrid Cloud Bursting

A Level-Wise Load Balanced Scientific Workflow Execution Optimization using NSGA-II

Defining Intercloud Security Framework and Architecture Components for Multi-Cloud Data Intensive Applications

Automatic Adaption of the Sampling Frequency for Detailed Performance Analysis

Exploring Shared State in Key-Value Store for Window-Based Multi-Pattern Streaming Analytics

IT Security and Privacy Standards in Comparison: Improving FedRAMP Authorization for Cloud Service Providers

GPU in-memory processing using Spark for iterative computation

Performance Modelling and Cost Effective Execution for Distributed Graph Processing on Configurable VMs

QoS-Aware Virtual Infrastructures Allocation on SDN-based Clouds

Multi-dimensional admission control and capacity planning for IaaS clouds with multiple service classes

Preliminary Performance Analysis of Multi-rail Fat-tree Networks

Performance Modelling and Verification of Cloud-based Auto-Scaling Policies

On Estimating Minimum Bids for Amazon EC2 Spot Instances

LogAider: A tool for mining potential correlations of HPC log events

Modeling correlation between QoS attributes for trust computation in cloud computing environments

An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications

Optimized Cloud Deployment of Multi-tenant Software Considering Data Protection Concerns

Scaling HDFS to more than 1 million operations per second with HopsFS

Dynamic Resource Management Across Cloud-Edge Resources for Performance-Sensitive Applications

A Live Demo for Showing the Benefits of Applying the Remote GPU Virtualization Technique to Cloud Computing

BIOPET: towards Scalable, Maintainable, User-friendly, Robust and Flexible NGS data analysis pipelines

RAPID: A Fast Data Update Protocol in Erasure Coded Storage Systems for Big Data

Performance Models for Communication in Collective I/O Operations

Evaluation of HPC-Big Data Applications Using Cloud Platforms

A lightweight MapReduce framework for secure processing with SGX

Fine-grained Nested Virtual Machine Performance Analysis Through First Level Hypervisor Tracing

A Robust Tabu Search Heuristic for VM Consolidation under Demand Uncertainty in Virtualized Datacenters

SynAPTIC: Secure And Persistent connecTIvity for Containers

Practical Service Placement Approach for Microservices Architecture

Designing and Modelling Selective Replication for Fault-tolerant HPC Applications

Crowdsourced Data Integrity Verification for Key-Value Stores in the Cloud

Toward a Holistic Framework for Conducting Scientific Evaluations of OpenStack

Massively Parallel Simulations of Spread of Infectious Diseases over Realistic Social Networks

Optimization of checkpoints and execution model for an implementation of OpenMP on distributed memory architectures

Load and Video Performance Patterns of a Cloud Based WebRTC Architecture

Medical Imaging Processing on a Big Data platform using Python: Experiences with Heterogeneous and Homogeneous Architectures

MapReduce-based Algorithms for Managing Big RDF Graphs: State-of-the-Art Analysis, Paradigms, and Future Directions

Reducing Load Imbalance of Virtual Clusters via Reconfiguration and Adaptive Job Scheduling

A Data-driven Approach based on Auto-Regressive Models for Energy-Efficient Clouds

PRIVAaaS: privacy approach for a distributed cloud-based data analytics platforms

Secure cloud storage service for detection of security violations

Lemonade: A scalable and efficient Spark-based platform for data analytics

Optimized MapFile based Storage of Small files in Hadoop

Empirical Mode Decomposition for Modeling of Parallel Applications on Intel Xeon Phi Processors

Evaluating the performance of continuous test-based cloud service certification

Data-Aware Support for Hybrid HPC and Big Data Applications

IBM Research Hybrid Cloud Poster

An Accuracy-Aware Implementation of Two-Point Three-Dimensional Correlation Function using Bin-Recycling Strategy on GPU

Energy model for low-power cluster

A Game Theoretic Method for VM-to-Hypervisor Attacks Detection in Cloud Environment

Using the Jetstream Research Cloud to provide Science Gateway resources

A Big Data Architecture for Automotive Applications: PSA Group Deployment Experience

Near-optimal policies for energy-aware task assignment in server farms

DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions

AURA: Recovering from Transient Failures in Cloud Deployments

BBQ: Elastic MapReduce over Cloud Platforms

Multi-agent recommendation system in Internet of Things

Massive Data Load on Distributed Database Systems over HBase

Mermaid: Integrating Vertex-Centric with Edge-Centric for Real-World Graph Processing

Representing Variant Calling Format as Directed Acyclic Graphs to enable the use of cloud computing for efficient and cost effective genome analysis

Techniques for Handling Error in User-estimated Execution Times During Resource Management on Systems Processing MapReduce Jobs

EffiEye: Application-aware Large Flow Detection in Data Center

Dynamic Management of In-memory Storage for Efficiently Integrating Compute- and Data-intensive Computing on HPC Systems

Content Provider	ACM Digital Library
Author	Luo, Feng Ge, Rong Srimani, Pradip K. Xuan, Pengfei
Abstract	In order to boost the performance of data-intensive computing on HPC systems, in-memory computing frameworks, such as Apache Spark and Flink, use local DRAM for data storage. Optimizing the memory allocation to data storage is critical to delivering performance to traditional HPC compute jobs and throughput to data-intensive applications sharing the HPC resources. Current practices that statically configure in-memory storage may leave inadequate space for compute jobs or miss the opportunity to utilize available space for data-intensive applications. In this paper, we explore techniques to dynamically adjust in-memory storage allocation and provide optimum memory to compute jobs. We have developed a dynamic in-memory storage controller, DynIMS, which monitors memory demands of compute tasks in real time and employs a feedback-based control mechanism to adapt the allocation of in-memory storage. We test DynIMS using HPCC and Spark workloads on a HPC cluster. Experimental results show that DynIMS can achieve up to 5X performance improvement compared to systems with static memory allocations.
Starting Page	549
Ending Page	558
Page Count	10
File Format	PDF
ISBN	9781509066100
DOI	10.1109/CCGRID.2017.66
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2017-05-14
Access Restriction	Subscribed
Subject Keyword	Dynamic control Hpc In-memory storage Component Parallel file system
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in