NDLI: LetGo: A Lightweight Continuous Framework for HPC Applications Under Failures

Please wait, while we are loading the content...

Building Secure Platforms for Research on Human Subjects: The Importance of Computer Scientists

Enabling Workflow-Aware Scheduling on HPC Systems

Caches All the Way Down: Infrastructure for Data Intensive Science

NICE: Network-Integrated Cluster-Efficient Storage

Deep Learning in Cancer and Infectious Disease: Novel Driver Problems for Future HPC Architecture

knor: A NUMA-Optimized In-Memory, Distributed and Semi-External-Memory k-means Library

Better Safe than Sorry: Grappling with Failures of In-Memory Data Analytics Frameworks

Diagnosing Machine Learning Pipelines with Fine-grained Lineage

Using Scientific Computing to Advance Wildland Fire Monitoring and Prediction

Predicting Output Performance of a Petascale Supercomputer

IOGP: An Incremental Online Graph Partitioning Algorithm for Distributed Graph Databases

Parallel Stream Processing Against Workload Skewness and Variance

MaDaTS: Managing Data on Tiered Storage for Scientific Workflows

CuMF_SGD: Parallelized Stochastic Gradient Descent for Matrix Factorization on GPUs

LetGo: A Lightweight Continuous Framework for HPC Applications Under Failures

COS: A Parallel Performance Model for Dynamic Variations in Processor Speed, Memory Speed, and Thread Concurrency

TCP Throughput Profiles Using Measurements over Dedicated Connections

Machine and Application Aware Partitioning for Adaptive Mesh Refinement Applications

ArrayUDF: User-Defined Scientific Data Analysis on Arrays

To Push or To Pull: On Reducing Communication and Synchronization in Graph Computations

Towards a More Complete Understanding of SDC Propagation

Explaining Wide Area Data Transfer Performance

AllConcur: Leaderless Concurrent Atomic Broadcast

LetGo: A Lightweight Continuous Framework for HPC Applications Under Failures

Content Provider	ACM Digital Library
Author	Guan, Qiang Pattabiraman, Karthik Debardeleben, Nathan Ripeanu, Matei Fang, Bo
Abstract	Requirements for reliability, low power consumption, and performance place complex and conflicting demands on the design of high-performance computing (HPC) systems. Fault-tolerance techniques such as checkpoint/restart (C/R) protect HPC applications against hardware faults. These techniques, however, have non negligible overheads particularly when the fault rate exposed by the hardware is high: it is estimated that in future HPC systems, up to 60% of the computational cycles/power will be used for fault tolerance. To mitigate the overall overhead of fault-tolerance techniques, we propose LetGo, an approach that attempts to continue the execution of a HPC application when crashes would otherwise occur. Our hypothesis is that a class of HPC applications have good enough intrinsic fault tolerance so that its possible to re-purpose the default mechanism that terminates an application once a crash-causing error is signalled, and instead attempt to repair the corrupted application state, and continue the application execution. This paper explores this hypothesis, and quantifies the impact of using this observation in the context of checkpoint/restart (C/R) mechanisms. Our fault-injection experiments using a suite of five HPC applications show that, on average, LetGo is able to elide 62% of the crashes encountered by applications, of which 80% result in correct output, while incurring a negligible performance overhead. As a result, when LetGo is used in conjunction with a C/R scheme, it enables significantly higher efficiency thereby leading to faster time to solution.
Starting Page	117
Ending Page	130
Page Count	14
File Format	PDF
ISBN	9781450346993
DOI	10.1145/3078597.3078609
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2017-06-26
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Fault tolerance Forward recovery Hpc Checkpoint/restart
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

The design and implementation of checkpoint/restart process fault tolerance for Open MPI (2007)

The design and implementation of checkpoint/restart process tolerance for Open MPI (2007)

ECOFIT: A Framework to Estimate Energy Consumption of Fault Tolerance Protocols for HPC Applications

Coordinated checkpoint/restart process fault tolerance for mpi applications on hpc systems

Coordinated Checkpoint/Restart Process Fault Tolerance for MPI Applications on HPC Systems (2010)

A checkpoint and restart service specification for open mpi (2006)

CRFS: A Lightweight User-Level Filesystem for Generic Checkpoint/Restart

BlobCR: efficient checkpoint-restart for HPC applications on IaaS clouds using virtual disk image snapshots

BlobCR: Efficient checkpoint-restart for HPC applications on IaaS clouds using virtual disk image snapshots

LetGo: A Lightweight Continuous Framework for HPC Applications Under Failures

Similar Documents

The design and implementation of checkpoint/restart process fault tolerance for Open MPI (2007)

The design and implementation of checkpoint/restart process tolerance for Open MPI (2007)

ECOFIT: A Framework to Estimate Energy Consumption of Fault Tolerance Protocols for HPC Applications

Coordinated checkpoint/restart process fault tolerance for mpi applications on hpc systems

Coordinated Checkpoint/Restart Process Fault Tolerance for MPI Applications on HPC Systems (2010)

A checkpoint and restart service specification for open mpi (2006)

CRFS: A Lightweight User-Level Filesystem for Generic Checkpoint/Restart

BlobCR: efficient checkpoint-restart for HPC applications on IaaS clouds using virtual disk image snapshots

BlobCR: Efficient checkpoint-restart for HPC applications on IaaS clouds using virtual disk image snapshots

LetGo: A Lightweight Continuous Framework for HPC Applications Under Failures