NDLI: S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters

Please wait, while we are loading the content...

It's Time for a New Old Language

EffiSha: A Software Framework for Enabling Effficient Preemptive Scheduling of GPU

Checking Concurrent Data Structures Under the C/C++11 Memory Model

Thread Data Sharing in Cache: Theory and Measurement

Simple, Accurate, Analytical Time Modeling and Optimal Tile Size Selection for GPGPU Stencils

Pagoda: Fine-Grained GPU Resource Virtualization for Narrow Tasks

SC-Haskell: Sequential Consistency in Languages That Minimize Mutable Shared Heap

Using Butterfly-Patterned Partial Sums to Draw from Discrete Distributions

Self-Checkpoint: An In-Memory Checkpoint Method Using Less Space and Its Practice on Fault-Tolerant HPL

POSTER: Reuse, don't Recycle: Transforming Algorithms that Throw Away Descriptors

Layout Lock: A Scalable Locking Paradigm for Concurrent Data Layout Modifications

An Efficient Abortable-locking Protocol for Multi-level NUMA Systems

Exploiting Vector and Multicore Parallelism for Recursive, Data- and Task-Parallel Programs

Combining SIMD and Many/Multi-core Parallelism for Finite State Machines with Enumerative Speculation

Groute: An Asynchronous Multi-GPU Programming Model for Irregular Computations

Synchronized-by-Default Concurrency for Shared-Memory Systems

KiWi: A Key-Value Map for Scalable Real-Time Analytics

Silent Data Corruption Resilient Two-sided Matrix Factorizations

POSTER: An Architecture and Programming Model for Accelerating Parallel Commutative Computations via Privatization

Understanding the GPU Microarchitecture to Achieve Bare-Metal Performance Tuning

Contention in Structured Concurrency: Provably Efficient Dynamic Non-Zero Indicators for Nested Parallelism

Isoefficiency in Practice: Configuring and Understanding the Performance of Task-based Applications

S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters

Tapir: Embedding Fork-Join Parallelism into LLVM's Intermediate Representation

Function Call Re-Vectorization

Grammar-aware Parallelization for Scalable XPath Querying

POSTER: HythTM: Extending the Applicability of Intel TSX Hardware Transactional Support

Noise Injection Techniques to Expose Subtle and Unintended Message Races

Processor-Oblivious Record and Replay

Model-based Iterative CT Image Reconstruction on GPUs

A Multicore Path to Connectomics-on-Demand

Optimizing the Four-Index Integral Transform Using Data Movement Lower Bounds Analysis

Eunomia: Scaling Concurrent Search Trees under Contention Using HTM

POSTER: Provably Efficient Scheduling of Cache-Oblivious Wavefront Algorithms

POSTER: State Teleportation via Hardware Transactional Memory

POSTER: IOGP: An Incremental Online Graph Partitioning for Large-Scale Distributed Graph Databases

POSTER: Distributed Control: The Benefits of Eliminating Global Synchronization via Effective Scheduling

POSTER: MAPA: An Automatic Memory Access Pattern Analyzer for GPU Applications

POSTER: Cache-Oblivious MPI All-to-All Communications on Many-Core Architectures

POSTER: Automated Load Balancer Selection Based on Application Characteristics

POSTER: A GPU-Friendly Skiplist Algorithm

POSTER: Poor Man's URCU

POSTER: A Wait-Free Queue with Wait-Free Memory Reclamation

POSTER: STAR (Space-Time Adaptive and Reductive) Algorithms for Real-World Space-Time Optimality

POSTER: Recovering Performance for Vector-based Machine Learning on Managed Runtime

POSTER: On the Problem of Consistency Exceptions in the Context of Strong Memory Models

POSTER: An Infrastructure for HPC Knowledge Sharing and Reuse

S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters

Content Provider	ACM Digital Library
Author	Hamidouche, Khaled Panda, Dhabaleswar K. Awan, Ammar Ahmad Hashmi, Jahanzeb Maqbool
Abstract	Availability of large data sets like ImageNet and massively parallel computation support in modern HPC devices like NVIDIA GPUs have fueled a renewed interest in Deep Learning (DL) algorithms. This has triggered the development of DL frameworks like Caffe, Torch, TensorFlow, and CNTK. However, most DL frameworks have been limited to a single node. In order to scale out DL frameworks and bring HPC capabilities to the DL arena, we propose, S-Caffe; a scalable and distributed Caffe adaptation for modern multi-GPU clusters. With an in-depth analysis of new requirements brought forward by the DL frameworks and limitations of current communication runtimes, we present a co-design of the Caffe framework and the MVAPICH2-GDR MPI runtime. Using the co-design methodology, we modify Caffe's workflow to maximize the overlap of computation and communication with multi-stage data propagation and gradient aggregation schemes. We bring DL-Awareness to the MPI runtime by proposing a hierarchical reduction design that benefits from CUDA-Aware features and provides up to a massive 133x speedup over OpenMPI and 2.6x speedup over MVAPICH2 for 160 GPUs. S-Caffe successfully scales up to 160 K-80 GPUs for GoogLeNet (ImageNet) with a speedup of 2.5x over 32 GPUs. To the best of our knowledge, this is the first framework that scales up to 160 GPUs. Furthermore, even for single node training, S-Caffe shows an improvement of 14\% and 9\% over Nvidia's optimized Caffe for 8 and 16 GPUs, respectively. In addition, S-Caffe achieves up to 1395 samples per second for the AlexNet model, which is comparable to the performance of Microsoft CNTK.
Starting Page	193
Ending Page	205
Page Count	13
File Format	PDF
ISBN	9781450344937
DOI	10.1145/3018743.3018769
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2017-01-26
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Deep learning Mpi\_reduce Distributed training Cuda-aware mpi Caffe
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Design of a Hybrid MPI-CUDA Benchmark Suite for CPU-GPU Clusters

Designing efficient small message transfer mechanism for inter-node MPI communication on InfiniBand GPU clusters

Efficient Large Message Broadcast using NCCL and CUDA-Aware MPI for Deep Learning

MetaMorph: a library framework for interoperable kernels on multi- and many-core clusters

A CUDA-MPI Hybrid Bitonic Sorting Algorithm for GPU Clusters

Designing Deep Learning Neural Networks using Caffe

Asynchronous Communication for Finite-Difference Simulations on GPU Clusters using CUDA and MPI

Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation

CUDA-MPI Implementation of Fast Multipole Method on GPU Clusters for Dielectric Objects

S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters

Similar Documents

Design of a Hybrid MPI-CUDA Benchmark Suite for CPU-GPU Clusters

Designing efficient small message transfer mechanism for inter-node MPI communication on InfiniBand GPU clusters

Efficient Large Message Broadcast using NCCL and CUDA-Aware MPI for Deep Learning

MetaMorph: a library framework for interoperable kernels on multi- and many-core clusters

A CUDA-MPI Hybrid Bitonic Sorting Algorithm for GPU Clusters

Designing Deep Learning Neural Networks using Caffe

Asynchronous Communication for Finite-Difference Simulations on GPU Clusters using CUDA and MPI

Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation

CUDA-MPI Implementation of Fast Multipole Method on GPU Clusters for Dielectric Objects

S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters