NDLI: Design and evaluation of a novel dataflow based bigdata solution

Please wait, while we are loading the content...

Energy efficiency and performance frontiers for sparse computations on GPU supercomputers

Energy-efficient computing for HPC workloads on heterogeneous manycore chips

A performance study of Java garbage collectors on multicore architectures

Toward an evolutionary task parallel integrated MPI + X programming model

Design and evaluation of a novel dataflow based bigdata solution

Programming support for reconfigurable custom vector architectures

Thread-level parallelization and optimization of NWChem for the Intel MIC architecture

Parallelism vs. speculation: exploiting speculative genetic algorithm on GPU

GPU technology applied to reverse time migration and seismic modeling via OpenACC

Parallelizing a discrete event simulation application using the Habanero-Java multicore library

RaftLib: a C++ template library for high performance stream parallel processing

A Java util concurrent park contention tool

Debugging parallel programs using fork handlers

Effective communication for a system of cluster-on-a-chip processors

Exploiting communication concurrency on high performance computing systems

Patty: a pattern-based parallelization tool for the multicore age

Deadlock-free buffer configuration for stream computing

Supporting multiple accelerators in high-level programming models

Design and evaluation of a novel dataflow based bigdata solution

Content Provider	ACM Digital Library
Author	Heilig, Brian Gao, Guang R. Wu, Yao Zheng, Long
Abstract	As the attention given to big data grows, cluster computing systems for distributed processing of large data sets become the mainstream and critical requirement in high performance distributed system research. One of the most successful system is Hadoop which uses MapReduce as programming/execution model and takes disks as intermedia to process huge volume of data. However, currently, it is a consensus that Hadoop is not the final solution to BigData due to MapReduce programming model, disk based computing, synchronous execution model and the constraint that only supports batch processing, and so on. A new solution, especially, a fundamental evolution is needed to bring BigData solution into a new era. In this paper, we introduce a new cluster computing system called HAMR which supports both batch and streaming processing. To achieve better performance, HAMR integrates HPC approaches, i.e. DataFlow fundamental into a big data solution. With more speicifications, HAMR is fully designed based on In-Memory computing to reduce the unnecessary disk access overhead; task scheduling and memory management are in fine-grain manner to explore more parallelism; asynchronous execution improves efficiency of computation resource usage, and furtherly makes workload balance across the whole cluster better. The experimental results show that HAMR can outperform Hadoop by up to 10x in the same cluster environment.
Starting Page	40
Ending Page	48
Page Count	9
File Format	PDF
ISBN	9781450334044
DOI	10.1145/2712386.2712397
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2015-02-07
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	In-memory processing Bigdata Dataflow Fine-grain
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in