NDLI: Scalable in situ scientific data encoding for analytical query processing

Please wait, while we are loading the content...

Scalable in situ scientific data encoding for analytical query processing

Virtual TCP offload: optimizing ethernet overlay performance on advanced interconnects

NUMA-aware shared-memory collective communication for MPI

IBIS: interposed big-data I/O scheduler

Anton: a special-purpose machine that achieves a hundred-fold speedup in biomolecular simulations

WuKong: automatically detecting and localizing bugs that manifest at large system scales

A preemption-based runtime to efficiently schedule multi-process applications on heterogeneous clusters with GPUs

Interference and locality-aware task scheduling for MapReduce applications in virtual clusters

Taming massive distributed datasets: data sampling using bitmap indices

Scalanytics: a declarative multi-core platform for scalable composable traffic analytics

Modeling communication in cache-coherent SMP systems: a case-study with Xeon Phi

ACIC: automatic cloud I/O configurator for parallel applications

A 1 PB/s file system to checkpoint three million MPI tasks

On the efficacy of GPU-integrated MPI for scientific applications

A comparative study of high-performance computing on the cloud

I/O acceleration with pattern detection

CamCubeOS: a key-based network stack for 3D torus cluster topologies

Orthrus: a framework for implementing high-performance collective I/O in the multicore clusters

AI-Ckpt: leveraging memory access patterns for adaptive asynchronous incremental checkpointing

VGRIS: virtualized GPU resource isolation and scheduling in cloud gaming

kMemvisor: flexible system wide memory mirroring in virtual environments

MTC envelope: defining the capability of large scale computers in the context of parallel scripting applications

ElastMan: autonomic elasticity manager for cloud-based key-value stores

Correcting soft errors online in LU factorization

COSMIC: middleware for high performance and reliable multiprocessing on xeon phi coprocessors

Supporting parallel soft real-time applications in virtualized environment

Building and scaling virtual clusters with residual resources from interactive clouds

SCDA: SLA-aware cloud datacenter architecture for efficient content storage and retrieval

Load balancing in large-scale epidemiological simulations

Efficient analytics on ordered datasets using MapReduce

A framework for auto-tuning HDF5 applications

Scalable in situ scientific data encoding for analytical query processing

Content Provider	ACM Digital Library
Author	Boyuka, David A. Lakshminarasimhan, Sriram Pendse, Saurabh V. Zou, Xiaocheng Vishwanath, Venkatram Jenkins, John Samatova, Nagiza F. Papka, Michael E.
Abstract	The process of scientific data analysis in high-performance computing environments has been evolving along with the advancement of computing capabilities. With the onset of exascale computing, the increasing gap between compute performance and I/O bandwidth has rendered the traditional method of post-simulation processing a tedious process. Despite the challenges due to increased data production, there exists an opportunity to benefit from "cheap" computing power to perform query-driven exploration and visualization during simulation time. To accelerate such analyses, applications traditionally augment raw data with large indexes, post-simulation, which are then repeatedly utilized for data exploration. However, the generation of current state-of-the-art indexes involve a compute- and memory-intensive processing, thus rendering them inapplicable in an in situ context. In this paper we propose DIRAQ, a parallel in situ, in network data encoding and reorganization technique that enables the transformation of simulation output into a query-efficient form, with negligible runtime overhead to the simulation run. DIRAQ begins with an effective core-local, precision-based encoding approach, which incorporates an embedded compressed index that is 3 -- 6x smaller than current state-of-the-art indexing schemes. DIRAQ then applies an in network index merging strategy, enabling the creation of aggregated indexes ideally suited for spatial-context querying that speed up query responses by up to 10x versus alternative techniques. We also employ a novel aggregation strategy that is topology-, data-, and memory-aware, resulting in efficient I/O and yielding overall end-to-end encoding and I/O time that is less than that required to write the raw data with MPI collective I/O.
Starting Page	1
Ending Page	12
Page Count	12
File Format	PDF
ISBN	9781450319102
DOI	10.1145/2462902.2465527
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2013-06-17
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Compression Exascale computing Query processing Indexing
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

DIRAQ: scalable in situ data- and resource-aware indexing for optimized query performance

Damsel - A Data Model Storage Library for Exascale Science

In-memory Query System for Scientific Dataseis

ISABELA-QA: Query-driven analytics with ISABELA-compressed extreme-scale scientific data

A scalable XML indexing method using MapReduce

Exploiting locality for query processing and compression in scientific databases

Parallel in situ indexing for data-intensive computing

Parallel query evaluation as a Scientific Data Service

An efficient and compact indexing scheme for large-scale data store

Scalable in situ scientific data encoding for analytical query processing

Similar Documents

DIRAQ: scalable in situ data- and resource-aware indexing for optimized query performance

Damsel - A Data Model Storage Library for Exascale Science

In-memory Query System for Scientific Dataseis

ISABELA-QA: Query-driven analytics with ISABELA-compressed extreme-scale scientific data

A scalable XML indexing method using MapReduce

Exploiting locality for query processing and compression in scientific databases

Parallel in situ indexing for data-intensive computing

Parallel query evaluation as a Scientific Data Service

An efficient and compact indexing scheme for large-scale data store

Scalable in situ scientific data encoding for analytical query processing