NDLI: Datix:A System for Scalable Network Analytics

Please wait, while we are loading the content...

Datix:A System for Scalable Network Analytics

Content Provider	ACM Digital Library
Author	Papailiou, Nikolaos Smaragdakis, Georgios Konstantinou, Ioannis Sarlis, Dimitrios Koziris, Nectarios
Abstract	The ever-increasing Internet traffic poses challenges to network operators and administrators that have to analyze large network datasets in a timely manner to make decisions regarding network routing, dimensioning, accountability and security. Network datasets collected at large networks such as Internet Service Providers (ISPs) or Internet Exchange Points (IXPs) can be in the order of Terabytes per hour. Unfortunately, most of the current network analysis approaches are ad-hoc and centralized, and thus not scalable. In this paper, we present Datix, a fully decentralized, open-source analytics system for network traffic data that relies on smart partitioning storage schemes to support fast join algorithms and efficient execution of filtering queries. We outline the architecture and design of Datix and we present the evaluation of Datix using real traces from an operational IXP. Datix is a system that deals with an important problem in the intersection of data management and network monitoring while utilizing state-of-the-art distributed processing engines. In brief, Datix manages to efficiently answer queries within minutes compared to more than 24 hours processing when executing existing Python based code in single node setups. Datix also achieves nearly 70% speedup compared to baseline query implementations of popular big data analytics engines such as Hive and Shark.
Starting Page	21
Ending Page	28
Page Count	8
File Format	PDF
ISSN	01464833
DOI	10.1145/2831347.2831351
Journal	ACM SIGCOMM Computer Communication Review (CCRV)
Volume Number	45
Issue Number	5
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	1993-07-01
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Sflow K-d tree Hadoop Map-join Mapreduce Hbase
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

SmartJoin: a network-aware multiway join for MapReduce

Scatter-Gather-Merge: An efficient star-join query processing algorithm for data-parallel frameworks

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

A MapReduce task scheduling algorithm for deadline constraints

An efficient PAM spatial clustering algorithm based on MapReduce

Study on QC-Tree with MapReduce and Hbase in Hadoop

Scalable P2P bot detection system based on network data stream

Fluid Petri Nets for the Performance Evaluation of MapReduce and Spark Applications

Datix:A System for Scalable Network Analytics

Similar Documents

SmartJoin: a network-aware multiway join for MapReduce

Scatter-Gather-Merge: An efficient star-join query processing algorithm for data-parallel frameworks

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

A MapReduce task scheduling algorithm for deadline constraints

An efficient PAM spatial clustering algorithm based on MapReduce

Study on QC-Tree with MapReduce and Hbase in Hadoop

Scalable P2P bot detection system based on network data stream

Fluid Petri Nets for the Performance Evaluation of MapReduce and Spark Applications

Datix:A System for Scalable Network Analytics