NDLI: Scalable hybrid stream and hadoop network analysis system

Please wait, while we are loading the content...

Extreme big data processing in large-scale graph analytics and billion-scale social simulation

Uncertainties in the modeling of self-adaptive systems: a taxonomy and an example of availability evaluation

Speeding up processing data from millions of smart meters

Efficient optimization of software performance models via parameter-space pruning

The taming of the shrew: increasing performance by automatic parameter tuning for java garbage collectors

Engineering resource management middleware for optimizing the performance of clouds processing mapreduce jobs with deadlines

Understanding, modelling, and improving the performance of web applications in multicore virtualised environments

LIMBO: a tool for modeling variable load intensities

System performance analyses through object-oriented fault and coupling prisms

Application performance management using learning, optimization, and control

Software contention aware queueing network model of three-tier web systems

Modelling database lock-contention in architecture-level performance simulation

On the limits of modeling generational garbage collector performance

Automated analysis of performance and energy consumption for cloud applications

Exploring synergies between bottleneck analysis and performance antipatterns

Constructing performance model of JMS middleware platform

A power-measurement methodology for large-scale, high-performance computing

A meta-controller method for improving run-time self-architecting in SOA systems

An evaluation of zookeeper for high availability in system S

LibReDE: a library for resource demand estimation

Run-time performance optimization of a BigData query language

Efficient and accurate stack trace sampling in the Java hotspot virtual machine

Benchmarking graph-processing platforms: a vision

An experimental methodology to evaluate energy efficiency and performance in an enterprise virtualized environment

Adaptive model learning for continual verification of non-functional properties

Agile middleware for scheduling: meeting competing performance requirements of diverse tasks

Scalable hybrid stream and hadoop network analysis system

Server efficiency rating tool (SERT) 1.0.2: an overview

Model-driven engineering in practice: integrated performance decision support for process-centric business impact analysis

PowerPerfCenter: a power and performance prediction tool for multi-tier applications

Real-time multi-cloud management needs application awareness

Performance queries for architecture-level performance models

SPECjbb2013 1.0: an overview

Continuous validation of load test suites

Scalable hybrid stream and hadoop network analysis system

Content Provider	ACM Digital Library
Author	Bumgardner, Vernon K.C. Marek, Victor W.
Abstract	Collections of network traces have long been used in network traffic analysis. Flow analysis can be used in network anomaly discovery, intrusion detection and more generally, discovery of actionable events on the network. The data collected during processing may be also used for prediction and avoidance of traffic congestion, network capacity planning, and the development of software-defined networking rules. As network flow rates increase and new network technologies are introduced on existing hardware platforms, many organizations find themselves either technically or financially unable to generate, collect, and/or analyze network flow data. The continued rapid growth of network trace data, requires new methods of scalable data collection and analysis. We report on our deployment of a system designed and implemented at the University of Kentucky that supports analysis of network traffic across the enterprise. Our system addresses problems of scale in existing systems, by using distributed computing methodologies, and is based on a combination of stream and batch processing techniques. In addition to collection, stream processing using Storm is utilized to enrich the data stream with ephemeral environment data. Enriched stream-data is then used for event detection and near real-time flow analysis by an in-line complex event processor. Batch processing is performed by the Hadoop MapReduce framework, from data stored in HBase BigTable storage. In benchmarks on our 10 node cluster, using actual network data, we were able to stream process over 315k flows/sec. In batch analysis were we able to process over 2.6M flows/sec with a storage compression ratio of 6.7:1.
Starting Page	219
Ending Page	224
Page Count	6
File Format	PDF
ISBN	9781450327336
DOI	10.1145/2568088.2568103
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2014-03-22
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Netflow
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A Hybrid Scheduling Approach for Scalable Heterogeneous Hadoop Systems

A hybrid scheduling approach for scalable heterogeneous hadoop systems.

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

Netflow based system for NAT detection

Scalable analysis of network measurements with Hadoop and Pig

Exploring Netflow Data using Hadoop

Oozie: towards a scalable workflow management system for Hadoop

Report of the second workshop on the usage of netflow/ipfix in network management (2010).

Scalable hybrid stream and hadoop network analysis system

Similar Documents

A Hybrid Scheduling Approach for Scalable Heterogeneous Hadoop Systems

A hybrid scheduling approach for scalable heterogeneous hadoop systems.

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

Toward scalable internet traffic measurement and analysis with Hadoop (CCRV, Vol. 43, Issue. 1)

Netflow based system for NAT detection

Scalable analysis of network measurements with Hadoop and Pig

Exploring Netflow Data using Hadoop

Oozie: towards a scalable workflow management system for Hadoop

Report of the second workshop on the usage of netflow/ipfix in network management (2010).

Scalable hybrid stream and hadoop network analysis system