NDLI: Statistical technique for online anomaly detection using Spark over heterogeneous data from multi-source VMware performance data

Content Provider	IEEE Xplore Digital Library
Author	Solaimani, M. Iftekhar, M. Khan, L. Thuraisingham, B.
Copyright Year	2014
Description	Author affiliation: Dept. of Comput. Sci., Univ. of Texas at Dallas, Richardson, TX, USA (Solaimani, M.; Iftekhar, M.; Khan, L.; Thuraisingham, B.)
Abstract	Anomaly detection refers to the identification of patterns in a dataset that do not conform to expected patterns. Depending on the domain, the non-conformant patterns are assigned various tags, e.g. anomalies, outliers, exceptions, malwares and so forth. Online anomaly detection aims to detect anomalies in data flowing in a streaming fashion. Such stream data is commonplace in today's cloud data centers that house a large array of virtual machines(VM) producing vast amounts of performance data in real-time. Sophisticated detection mechanism will likely entail collation of data from heterogeneous sources with diversified data format and semantics. Therefore, detection of performance anomaly in this context requires a distributed framework with high throughput and low latency. Apache Spark is one such framework that represents the bleeding-edge amongst its contemporaries. In this paper, we have taken up the challenge of anomaly detection in VMware based cloud data centers. We have employed a Chi-square based statistical anomaly detection technique in Spark. We have demonstrated how to take advantage of the high processing power of Spark to perform anomaly detection on heterogeneous data using statistical techniques. Our approach is optimally designed to cope with the heterogeneity of input data streams and the experiments we conducted testify to its efficacy in online anomaly detection.
Starting Page	1086
Ending Page	1094
File Size	887499
Page Count	9
File Format	PDF
ISBN	9781479956661
DOI	10.1109/BigData.2014.7004343
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-10-27
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Computational modeling Radiation detectors Online anomaly detection Virtual machining Data center Sparks Data mining Anomaly detection Apache Spark Real-time anomaly detection Distributed databases Real-time systems Chi-square test
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Statistical Techniques For Real-time Anomaly Detection Using Spark Over Multi-source VMware Performance Data

Spark-based anomaly detection over multi-source VMware performance data in real-time

Spark-based Anomaly Detection Over Multi-source VMware Performance Data In Real-time.

Distributed Entropy Minimization Discretizer for Big Data Analysis under Apache Spark

Pipeline for Real-time Anomaly Detection in Log Data Streams using Apache Kafka and Apache Spark

Network anomalies detection using statistical technique: a chi- square approach.

Efficient Distributed Data Clustering on Spark

Chi-Square Test Based Decision Trees Induction in Distributed Environment

Distributed Discord Discovery: Spark Based Anomaly Detection in Time Series

Statistical technique for online anomaly detection using Spark over heterogeneous data from multi-source VMware performance data

Similar Documents

Statistical Techniques For Real-time Anomaly Detection Using Spark Over Multi-source VMware Performance Data

Spark-based anomaly detection over multi-source VMware performance data in real-time

Spark-based Anomaly Detection Over Multi-source VMware Performance Data In Real-time.

Distributed Entropy Minimization Discretizer for Big Data Analysis under Apache Spark

Pipeline for Real-time Anomaly Detection in Log Data Streams using Apache Kafka and Apache Spark

Network anomalies detection using statistical technique: a chi- square approach.

Efficient Distributed Data Clustering on Spark

Chi-Square Test Based Decision Trees Induction in Distributed Environment

Distributed Discord Discovery: Spark Based Anomaly Detection in Time Series

Statistical technique for online anomaly detection using Spark over heterogeneous data from multi-source VMware performance data