NDLI: System anomaly detection in distributed systems through MapReduce-Based log analysis

Content Provider	IEEE Xplore Digital Library
Author	Yan Liu Wei Pan Ning Cao Guangwei Qiao
Copyright Year	2010
Description	Author affiliation: Ideal Institute of Information and Technology, Changchun, China (Yan Liu; Ning Cao; Guangwei Qiao) \|\| College of Information Science and Engineering, Northeastern University, Shenyang, China (Wei Pan)
Abstract	System anomaly detection is very important for development, maintenance and performance refinement in large scale distributed systems. It's a good way to obtain the troubleshooting and problem diagnosis by analyzing system logs produced by distributed systems. However, due to the increasing scale and complexity of distributed systems, the size of logs must be very large. Thus, it's inefficient for common methods to analyze system logs on single node. Therefore, there is a great demand to adopt a distributed method for anomaly detection techniques based on log analysis. In this paper, a MapReduce-Based Framework is implemented to analyze the distributed log for detecting anomaly. The framework is built on top of Hadoop, an open source distributed file system and MapReduce implementation. We first make use of Random Access File to realize an incremental way for aggregating system logs from each node of the monitored cluster, and collect them to the analysis cluster. Then, we apply the K-means clustering algorithm to integrate the collected logs. After that, we implement a MapReduce-Based algorithm to parser these clustered log files. Furthermore, in order to make the best use of this collected data, a flexible and powerful way is utilized to display monitoring and analysis results. Thus, we can monitor system status of large distributed cluster and detect its anomalies.
File Size	932150
File Format	PDF
ISBN	9781424465392
ISSN	21547491
e-ISBN	9781424465422
DOI	10.1109/ICACTE.2010.5579173
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2010-08-20
Publisher Place	China
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Algorithm design and analysis K-means Distributed databases Clustering algorithms Programming distributed system anomaly detection Data mining log analysis Monitoring Graphical user interfaces MapReduce
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

MapReduce Design of K-Means Clustering Algorithm

Intrusion detection based on MinMax K-means clustering

Distributed clustering algorithm for spatial data mining

An Optimal Distributed K-Means Clustering Algorithm Based on CloudStack

An optimal distributed K-Means clustering algorithm based on cloudstack

Research and improve on K-means algorithm based on hadoop

Research and Implementation of an Anomaly Detection Model Based on Clustering Analysis

Optimization of the distributed K-means clustering algorithm based on set pair analysis

An analysis of suitable parameters for efficiently applying K-means clustering to large TCPdump data set using Hadoop framework

System anomaly detection in distributed systems through MapReduce-Based log analysis

Similar Documents

MapReduce Design of K-Means Clustering Algorithm

Intrusion detection based on MinMax K-means clustering

Distributed clustering algorithm for spatial data mining

An Optimal Distributed K-Means Clustering Algorithm Based on CloudStack

An optimal distributed K-Means clustering algorithm based on cloudstack

Research and improve on K-means algorithm based on hadoop

Research and Implementation of an Anomaly Detection Model Based on Clustering Analysis

Optimization of the distributed K-means clustering algorithm based on set pair analysis

An analysis of suitable parameters for efficiently applying K-means clustering to large TCPdump data set using Hadoop framework

System anomaly detection in distributed systems through MapReduce-Based log analysis