NDLI: An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment

Content Provider	Springer Nature Link
Author	Shi, Weiwei Zhu, Yongxin Huang, Tian Sheng, Gehao Lian, Yong Wang, Guoxing Chen, Yufeng
Copyright Year	2016
Abstract	Big data techniques have been applied to power grid for the prediction and evaluation of grid conditions. However, the raw data quality can rarely meet the requirement of precise data analytics since raw data set usually contains samples with missing data to which the common data mining models are sensitive. Besides, the raw training data from a single monitoring system, e.g. dissolved gas analysis (DGA), are rarely sufficient for training in the form of valid instances since raw data set usually contains samples with noisy data. Though classic methods like neural network can be used to fill the gaps of missing data and classify the fault type, their models often fail to fit the rules of power grid conditions. This paper presents an integrated data preprocessing framework (DPF) based on Apache Spark to improve the prediction accuracy for data sets with missing data points and classification accuracy with noise data as well as to meet the big data requirement, which mainly combines missing data prediction, data fusion, data cleansing and fault type classification. First, the prediction model is trained based on the linear regression (LinR). Afterwards, we propose an optimized linear method (OLR) to improve the prediction accuracy. Then, to better utilize the strong correlation among different data sources, new data features extracted by persons correlation coefficient (PCC) are fused into a training data set. Next, principal component analysis (PCA) is taken to reduce the side effect brought by the new feature as well as retaining significant information for classification. Finally, the classification model based on logistic regression (LogR) and support vector machine (SVM) is trained to classify the fault type of electric equipment. We test the DPF framework on missing data prediction and fault type classification of power transformers in power grid system. The experimental results show that the predictors based on the proposed framework achieve lower mean square error and the classifiers obtain higher accuracy than traditional ones. Besides, the training time required for training large-scale data shows a decreasing trend. Therefore, the data preprocessing framework DPF would be a good candidate to predict the missing data and classify the fault type in power grid system.
Starting Page	221
Ending Page	236
Page Count	16
File Format	PDF
ISSN	19398018
Journal	Journal of Signal Processing Systems
Volume Number	86
Issue Number	2-3
e-ISSN	19398115
Language	English
Publisher	Springer US
Publisher Date	2016-03-02
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Big data Apache spark Framework Missing data prediction Fault diagnose Signal,Image and Speech Processing Circuits and Systems Electrical Engineering Image Processing and Computer Vision Pattern Recognition Computer Imaging, Vision, Pattern Recognition and Graphics
Content Type	Text
Resource Type	Article
Subject	Theoretical Computer Science Signal Processing Control and Systems Engineering Information Systems Modeling and Simulation Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Column Access-aware In-stream Data Cache with Stream Processing Framework

Missing Data Imputation for Time-Frequency Representations of Audio Signals

Feature Fusion Applied to Missing Data ASR with the Combination of Recognizers

Instruction-Level Fault Tolerance Configurability

Real Time Diagnosis & Fault Detection for the Reliability Improvement of the Embedded Systems

Pipelined HAC Estimation Engines for Multivariate Time Series

The Role of Context in Cognitive Systems

Speech Enhancement Based on Analysis–Synthesis Framework with Improved Parameter Domain Enhancement

A VLSI 8 × 8 MIMO Near-ML Detector with Preprocessing

An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment

Similar Documents

Column Access-aware In-stream Data Cache with Stream Processing Framework

Missing Data Imputation for Time-Frequency Representations of Audio Signals

Feature Fusion Applied to Missing Data ASR with the Combination of Recognizers

Instruction-Level Fault Tolerance Configurability

Real Time Diagnosis & Fault Detection for the Reliability Improvement of the Embedded Systems

Pipelined HAC Estimation Engines for Multivariate Time Series

The Role of Context in Cognitive Systems

Speech Enhancement Based on Analysis–Synthesis Framework with Improved Parameter Domain Enhancement

A VLSI 8 × 8 MIMO Near-ML Detector with Preprocessing

An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment