NDLI: A Succinct Distributive Big Data Clustering Algorithm Based on Local-Remote Coordination

Content Provider	IEEE Xplore Digital Library
Author	Chao Ma Xun Liang Yuefeng Ma
Copyright Year	2015
Description	Author affiliation: Sch. of Inf., Renmin Univ. of China, Beijing, China (Chao Ma; Xun Liang; Yuefeng Ma)
Abstract	Mining global patterns on big data distributed in many remote locations is a challenging task since transmitting big data in different remote data servers to the central server is prohibitively expensive. In this paper, we present a succinct distributive big data clustering algorithm based on local-remote coordination (DBDC-LRC) that aims to reduce the cost of big data transmission while maintaining an acceptable overall clustering accuracy. The algorithm is divided into three phases. In the first phase, the idea of Canopy algorithm is improved in the search for representative points with a clustering assumption that the decision boundary should lie in a low-density region, during which controllable thresholds are optimized. Noticing that in data mining a hyperellipsoid is more adaptable in shaping unknown data than a hypercube, we employ Mahalanobis distance as opposed to the Euclidean distance in determining the representative points in different remote data servers. Given that only a limited number of representative points, instead of all the remote data, are transmitted to the central server for clustering, the transmitting cost is reduced significantly. In the second phase, a weighted clustering method is used in mining the global patterns from the gathered information of representative points from various remote data servers. In the third phase, the mined global patterns are sent back to the original remote server and the related data are labeled with the same patterns according their representative points nearby. In this phase, Bayesian method is used to resolve the conflicts that one point is covered by several representative points in its neighborhood. Experiments show that DBDC-LRC is highly suitable for mining patterns from distributive big data, and the advantages of this approach include low cost, high accuracy, high robustness, and good expansibility.
Starting Page	1839
Ending Page	1844
File Size	323777
Page Count	6
File Format	PDF
e-ISBN	9781479986972
DOI	10.1109/SMC.2015.322
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2015-10-09
Publisher Place	China
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Clustering algorithms Big data Servers Covariance matrices Data mining Euclidean distance Bayesian method DBDC-LRC representative points local Mahalanobis distance weighted clustering Canopy algorithm
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Intelligent data cluster algorithm based on hierarchical FCM and Mahalanobis distance

Navigation pattern discovery on Web site based on the distance between sequences

Clustering Algorithms Based on Mahalanobis Distances

Study on a modified Fuzzy C-Means Clustering Algorithm

Hybrid clustering algorithm based on mahalanobis distance and mst.

Weighted k-Means Algorithm Based Text Clustering

DBDC: Density Based Distributed Clustering (2004)

Probabilistic-fuzzy clustering algorithm

Fuzzy Partition Clustering Algorithms Based on Alternative Mahalanobis Distances (2007)

A Succinct Distributive Big Data Clustering Algorithm Based on Local-Remote Coordination

Similar Documents

Intelligent data cluster algorithm based on hierarchical FCM and Mahalanobis distance

Navigation pattern discovery on Web site based on the distance between sequences

Clustering Algorithms Based on Mahalanobis Distances

Study on a modified Fuzzy C-Means Clustering Algorithm

Hybrid clustering algorithm based on mahalanobis distance and mst.

Weighted k-Means Algorithm Based Text Clustering

DBDC: Density Based Distributed Clustering (2004)

Probabilistic-fuzzy clustering algorithm

Fuzzy Partition Clustering Algorithms Based on Alternative Mahalanobis Distances (2007)

A Succinct Distributive Big Data Clustering Algorithm Based on Local-Remote Coordination