NDLI: Distributed log information processing with Map-Reduce: A case study from raw data to final models

Content Provider	IEEE Xplore Digital Library
Author	Mingyue Luo Gang Liu
Copyright Year	2010
Description	Author affiliation: School of Electronic Engineering, Beijing University of Posts and Telecommunications, China (Mingyue Luo; Gang Liu)
Abstract	With the high development of Internet, e-commerce websites now routinely have to work with log datasets which are up to a few terabytes in size. How to remove messy data timely with low cost and find out useful information is a problem we have to face. The mining process involves several steps from pre-processing the raw data to establishing the final models. In this paper we describe our method to solve the problem with Map-Reduce. Hadoop[7] is a Map-Reduce implementation develops open-source software for reliable, scalable, distributed computing. Several applications which we have proposed: data extracting, sum operation, join operation and clustering algorithm are applied on hadoop. We can apply them on data pre-processing and detect users with the same interests. In particular, we focus on clustering algorithms. A clustering algorithms which integrate SOM(Self-Organized Map) and fuzzy[13] logic is combined with Map-Reduce and we call it MRSF here. With the help of hadoop cluster, large calculation of jobs with MRSF can be accommodated easily by just adding more nodes or computers to the cluster. From the experiment, we show that MRSF can scale well and efficiently process and analyze extremely large datasets.
Starting Page	1143
Ending Page	1146
File Size	144102
Page Count	4
File Format	PDF
ISBN	9781424469420
e-ISBN	9781424469437
DOI	10.1109/ICITIS.2010.5689760
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2010-12-17
Publisher Place	China
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Training Computers Computational modeling Clustering algorithms Distributed databases Join operation Data models Map-Reduce Data pre-processing Data mining Distributed Data Mining
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Parallel Simultaneous Co-clustering and Learning with Map-Reduce

A 2-Tier Clustering Algorithm with Map-Reduce

Research of Cloud Computing Based on the Hadoop Platform

A Distributed Data Mining Framework Accelerated with Graphics Processing Units

An efficient improved join algorithm using map reduce in hadoop

A novel algorithm for distributed data mining in HDFS

DisCo: Distributed Co-clustering with Map-Reduce: A Case Study towards Petabyte-Scale End-to-End Mining

Research of Massive Web Log Data Mining Based on Cloud Computing

A Method of Data Distribution for Distributed Cross Join

Distributed log information processing with Map-Reduce: A case study from raw data to final models

Similar Documents

Parallel Simultaneous Co-clustering and Learning with Map-Reduce

A 2-Tier Clustering Algorithm with Map-Reduce

Research of Cloud Computing Based on the Hadoop Platform

A Distributed Data Mining Framework Accelerated with Graphics Processing Units

An efficient improved join algorithm using map reduce in hadoop

A novel algorithm for distributed data mining in HDFS

DisCo: Distributed Co-clustering with Map-Reduce: A Case Study towards Petabyte-Scale End-to-End Mining

Research of Massive Web Log Data Mining Based on Cloud Computing

A Method of Data Distribution for Distributed Cross Join

Distributed log information processing with Map-Reduce: A case study from raw data to final models