NDLI: Distributed Diversification of Large Datasets

Content Provider	IEEE Xplore Digital Library
Author	Hasan, M. Mueen, A. Tsotras, V.
Copyright Year	2014
Description	Author affiliation: Univ. of New Mexico, Albuquerque, NM, USA (Mueen, A.) \|\| Univ. of California, Riverside, Riverside, CA, USA (Hasan, M.; Tsotras, V.)
Abstract	Diversification has been recently proposed as an approach to allow a user to better grasp a large result set without having to look through all relevant results. In this paper, we expand the use of diversification as an analytical tool to explore large datasets dispersed over many nodes. The diversification problem is in general NP-complete and existing uniprocessor algorithms are unfortunately not suitable for the distributed setting of our environment. Using the MapReduce framework we consider two distinct approaches to solve the distributed diversification problem, one that focuses on optimizing disk I/O and one that optimizes for network I/O. Our approaches are iterative in nature, allowing the user to continue refining the diversification process if more time is available. Moreover, we prove that (i) this iteration process converges and (ii) it produces a 2-approximate diversified result set when compared to the optimal solution. We also develop a cost model to predict the run-time for both approaches based on the network and disk characteristics. We implemented our approaches on a cluster of 40 cores and showed that they are scalable and produce the same quality results as the state-of-the-art uniprocessor algorithms.
Starting Page	67
Ending Page	76
File Size	340012
Page Count	10
File Format	PDF
ISBN	9781479937660
DOI	10.1109/IC2E.2014.19
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-03-11
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Computational modeling Clustering algorithms Diversity Approximation algorithms Data models Parallel Processing Partitioning algorithms Iterative methods Cost benefit analysis MapReduce
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

DBSCAN on Resilient Distributed Datasets

Research on algorithm of point cloud MapReduce registration

Distributed, Scalable Clustering for Detecting Halos in Terascale Astronomy Datasets

Comparison of a sequential and a MapReduce approach to joining large datasets

A parallel computing model for large-graph mining with MapReduce

A stream partitioning approach to processing large scale distributed graph datasets

A Comparative Analysis of Distributed Clustering Algorithms: A Survey

Exploring Computation Locality of Graph Mining Algorithms on MapReduce

Efficient Similarity Joins on Massive High-Dimensional Datasets Using MapReduce

Distributed Diversification of Large Datasets

Similar Documents

DBSCAN on Resilient Distributed Datasets

Research on algorithm of point cloud MapReduce registration

Distributed, Scalable Clustering for Detecting Halos in Terascale Astronomy Datasets

Comparison of a sequential and a MapReduce approach to joining large datasets

A parallel computing model for large-graph mining with MapReduce

A stream partitioning approach to processing large scale distributed graph datasets

A Comparative Analysis of Distributed Clustering Algorithms: A Survey

Exploring Computation Locality of Graph Mining Algorithms on MapReduce

Efficient Similarity Joins on Massive High-Dimensional Datasets Using MapReduce

Distributed Diversification of Large Datasets