NDLI: Fast and scalable vector similarity joins with MapReduce

Content Provider	Springer Nature Link
Author	Yang, Byoungju Kim, Hyun Joon Shim, Junho Lee, Dongjoo Lee, Sang goo
Copyright Year	2015
Abstract	Vector similarity join, which finds similar pairs of vector objects, is a computationally expensive process. As its number of vectors increases, the time needed for join operation increases proportional to the square of the number of vectors. Various filtering techniques have been proposed to reduce its computational load. On the other hand, MapReduce algorithms have been studied to manage large datasets. The recent improvements, however, still suffer from its computational time and scalability. In this paper, we propose a MapReduce algorithm FACET(FAst and sCalable maprEduce similariTy join) to efficiently solve the vector similarity join problem on large datasets. FACET is an all-pair exact join algorithm, composed of two stages. In the first stage, we use our own novel filtering techniques to eliminate dissimilar pairs to generate non-redundant candidate pairs. The second stage matches candidate pairs with the vector data so that similar pairs are produced as the output. Both stages employ parallelism offered by MapReduce. The algorithm is currently designed for cosine similarity and Self Join case. Extensions to other similarity measures and R-S Join case are also discussed. We provide the I/O analysis of the algorithm. We evaluate the performance of the algorithm on multiple real world datasets. The experiment results show that our algorithm performs, on average, 40 % upto 800 % better than the previous state-of-the-art MapReduce algorithms.
Starting Page	473
Ending Page	497
Page Count	25
File Format	PDF
ISSN	09259902
Journal	Journal of Intelligent Information Systems
Volume Number	46
Issue Number	3
e-ISSN	15737675
Language	English
Publisher	Springer US
Publisher Date	2015-05-22
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Similarity join MapReduce Cosine similarity Filtering Information Storage and Retrieval Data Structures, Cryptology and Information Theory Artificial Intelligence (incl. Robotics) IT in Business Document Preparation and Text Processing
Content Type	Text
Resource Type	Article
Subject	Artificial Intelligence Computer Networks and Communications Information Systems Software Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Sentence similarity based on semantic kernels for intelligent text retrieval

TasteMiner: Mining partial tastes for neighbor-based collaborative filtering

Provisional reporting for rank joins

A novel method for dictionary translation

Permutation anonymization

Time-slide window join over data streams

A scalable approach to spectral clustering with SDD solvers

Measuring similarity of individuals in description logics over the refinement space of conjunctive queries

An end to end framework for building data cubes over trajectory data streams

Fast and scalable vector similarity joins with MapReduce

Similar Documents

Sentence similarity based on semantic kernels for intelligent text retrieval

TasteMiner: Mining partial tastes for neighbor-based collaborative filtering

Provisional reporting for rank joins

A novel method for dictionary translation

Permutation anonymization

Time-slide window join over data streams

A scalable approach to spectral clustering with SDD solvers

Measuring similarity of individuals in description logics over the refinement space of conjunctive queries

An end to end framework for building data cubes over trajectory data streams

Fast and scalable vector similarity joins with MapReduce