A generic method for accelerating LSH-based similarity join processing

Chenyun Yu, Sarana Nutanong, Hangyu Li, Cong Wang, Xingliang Yuan

Research output: Contribution to journalArticleResearchpeer-review

8 Citations (Scopus)

Abstract

Locality sensitive hashing (LSH) is an efficient method for solving the problem of approximate similarity search in high-dimensional spaces. Through LSH, a high-dimensional similarity join can be processed in the same way as hash join, making the cost of joining two large datasets linear. By judicially analyzing the properties of multiple LSH algorithms, we propose a generic method to speed up the process of joining two large datasets using LSH. The crux of our method lies in the way which we identify a set of representative points to reduce the number of LSH lookups. Theoretical analyzes show that our proposed method can greatly reduce the number of lookup operations and retain the same result accuracy compared to executing LSH lookups for every query point. Furthermore, we demonstrate the generality of our method by showing that the same principle can be applied to LSH algorithms for three different metrics: the Euclidean distance (QALSH), Jaccard similarity measure (MinHash), and Hamming distance (sequence hashing). Results from experimental studies using real datasets confirm our error analyzes and show significant improvements of our method over the state-of-the-art LSH method: to achieve over 0.95 recall, we only need to operate LSH lookups for at most 15 percent of the query points.

Original languageEnglish
Article number7782356
Pages (from-to)712-726
Number of pages15
JournalIEEE Transactions on Knowledge and Data Engineering
Volume29
Issue number4
DOIs
Publication statusPublished - 1 Apr 2017
Externally publishedYes

Keywords

  • locality sensitive hashing
  • query processing
  • representative selection
  • Similarity join

Cite this