Lazy exact deduplication

Jingwei Ma; Rebecca J. Stones; Yuxiang Ma; Jingui Wang; Junjie Ren; Gang Wang; Xiaoguang Liu

doi:10.1109/MSST.2016.7897081

Lazy exact deduplication

Jingwei Ma
, Rebecca J. Stones
, Yuxiang Ma
, Jingui Wang
, Junjie Ren
, Gang Wang
, Xiaoguang Liu

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Other

9 Citations (Scopus)

Abstract

During data deduplication, on-disk fingerprint lookups lead to high disk traffic, resulting in a bottleneck. In this paper, we propose a "lazy" data deduplication method which buffers incoming fingerprints and performs on-disk lookups in batches, aiming to reduce the disk bottleneck. In deduplication in general, prefetching is used to improve the cache hit rate by exploiting locality within the incoming fingerprint stream. For lazy deduplication, we design a buffering strategy that preserves locality in order to similarly facilitate prefetching. Experimental results indicate that the lazy method improves fingerprint identification performance by over 50% compared with an "eager" method with the same data layout.

Original language	English
Title of host publication	2016 32nd Symposium on Mass Storage Systems and Technologies (MSST 2016)
Subtitle of host publication	Santa Clara, California, USA, 2-6 May 2016
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	93-102
Number of pages	10
ISBN (Electronic)	9781467390552
ISBN (Print)	9781467390569
DOIs	https://doi.org/10.1109/MSST.2016.7897081
Publication status	Published - 11 Apr 2017
Externally published	Yes
Event	Symposium on Mass Storage Systems and Technologies, MSST 2016 - Santa Clara, United States of America Duration: 2 May 2016 → 6 May 2016 Conference number: 32nd

Conference

Conference	Symposium on Mass Storage Systems and Technologies, MSST 2016
Abbreviated title	MSST 2016
Country/Territory	United States of America
City	Santa Clara
Period	2/05/16 → 6/05/16

Access to Document

10.1109/MSST.2016.7897081

Cite this

@inproceedings{c1f3eae77e3b40d49d0c2f8982ed47e5,

title = "Lazy exact deduplication",

abstract = "During data deduplication, on-disk fingerprint lookups lead to high disk traffic, resulting in a bottleneck. In this paper, we propose a {"}lazy{"} data deduplication method which buffers incoming fingerprints and performs on-disk lookups in batches, aiming to reduce the disk bottleneck. In deduplication in general, prefetching is used to improve the cache hit rate by exploiting locality within the incoming fingerprint stream. For lazy deduplication, we design a buffering strategy that preserves locality in order to similarly facilitate prefetching. Experimental results indicate that the lazy method improves fingerprint identification performance by over 50\% compared with an {"}eager{"} method with the same data layout.",

author = "Jingwei Ma and Stones, \{Rebecca J.\} and Yuxiang Ma and Jingui Wang and Junjie Ren and Gang Wang and Xiaoguang Liu",

year = "2017",

month = apr,

day = "11",

doi = "10.1109/MSST.2016.7897081",

language = "English",

isbn = "9781467390569",

pages = "93--102",

booktitle = "2016 32nd Symposium on Mass Storage Systems and Technologies (MSST 2016)",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

address = "United States of America",

note = "Symposium on Mass Storage Systems and Technologies, MSST 2016, MSST 2016 ; Conference date: 02-05-2016 Through 06-05-2016",

}

Ma, J, Stones, RJ, Ma, Y, Wang, J, Ren, J, Wang, G & Liu, X 2017, Lazy exact deduplication. in 2016 32nd Symposium on Mass Storage Systems and Technologies (MSST 2016): Santa Clara, California, USA, 2-6 May 2016. IEEE, Institute of Electrical and Electronics Engineers, pp. 93-102, Symposium on Mass Storage Systems and Technologies, MSST 2016, Santa Clara, United States of America, 2/05/16. https://doi.org/10.1109/MSST.2016.7897081

TY - GEN

T1 - Lazy exact deduplication

AU - Ma, Jingwei

AU - Stones, Rebecca J.

AU - Ma, Yuxiang

AU - Wang, Jingui

AU - Ren, Junjie

AU - Wang, Gang

AU - Liu, Xiaoguang

N1 - Conference code: 32nd

PY - 2017/4/11

Y1 - 2017/4/11

N2 - During data deduplication, on-disk fingerprint lookups lead to high disk traffic, resulting in a bottleneck. In this paper, we propose a "lazy" data deduplication method which buffers incoming fingerprints and performs on-disk lookups in batches, aiming to reduce the disk bottleneck. In deduplication in general, prefetching is used to improve the cache hit rate by exploiting locality within the incoming fingerprint stream. For lazy deduplication, we design a buffering strategy that preserves locality in order to similarly facilitate prefetching. Experimental results indicate that the lazy method improves fingerprint identification performance by over 50% compared with an "eager" method with the same data layout.

AB - During data deduplication, on-disk fingerprint lookups lead to high disk traffic, resulting in a bottleneck. In this paper, we propose a "lazy" data deduplication method which buffers incoming fingerprints and performs on-disk lookups in batches, aiming to reduce the disk bottleneck. In deduplication in general, prefetching is used to improve the cache hit rate by exploiting locality within the incoming fingerprint stream. For lazy deduplication, we design a buffering strategy that preserves locality in order to similarly facilitate prefetching. Experimental results indicate that the lazy method improves fingerprint identification performance by over 50% compared with an "eager" method with the same data layout.

UR - https://www.scopus.com/pages/publications/85015251737

U2 - 10.1109/MSST.2016.7897081

DO - 10.1109/MSST.2016.7897081

M3 - Conference Paper

SN - 9781467390569

SP - 93

EP - 102

BT - 2016 32nd Symposium on Mass Storage Systems and Technologies (MSST 2016)

PB - IEEE, Institute of Electrical and Electronics Engineers

T2 - Symposium on Mass Storage Systems and Technologies, MSST 2016

Y2 - 2 May 2016 through 6 May 2016

ER -

Lazy exact deduplication

Abstract

Conference

Access to Document

Other files and links

Cite this