Identifying equivalent URLs using URL signatures

Lay Ki Soon; Sang Ho Lee

doi:10.1109/SITIS.2008.21

Identifying equivalent URLs using URL signatures

Lay Ki Soon, Sang Ho Lee

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

6 Citations (Scopus)

Abstract

In the standard URL normalization mechanism, URLs are normalized syntactically by a set of predefined steps. In this paper, we propose to enhance the standard URL normalization by incorporating the semantically meaningful metadata of the Web pages. The metadata taken into account are the body texts of the Web pages, which can be extracted during HTML parsing. Given a URL which has undergone the standard normalization mechanism, we construct its URL signature by hashing or fingerprinting the body text of the associated Web page using Message-Digest algorithm 5. URLs which share identical signatures are considered to be equivalent in our scheme. The experimental results show that our proposed method helps to further reduce redundant Web information retrieval by 34.57% in comparison with the standard URL normalization mechanism.

Original language	English
Title of host publication	SITIS 2008 - Proceedings of the 4th International Conference on Signal Image Technology and Internet Based Systems
Pages	203-210
Number of pages	8
DOIs	https://doi.org/10.1109/SITIS.2008.21
Publication status	Published - 2008
Externally published	Yes
Event	International Conference on Signal Image Technology & Internet Based Systems 2008 - Bali, Indonesia Duration: 30 Nov 2008 → 3 Dec 2008 Conference number: 4TH https://ieeexplore.ieee.org/xpl/conhome/4725760/proceeding (Proceedings)

Conference

Conference	International Conference on Signal Image Technology & Internet Based Systems 2008
Abbreviated title	SITIS 2008
Country/Territory	Indonesia
City	Bali
Period	30/11/08 → 3/12/08
Internet address	https://ieeexplore.ieee.org/xpl/conhome/4725760/proceeding (Proceedings)

Access to Document

10.1109/SITIS.2008.21

Cite this

@inproceedings{a4641aeeb31d4a28aeea863e94b463c7,

title = "Identifying equivalent URLs using URL signatures",

abstract = "In the standard URL normalization mechanism, URLs are normalized syntactically by a set of predefined steps. In this paper, we propose to enhance the standard URL normalization by incorporating the semantically meaningful metadata of the Web pages. The metadata taken into account are the body texts of the Web pages, which can be extracted during HTML parsing. Given a URL which has undergone the standard normalization mechanism, we construct its URL signature by hashing or fingerprinting the body text of the associated Web page using Message-Digest algorithm 5. URLs which share identical signatures are considered to be equivalent in our scheme. The experimental results show that our proposed method helps to further reduce redundant Web information retrieval by 34.57% in comparison with the standard URL normalization mechanism.",

author = "Soon, {Lay Ki} and Lee, {Sang Ho}",

note = "Copyright: Copyright 2009 Elsevier B.V., All rights reserved.; International Conference on Signal Image Technology & Internet Based Systems 2008, SITIS 2008 ; Conference date: 30-11-2008 Through 03-12-2008",

year = "2008",

doi = "10.1109/SITIS.2008.21",

language = "English",

isbn = "9780769534930",

pages = "203--210",

booktitle = "SITIS 2008 - Proceedings of the 4th International Conference on Signal Image Technology and Internet Based Systems",

url = "https://ieeexplore.ieee.org/xpl/conhome/4725760/proceeding",

}

TY - GEN

T1 - Identifying equivalent URLs using URL signatures

AU - Soon, Lay Ki

AU - Lee, Sang Ho

N1 - Conference code: 4TH

PY - 2008

Y1 - 2008

N2 - In the standard URL normalization mechanism, URLs are normalized syntactically by a set of predefined steps. In this paper, we propose to enhance the standard URL normalization by incorporating the semantically meaningful metadata of the Web pages. The metadata taken into account are the body texts of the Web pages, which can be extracted during HTML parsing. Given a URL which has undergone the standard normalization mechanism, we construct its URL signature by hashing or fingerprinting the body text of the associated Web page using Message-Digest algorithm 5. URLs which share identical signatures are considered to be equivalent in our scheme. The experimental results show that our proposed method helps to further reduce redundant Web information retrieval by 34.57% in comparison with the standard URL normalization mechanism.

AB - In the standard URL normalization mechanism, URLs are normalized syntactically by a set of predefined steps. In this paper, we propose to enhance the standard URL normalization by incorporating the semantically meaningful metadata of the Web pages. The metadata taken into account are the body texts of the Web pages, which can be extracted during HTML parsing. Given a URL which has undergone the standard normalization mechanism, we construct its URL signature by hashing or fingerprinting the body text of the associated Web page using Message-Digest algorithm 5. URLs which share identical signatures are considered to be equivalent in our scheme. The experimental results show that our proposed method helps to further reduce redundant Web information retrieval by 34.57% in comparison with the standard URL normalization mechanism.

UR - http://www.scopus.com/inward/record.url?scp=60349095201&partnerID=8YFLogxK

U2 - 10.1109/SITIS.2008.21

DO - 10.1109/SITIS.2008.21

M3 - Conference Paper

AN - SCOPUS:60349095201

SN - 9780769534930

SP - 203

EP - 210

BT - SITIS 2008 - Proceedings of the 4th International Conference on Signal Image Technology and Internet Based Systems

T2 - International Conference on Signal Image Technology & Internet Based Systems 2008

Y2 - 30 November 2008 through 3 December 2008

ER -

Identifying equivalent URLs using URL signatures

Abstract

Conference

Access to Document

Other files and links

Cite this