Enhancing URL normalization using metadata of web pages

Lay Ki Soon; Sang Ho Lee

doi:10.1109/ICCEE.2008.112

Enhancing URL normalization using metadata of web pages

Lay Ki Soon, Sang Ho Lee

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

3 Citations (Scopus)

Abstract

In this paper, we present our proposed method of incorporating metadata of Web pages to identify equivalent URLs in addition to the standard URL normalization methodology. The metadata considered are the page size and the body text of Web pages. These metadata can be obtained during HTML parsing in the process of crawling without incurring unnecessary cost. Our experiment shows an accuracy of up to 95.38% in identifying equivalent URLs by using the body text of Web pages.

Original language	English
Title of host publication	Proceedings of the 2008 International Conference on Computer and Electrical Engineering, ICCEE 2008
Pages	331-335
Number of pages	5
DOIs	https://doi.org/10.1109/ICCEE.2008.112
Publication status	Published - 2008
Externally published	Yes
Event	International Conference on Computer and Electrical Engineering 2008 - Phuket, Thailand Duration: 20 Dec 2008 → 22 Dec 2008 https://ieeexplore.ieee.org/xpl/conhome/4740925/proceeding (Proceedings)

Conference

Conference	International Conference on Computer and Electrical Engineering 2008
Abbreviated title	ICCEE 2008
Country/Territory	Thailand
City	Phuket
Period	20/12/08 → 22/12/08
Internet address	https://ieeexplore.ieee.org/xpl/conhome/4740925/proceeding (Proceedings)

Access to Document

10.1109/ICCEE.2008.112

Cite this

@inproceedings{c64fec6844494d5cbc4f5999f0fecbfd,

title = "Enhancing URL normalization using metadata of web pages",

abstract = "In this paper, we present our proposed method of incorporating metadata of Web pages to identify equivalent URLs in addition to the standard URL normalization methodology. The metadata considered are the page size and the body text of Web pages. These metadata can be obtained during HTML parsing in the process of crawling without incurring unnecessary cost. Our experiment shows an accuracy of up to 95.38% in identifying equivalent URLs by using the body text of Web pages.",

author = "Soon, {Lay Ki} and Lee, {Sang Ho}",

note = "Copyright: Copyright 2009 Elsevier B.V., All rights reserved.; International Conference on Computer and Electrical Engineering 2008, ICCEE 2008 ; Conference date: 20-12-2008 Through 22-12-2008",

year = "2008",

doi = "10.1109/ICCEE.2008.112",

language = "English",

isbn = "9780769535043",

pages = "331--335",

booktitle = "Proceedings of the 2008 International Conference on Computer and Electrical Engineering, ICCEE 2008",

url = "https://ieeexplore.ieee.org/xpl/conhome/4740925/proceeding",

}

TY - GEN

T1 - Enhancing URL normalization using metadata of web pages

AU - Soon, Lay Ki

AU - Lee, Sang Ho

PY - 2008

Y1 - 2008

N2 - In this paper, we present our proposed method of incorporating metadata of Web pages to identify equivalent URLs in addition to the standard URL normalization methodology. The metadata considered are the page size and the body text of Web pages. These metadata can be obtained during HTML parsing in the process of crawling without incurring unnecessary cost. Our experiment shows an accuracy of up to 95.38% in identifying equivalent URLs by using the body text of Web pages.

AB - In this paper, we present our proposed method of incorporating metadata of Web pages to identify equivalent URLs in addition to the standard URL normalization methodology. The metadata considered are the page size and the body text of Web pages. These metadata can be obtained during HTML parsing in the process of crawling without incurring unnecessary cost. Our experiment shows an accuracy of up to 95.38% in identifying equivalent URLs by using the body text of Web pages.

UR - http://www.scopus.com/inward/record.url?scp=62949146333&partnerID=8YFLogxK

U2 - 10.1109/ICCEE.2008.112

DO - 10.1109/ICCEE.2008.112

M3 - Conference Paper

AN - SCOPUS:62949146333

SN - 9780769535043

SP - 331

EP - 335

BT - Proceedings of the 2008 International Conference on Computer and Electrical Engineering, ICCEE 2008

T2 - International Conference on Computer and Electrical Engineering 2008

Y2 - 20 December 2008 through 22 December 2008

ER -

Enhancing URL normalization using metadata of web pages

Abstract

Conference

Access to Document

Other files and links

Cite this