A self-attention network based node embedding model

Dai Quoc Nguyen; Tu Dinh Nguyen; Dinh Phung

doi:10.1007/978-3-030-67664-3_22

A self-attention network based node embedding model

Dai Quoc Nguyen, Tu Dinh Nguyen, Dinh Phung

Department of Data Science & AI

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

6 Citations (Scopus)

Abstract

Despite several signs of progress have been made recently, limited research has been conducted for an inductive setting where embeddings are required for newly unseen nodes – a setting encountered commonly in practical applications of deep learning for graph networks. This significantly affects the performances of downstream tasks such as node classification, link prediction or community extraction. To this end, we propose SANNE – a novel unsupervised embedding model – whose central idea is to employ a transformer self-attention network to iteratively aggregate vector representations of nodes in random walks. Our SANNE aims to produce plausible embeddings not only for present nodes, but also for newly unseen nodes. Experimental results show that the proposed SANNE obtains state-of-the-art results for the node classification task on well-known benchmark datasets.

Original language	English
Title of host publication	European Conference, ECML PKDD 2020 Ghent, Belgium, September 14–18, 2020 Proceedings, Part III
Editors	Frank Hutter, Kristian Kersting, Jefrey Lijffijt, Isabel Valera
Place of Publication	Cham Switzerland
Publisher	Springer
Pages	364-377
Number of pages	14
ISBN (Electronic)	9783030676643
ISBN (Print)	9783030676636
DOIs	https://doi.org/10.1007/978-3-030-67664-3_22
Publication status	Published - 2021
Event	European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2020 - Ghent, Belgium Duration: 14 Sept 2020 → 18 Sept 2020 https://ecmlpkdd2020.net/ https://link.springer.com/book/10.1007/978-3-030-67661-2 (Proceedings)

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	12459
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2020
Abbreviated title	ECML PKDD 2020
Country/Territory	Belgium
City	Ghent
Period	14/09/20 → 18/09/20
Internet address	https://ecmlpkdd2020.net/ https://link.springer.com/book/10.1007/978-3-030-67661-2 (Proceedings)

Keywords

Node classification
Node embeddings
Self-attention network
Transformer

Access to Document

10.1007/978-3-030-67664-3_22

Cite this

Nguyen, D. Q., Nguyen, T. D., & Phung, D. (2021). A self-attention network based node embedding model. In F. Hutter, K. Kersting, J. Lijffijt, & I. Valera (Eds.), European Conference, ECML PKDD 2020 Ghent, Belgium, September 14–18, 2020 Proceedings, Part III (pp. 364-377). (Lecture Notes in Computer Science; Vol. 12459). Springer. https://doi.org/10.1007/978-3-030-67664-3_22

@inproceedings{2314a80d01fc45cdb70efc4e2228211c,

title = "A self-attention network based node embedding model",

abstract = "Despite several signs of progress have been made recently, limited research has been conducted for an inductive setting where embeddings are required for newly unseen nodes – a setting encountered commonly in practical applications of deep learning for graph networks. This significantly affects the performances of downstream tasks such as node classification, link prediction or community extraction. To this end, we propose SANNE – a novel unsupervised embedding model – whose central idea is to employ a transformer self-attention network to iteratively aggregate vector representations of nodes in random walks. Our SANNE aims to produce plausible embeddings not only for present nodes, but also for newly unseen nodes. Experimental results show that the proposed SANNE obtains state-of-the-art results for the node classification task on well-known benchmark datasets.",

keywords = "Node classification, Node embeddings, Self-attention network, Transformer",

author = "Nguyen, {Dai Quoc} and Nguyen, {Tu Dinh} and Dinh Phung",

note = "Funding Information: This research was partially supported by the ARC Discovery Projects DP150100031 and DP160103934. Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2020, ECML PKDD 2020 ; Conference date: 14-09-2020 Through 18-09-2020",

year = "2021",

doi = "10.1007/978-3-030-67664-3_22",

language = "English",

isbn = "9783030676636",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "364--377",

editor = "Frank Hutter and Kristian Kersting and Jefrey Lijffijt and Isabel Valera",

booktitle = "European Conference, ECML PKDD 2020 Ghent, Belgium, September 14–18, 2020 Proceedings, Part III",

url = "https://ecmlpkdd2020.net/, https://link.springer.com/book/10.1007/978-3-030-67661-2",

}

Nguyen, DQ, Nguyen, TD & Phung, D 2021, A self-attention network based node embedding model. in F Hutter, K Kersting, J Lijffijt & I Valera (eds), European Conference, ECML PKDD 2020 Ghent, Belgium, September 14–18, 2020 Proceedings, Part III. Lecture Notes in Computer Science, vol. 12459, Springer, Cham Switzerland, pp. 364-377, European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2020, Ghent, Belgium, 14/09/20. https://doi.org/10.1007/978-3-030-67664-3_22

A self-attention network based node embedding model. / Nguyen, Dai Quoc; Nguyen, Tu Dinh; Phung, Dinh.
European Conference, ECML PKDD 2020 Ghent, Belgium, September 14–18, 2020 Proceedings, Part III. ed. / Frank Hutter; Kristian Kersting; Jefrey Lijffijt; Isabel Valera. Cham Switzerland: Springer, 2021. p. 364-377 (Lecture Notes in Computer Science; Vol. 12459).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - A self-attention network based node embedding model

AU - Nguyen, Dai Quoc

AU - Nguyen, Tu Dinh

AU - Phung, Dinh

PY - 2021

Y1 - 2021

N2 - Despite several signs of progress have been made recently, limited research has been conducted for an inductive setting where embeddings are required for newly unseen nodes – a setting encountered commonly in practical applications of deep learning for graph networks. This significantly affects the performances of downstream tasks such as node classification, link prediction or community extraction. To this end, we propose SANNE – a novel unsupervised embedding model – whose central idea is to employ a transformer self-attention network to iteratively aggregate vector representations of nodes in random walks. Our SANNE aims to produce plausible embeddings not only for present nodes, but also for newly unseen nodes. Experimental results show that the proposed SANNE obtains state-of-the-art results for the node classification task on well-known benchmark datasets.

AB - Despite several signs of progress have been made recently, limited research has been conducted for an inductive setting where embeddings are required for newly unseen nodes – a setting encountered commonly in practical applications of deep learning for graph networks. This significantly affects the performances of downstream tasks such as node classification, link prediction or community extraction. To this end, we propose SANNE – a novel unsupervised embedding model – whose central idea is to employ a transformer self-attention network to iteratively aggregate vector representations of nodes in random walks. Our SANNE aims to produce plausible embeddings not only for present nodes, but also for newly unseen nodes. Experimental results show that the proposed SANNE obtains state-of-the-art results for the node classification task on well-known benchmark datasets.

KW - Node classification

KW - Node embeddings

KW - Self-attention network

KW - Transformer

UR - http://www.scopus.com/inward/record.url?scp=85103237608&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-67664-3_22

DO - 10.1007/978-3-030-67664-3_22

M3 - Conference Paper

AN - SCOPUS:85103237608

SN - 9783030676636

T3 - Lecture Notes in Computer Science

SP - 364

EP - 377

BT - European Conference, ECML PKDD 2020 Ghent, Belgium, September 14–18, 2020 Proceedings, Part III

A2 - Hutter, Frank

A2 - Kersting, Kristian

A2 - Lijffijt, Jefrey

A2 - Valera, Isabel

PB - Springer

CY - Cham Switzerland

T2 - European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2020

Y2 - 14 September 2020 through 18 September 2020

ER -

A self-attention network based node embedding model

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Nonparametric Machine Learning for Modern Data Analytics (ARC DP)

Stay Well: Analysing Lifestyle Data from Smart Monitoring Devices (ARC DP)

Cite this