Sparse horseshoe estimation via expectation-maximisation

Shu Yu Tew; Daniel F. Schmidt; Enes Makalic

doi:10.1007/978-3-031-26419-1_8

Sparse horseshoe estimation via expectation-maximisation

Department of Data Science & AI

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

3 Citations (Scopus)

Abstract

The horseshoe prior is known to possess many desirable properties for Bayesian estimation of sparse parameter vectors, yet its density function lacks an analytic form. As such, it is challenging to find a closed-form solution for the posterior mode. Conventional horseshoe estimators use the posterior mean to estimate the parameters, but these estimates are not sparse. We propose a novel expectation-maximisation (EM) procedure for computing the MAP estimates of the parameters in the case of the standard linear model. A particular strength of our approach is that the M-step depends only on the form of the prior and it is independent of the form of the likelihood. We introduce several simple modifications of this EM procedure that allow for straightforward extension to generalised linear models. In experiments performed on simulated and real data, our approach performs comparable, or superior to, state-of-the-art sparse estimation methods in terms of statistical performance and computational cost.

Original language	English
Title of host publication	European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V
Editors	Massih-Reza Amini, Stéphane Canu, Asja Fischer, Tias Guns, Petra Kralj Novak, Grigorios Tsoumakas
Place of Publication	Cham Switzerland
Publisher	Springer
Pages	123-139
Number of pages	17
ISBN (Electronic)	9783031264191
ISBN (Print)	9783031264184
DOIs	https://doi.org/10.1007/978-3-031-26419-1_8
Publication status	Published - 2023
Event	European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2022 - Grenoble, France Duration: 19 Sept 2022 → 23 Sept 2022 Conference number: 22nd https://2022.ecmlpkdd.org/ (Website) https://link.springer.com/book/10.1007/978-3-031-26419-1 (Proceedings)

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	13717
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2022
Abbreviated title	ECML PKDD 2022
Country/Territory	France
City	Grenoble
Period	19/09/22 → 23/09/22
Internet address	https://2022.ecmlpkdd.org/ (Website) https://link.springer.com/book/10.1007/978-3-031-26419-1 (Proceedings)

Keywords

Expectation-maximisation
Horseshoe regression
Maximum a posteriori estimation
Non-convex penalised regression
Sparse regression

Access to Document

10.1007/978-3-031-26419-1_8

Cite this

Tew, S. Y., Schmidt, D. F., & Makalic, E. (2023). Sparse horseshoe estimation via expectation-maximisation. In M.-R. Amini, S. Canu, A. Fischer, T. Guns, P. Kralj Novak, & G. Tsoumakas (Eds.), European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V (pp. 123-139). (Lecture Notes in Computer Science; Vol. 13717). Springer. https://doi.org/10.1007/978-3-031-26419-1_8

Tew, Shu Yu ; Schmidt, Daniel F. ; Makalic, Enes. / Sparse horseshoe estimation via expectation-maximisation. European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V. editor / Massih-Reza Amini ; Stéphane Canu ; Asja Fischer ; Tias Guns ; Petra Kralj Novak ; Grigorios Tsoumakas. Cham Switzerland : Springer, 2023. pp. 123-139 (Lecture Notes in Computer Science).

@inproceedings{ccb7881178024a559fa98934453e4d02,

title = "Sparse horseshoe estimation via expectation-maximisation",

abstract = "The horseshoe prior is known to possess many desirable properties for Bayesian estimation of sparse parameter vectors, yet its density function lacks an analytic form. As such, it is challenging to find a closed-form solution for the posterior mode. Conventional horseshoe estimators use the posterior mean to estimate the parameters, but these estimates are not sparse. We propose a novel expectation-maximisation (EM) procedure for computing the MAP estimates of the parameters in the case of the standard linear model. A particular strength of our approach is that the M-step depends only on the form of the prior and it is independent of the form of the likelihood. We introduce several simple modifications of this EM procedure that allow for straightforward extension to generalised linear models. In experiments performed on simulated and real data, our approach performs comparable, or superior to, state-of-the-art sparse estimation methods in terms of statistical performance and computational cost.",

keywords = "Expectation-maximisation, Horseshoe regression, Maximum a posteriori estimation, Non-convex penalised regression, Sparse regression",

author = "Tew, \{Shu Yu\} and Schmidt, \{Daniel F.\} and Enes Makalic",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2022, ECML PKDD 2022 ; Conference date: 19-09-2022 Through 23-09-2022",

year = "2023",

doi = "10.1007/978-3-031-26419-1\_8",

language = "English",

isbn = "9783031264184",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "123--139",

editor = "Massih-Reza Amini and St{\'e}phane Canu and Asja Fischer and Tias Guns and \{Kralj Novak\}, Petra and Grigorios Tsoumakas",

booktitle = "European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V",

address = "Switzerland",

url = "https://2022.ecmlpkdd.org/, https://link.springer.com/book/10.1007/978-3-031-26419-1",

}

Tew, SY, Schmidt, DF & Makalic, E 2023, Sparse horseshoe estimation via expectation-maximisation. in M-R Amini, S Canu, A Fischer, T Guns, P Kralj Novak & G Tsoumakas (eds), European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V. Lecture Notes in Computer Science, vol. 13717, Springer, Cham Switzerland, pp. 123-139, European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2022, Grenoble, France, 19/09/22. https://doi.org/10.1007/978-3-031-26419-1_8

Sparse horseshoe estimation via expectation-maximisation. / Tew, Shu Yu; Schmidt, Daniel F.; Makalic, Enes.
European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V. ed. / Massih-Reza Amini; Stéphane Canu; Asja Fischer; Tias Guns; Petra Kralj Novak; Grigorios Tsoumakas. Cham Switzerland: Springer, 2023. p. 123-139 (Lecture Notes in Computer Science; Vol. 13717).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Sparse horseshoe estimation via expectation-maximisation

AU - Tew, Shu Yu

AU - Schmidt, Daniel F.

AU - Makalic, Enes

N1 - Conference code: 22nd

PY - 2023

Y1 - 2023

N2 - The horseshoe prior is known to possess many desirable properties for Bayesian estimation of sparse parameter vectors, yet its density function lacks an analytic form. As such, it is challenging to find a closed-form solution for the posterior mode. Conventional horseshoe estimators use the posterior mean to estimate the parameters, but these estimates are not sparse. We propose a novel expectation-maximisation (EM) procedure for computing the MAP estimates of the parameters in the case of the standard linear model. A particular strength of our approach is that the M-step depends only on the form of the prior and it is independent of the form of the likelihood. We introduce several simple modifications of this EM procedure that allow for straightforward extension to generalised linear models. In experiments performed on simulated and real data, our approach performs comparable, or superior to, state-of-the-art sparse estimation methods in terms of statistical performance and computational cost.

AB - The horseshoe prior is known to possess many desirable properties for Bayesian estimation of sparse parameter vectors, yet its density function lacks an analytic form. As such, it is challenging to find a closed-form solution for the posterior mode. Conventional horseshoe estimators use the posterior mean to estimate the parameters, but these estimates are not sparse. We propose a novel expectation-maximisation (EM) procedure for computing the MAP estimates of the parameters in the case of the standard linear model. A particular strength of our approach is that the M-step depends only on the form of the prior and it is independent of the form of the likelihood. We introduce several simple modifications of this EM procedure that allow for straightforward extension to generalised linear models. In experiments performed on simulated and real data, our approach performs comparable, or superior to, state-of-the-art sparse estimation methods in terms of statistical performance and computational cost.

KW - Expectation-maximisation

KW - Horseshoe regression

KW - Maximum a posteriori estimation

KW - Non-convex penalised regression

KW - Sparse regression

UR - https://www.scopus.com/pages/publications/85151061196

U2 - 10.1007/978-3-031-26419-1_8

DO - 10.1007/978-3-031-26419-1_8

M3 - Conference Paper

AN - SCOPUS:85151061196

SN - 9783031264184

T3 - Lecture Notes in Computer Science

SP - 123

EP - 139

BT - European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V

A2 - Amini, Massih-Reza

A2 - Canu, Stéphane

A2 - Fischer, Asja

A2 - Guns, Tias

A2 - Kralj Novak, Petra

A2 - Tsoumakas, Grigorios

PB - Springer

CY - Cham Switzerland

T2 - European Conference on Machine Learning European Conference on Principles and Practice of Knowledge Discovery in Databases 2022

Y2 - 19 September 2022 through 23 September 2022

ER -

Tew SY, Schmidt DF , Makalic E. Sparse horseshoe estimation via expectation-maximisation. In Amini MR, Canu S, Fischer A, Guns T, Kralj Novak P, Tsoumakas G, editors, European Conference, ECML PKDD 2022 Grenoble, France, September 19–23, 2022 Proceedings, Part V. Cham Switzerland: Springer. 2023. p. 123-139. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-26419-1_8