TY - JOUR
T1 - Fast and accurate variational inference for models with many latent variables
AU - Loaiza-Maya, Rubén
AU - Smith, Michael Stanley
AU - Nott, David J.
AU - Danaher, Peter J.
N1 - Funding Information:
The authors thank the Editor, Associate Editor and two referees for their encouragement and comments that helped improve the paper. They also thank Wharton Customer Analytics for providing the consumer panel data used in the second example. Rub?n Loaiza-Maya is an associate investigator with the Australian Centre of Excellence for Mathematical and Statistical Frontiers. David Nott is affiliated with the Operations Research and Analytics Research cluster at the National University of Singapore.
Publisher Copyright:
© 2021 Elsevier B.V.
Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.
PY - 2022/10
Y1 - 2022/10
N2 - Models with a large number of latent variables are often used to utilize the information in big or complex data, but can be difficult to estimate. Variational inference methods provide an attractive solution. These methods use an approximation to the posterior density, yet for large latent variable models existing choices can be inaccurate or slow to calibrate. Here, we propose a family of tractable variational approximations that are more accurate and faster to calibrate for this case. It combines a parsimonious approximation for the parameter posterior with the exact conditional posterior of the latent variables. We derive a simplified expression for the re-parameterization gradient of the variational lower bound, which is the main ingredient of optimization algorithms used for calibration. Implementation only requires exact or approximate generation from the conditional posterior of the latent variables, rather than computation of their density. In effect, our method provides a new way to employ Markov chain Monte Carlo (MCMC) within variational inference. We illustrate using two complex contemporary econometric examples. The first is a nonlinear multivariate state space model for U.S. macroeconomic variables. The second is a random coefficients tobit model applied to two million sales by 20,000 individuals in a consumer panel. In both cases, our approximating family is considerably more accurate than mean field or structured Gaussian approximations, and faster than MCMC. Last, we show how to implement data sub-sampling in variational inference for our approximation, further reducing computation time. MATLAB code implementing the method is provided.
AB - Models with a large number of latent variables are often used to utilize the information in big or complex data, but can be difficult to estimate. Variational inference methods provide an attractive solution. These methods use an approximation to the posterior density, yet for large latent variable models existing choices can be inaccurate or slow to calibrate. Here, we propose a family of tractable variational approximations that are more accurate and faster to calibrate for this case. It combines a parsimonious approximation for the parameter posterior with the exact conditional posterior of the latent variables. We derive a simplified expression for the re-parameterization gradient of the variational lower bound, which is the main ingredient of optimization algorithms used for calibration. Implementation only requires exact or approximate generation from the conditional posterior of the latent variables, rather than computation of their density. In effect, our method provides a new way to employ Markov chain Monte Carlo (MCMC) within variational inference. We illustrate using two complex contemporary econometric examples. The first is a nonlinear multivariate state space model for U.S. macroeconomic variables. The second is a random coefficients tobit model applied to two million sales by 20,000 individuals in a consumer panel. In both cases, our approximating family is considerably more accurate than mean field or structured Gaussian approximations, and faster than MCMC. Last, we show how to implement data sub-sampling in variational inference for our approximation, further reducing computation time. MATLAB code implementing the method is provided.
KW - Large consumer panels
KW - Latent variable models
KW - Stochastic gradient ascent
KW - Sub-sampling variational inference
KW - Time-varying VAR with stochastic volatility
UR - http://www.scopus.com/inward/record.url?scp=85107619994&partnerID=8YFLogxK
U2 - 10.1016/j.jeconom.2021.05.002
DO - 10.1016/j.jeconom.2021.05.002
M3 - Article
AN - SCOPUS:85107619994
VL - 230
SP - 339
EP - 362
JO - Journal of Econometrics
JF - Journal of Econometrics
SN - 0304-4076
IS - 2
ER -