Debiasing Evidence Approximations: On Importance-Weighted Autoencoders and Jackknife Variational Inference

The importance-weighted autoencoder (IWAE) approach of Burda et al. (2015) defines a sequence of increasingly tighter bounds on the marginal likelihood of latent variable models. Recently, Cremer et al. (2017) reinterpreted the IWAE bounds as ordinary variational evidence lower bounds (ELBO) applied to increasingly accurate variational distributions. In this work, we provide yet another perspective on the IWAE bounds. We interpret each IWAE bound as a biased estimator of the true marginal likelihood where for the bound defined on K samples we show the bias to be of order O(1/K). In our theoretical analysis of the IWAE objective we derive asymptotic bias and variance expressions. Based on this analysis we develop jackknife variational inference (JVI), a family of bias-reduced estimators reducing the bias to O(K^{-(m+1)}) for any given m < K while retaining computational efficiency. Finally, we demonstrate that JVI leads to improved evidence estimates in variational autoencoders. We also report first results on applying JVI to learning variational autoencoders.

Speaker Details

Sebastian Nowozin is a researcher in the Machine Learning and Perception group at Microsoft Research Cambridge. He received his Master of Engineering degree from the Shanghai Jiaotong University and his diploma degree in computer
science with distinction from the Technical University of Berlin in 2006. He received his PhD degree summa cum laude in 2009 for his thesis on learning with structured data in computer vision, completed at the Max Planck Institute
for Biological Cybernetics, Tuebingen and the Technical University of Berlin. His research interest is diverse and includes computer vision, machine learning, and continuous and discrete optimization. He organizes the successful “Optimization for Machine Learning” workshop series at NIPS (OPT 2008-2011) and serves as PC-member/reviewer for machine learning (e.g. NIPS, ICML, AISTATS, UAI, ECML, JMLR) and computer vision (e.g. CVPR, ICCV, ECCV, PAMI, IJCV) conferences/journals.

Date:
Speakers:
Sebastian Nowozin
Affiliation:
Microsoft Research