Approximate Bayesian computation (ABC) can be used for model fitting when the
likelihood function is intractable but simulating from the model is feasible.
However, even a single evaluation of a complex model may take several hours,
limiting the number of model evaluations available. Modelling the discrepancy
between the simulated and observed data using a Gaussian process (GP) can be
used to reduce the number of model evaluations required by ABC, but the
sensitivity of this approach to a specific GP formulation has not yet been
thoroughly investigated. We begin with a comprehensive empirical evaluation of
using GPs in ABC, including various transformations of the discrepancies and
two novel GP formulations. Our results indicate the choice of GP may
significantly affect the accuracy of the estimated posterior distribution.
Selection of an appropriate GP model is thus important. We formulate expected
utility to measure the accuracy of classifying discrepancies below or above the
ABC threshold, and show that it can be used to automate the GP model selection
step. Finally, based on the understanding gained with toy examples, we fit a
population genetic model for bacteria, providing insight into horizontal gene
transfer events within the population and from external origins.
Deep generative models provide powerful tools for distributions over
complicated manifolds, such as those of natural images. But many of these
methods, including generative adversarial networks (GANs), can be difficult to
train, in part because they are prone to mode collapse, which means that they
characterize only a few modes of the true distribution. To address this, we
introduce VEEGAN, which features a reconstructor network, reversing the action
of the generator by mapping from data to noise. Our training objective retains
the original asymptotic consistency guarantee of GANs, and can be interpreted
as a novel autoencoder loss over the noise. In sharp contrast to a traditional
autoencoder over data points, VEEGAN does not require specifying a loss
function over the data, but rather only over the representations, which are
standard normal by assumption. On an extensive set of synthetic and real world
image datasets, VEEGAN indeed resists mode collapsing to a far greater extent
than other recent GAN variants, and produces more realistic samples.
We introduce a new family of estimators for unnormalized statistical models.
Our family of estimators is parameterized by two nonlinear functions and uses a
single sample from an auxiliary distribution, generalizing Maximum Likelihood
Monte Carlo estimation of Geyer and Thompson (1992). The family is such that we
can estimate the partition function like any other parameter in the model. The
estimation is done by optimizing an algebraically simple, well defined
objective function, which allows for the use of dedicated optimization methods.
We establish consistency of the estimator family and give an expression for the
asymptotic covariance matrix, which enables us to further analyze the influence
of the nonlinearities and the auxiliary density on estimation performance. Some
estimators in our family are particularly stable for a wide range of auxiliary
densities. Interestingly, a specific choice of the nonlinearity establishes a
connection between density estimation and classification by nonlinear logistic
regression. Finally, the optimal amount of auxiliary samples relative to the
given amount of the data is considered from the perspective of computational
We show that the Bregman divergence provides a rich framework to estimate
unnormalized statistical models for continuous or discrete random variables,
that is, models which do not integrate or sum to one, respectively. We prove
that recent estimation methods such as noise-contrastive estimation, ratio
matching, and score matching belong to the proposed framework, and explain
their interconnection based on supervised learning. Further, we discuss the
role of boosting in unsupervised learning.