Gaussian processes

The following happens a lot in science. We vary variables \(\mathbf{x}\) (like time, pH, etc.) and make observations \(\mathbf{y}\). We perform a regression using some theoretical function \(f(\mathbf{x})\), which describes how we expect \(y\) to vary with \(\mathbf{x}\). We then can have a pretty good idea what we would measure for some other value of \(\mathbf{x}\). By making a few measurements, the regression helps us say things more generally, even for \(\mathbf{x}\) values we didn’t explicitly use in an experiment.

We have seen this in this class. We perform a regression, getting samples from the posterior distribution. We then sample out of the posterior predictive distribution to get what we might expect for performing an experiment, perhaps with a difference \(x\) values. In practice, this is performing a posterior predictive check with some values of \(x\) that we hadn’t used in a measurement.

This is an example of parametric inference, in which we have a specific mathematical model in mind, complete with parameters. But what if we did not have a specific function in mind? Rather, we just would like to be able to predict what value of \(y\) we might get for some untested \(x\) value and we do not really care what the underlying model is. In other words, we just would like to consider a family of functions that do not vary too rapidly or with too large of amplitude. There are an infinity of such functions. This is an example of nonparametric inference. In a Bayesian context, nonparametric inference involves consideration of an infinite number of models. We will explore nonparametric inference in the context of Gaussian processes.