Bayesian linear regression

Bayesian linear regression is a type of conditional modeling in which the mean of one variable is described by a linear combination of other variables, with the goal of obtaining the posterior probability of the regression coefficients (as well as other parameters describing the distribution of the regressand) and ultimately allowing the out-of-sample prediction of the regressand (often labelled ${\displaystyle y}$) conditional on observed values of the regressors (usually ${\displaystyle X}$). The simplest and most widely used version of this model is the normal linear model, in which ${\displaystyle y}$ given ${\displaystyle X}$ is distributed Gaussian. In this model, and under a particular choice of prior probabilities for the parameters—so-called conjugate priors—the posterior can be found analytically. With more arbitrarily chosen priors, the posteriors generally have to be approximated.

Consider a standard linear regression problem, in which for ${\displaystyle i=1,\ldots ,n}$ we specify the mean of the conditional distribution of ${\displaystyle y_{i}}$ given a ${\displaystyle k\times 1}$ predictor vector ${\displaystyle \mathbf {x} _{i}}$: $${\displaystyle y_{i}=\mathbf {x} _{i}^{\mathsf {T}}{\boldsymbol {\beta }}+\varepsilon _{i},}$$

where ${\displaystyle {\boldsymbol {\beta }}}$ is a ${\displaystyle k\times 1}$ vector, and the ${\displaystyle \varepsilon _{i}}$ are independent and identically normally distributed random variables: $${\displaystyle \varepsilon _{i}\sim N(0,\sigma ^{2}).}$$

This corresponds to the following likelihood function:

$${\displaystyle \rho (\mathbf {y} \mid \mathbf {X} ,{\boldsymbol {\beta }},\sigma ^{2})\propto (\sigma ^{2})^{-n/2}\exp \left(-{\frac {1}{2\sigma ^{2}}}(\mathbf {y} -\mathbf {X} {\boldsymbol {\beta }})^{\mathsf {T}}(\mathbf {y} -\mathbf {X} {\boldsymbol {\beta }})\right).}$$

The ordinary least squares solution is used to estimate the coefficient vector using the Moore–Penrose pseudoinverse: $${\displaystyle {\hat {\boldsymbol {\beta }}}=(\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} }$$

where ${\displaystyle \mathbf {X} }$ is the ${\displaystyle n\times k}$ design matrix, each row of which is a predictor vector ${\displaystyle \mathbf {x} _{i}^{\mathsf {T}}}$; and ${\displaystyle \mathbf {y} }$ is the column ${\displaystyle n}$-vector ${\displaystyle [y_{1}\;\cdots \;y_{n}]^{\mathsf {T}}}$.

This is a frequentist approach, and it assumes that there are enough measurements to say something meaningful about ${\displaystyle {\boldsymbol {\beta }}}$. In the Bayesian approach, the data is supplemented with additional information in the form of a prior probability distribution. The prior belief about the parameters is combined with the data's likelihood function according to Bayes' theorem to yield the posterior belief about the parameters ${\displaystyle {\boldsymbol {\beta }}}$ and ${\displaystyle \sigma }$. The prior can take different functional forms depending on the domain and the information that is available a priori.