Linear least squares (LLS) is the least squares approximation of linear functions to data. It is a set of formulations for solving statistical problems involved in linear regression, including variants for ordinary (unweighted), weighted, and generalized (correlated) residuals. Numerical methods for linear least squares include inverting the matrix of the normal equations and orthogonal decomposition methods.

Consider the linear equation

where ${\displaystyle A\in \mathbb {R} ^{m\times n}}$ and ${\displaystyle b\in \mathbb {R} ^{m}}$ are given and ${\displaystyle x\in \mathbb {R} ^{n}}$ is variable to be computed. When ${\displaystyle m>n,}$ it is generally the case that (1) has no solution. For example, there is no value of ${\displaystyle x}$ that satisfies $${\displaystyle {\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}}x={\begin{bmatrix}1\\1\\0\end{bmatrix}},}$$ because the first two rows require that ${\displaystyle x=(1,1),}$ but then the third row is not satisfied. Thus, for ${\displaystyle m>n,}$ the goal of solving (1) exactly is typically replaced by finding the value of ${\displaystyle x}$ that minimizes some error. There are many ways that the error can be defined, but one of the most common is to define it as ${\displaystyle \|Ax-b\|^{2}.}$ This produces a minimization problem, called a least squares problem

The solution to the least squares problem (1) is computed by solving the normal equation

where ${\displaystyle A^{\top }}$ denotes the transpose of ${\displaystyle A}$.

Continuing the example, above, with $${\displaystyle A={\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}}\quad {\text{and}}\quad b={\begin{bmatrix}1\\1\\0\end{bmatrix}},}$$ we find $${\displaystyle A^{\top }A={\begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}}{\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}}={\begin{bmatrix}2&1\\1&2\end{bmatrix}}}$$ and $${\displaystyle A^{\top }b={\begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}}{\begin{bmatrix}1\\1\\0\end{bmatrix}}={\begin{bmatrix}1\\1\end{bmatrix}}.}$$ Solving the normal equation gives ${\displaystyle x=(1/3,1/3).}$

The three main linear least squares formulations are:

Other formulations include: