Factor analysis

Factor analysis is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. For example, it is possible that variations in six observed variables mainly reflect the variations in two unobserved (underlying) variables. Factor analysis searches for such joint variations in response to unobserved latent variables. The observed variables are modelled as linear combinations of the potential factors plus "error" terms, hence factor analysis can be thought of as a special case of errors-in-variables models.

The correlation between a variable and a given factor, called the variable's factor loading, indicates the extent to which the two are related.

A common rationale behind factor analytic methods is that the information gained about the interdependencies between observed variables can be used later to reduce the set of variables in a dataset. Factor analysis is commonly used in psychometrics, personality psychology, biology, marketing, product management, operations research, finance, and machine learning. It may help to deal with data sets where there are large numbers of observed variables that are thought to reflect a smaller number of underlying/latent variables. It is one of the most commonly used inter-dependency techniques and is used when the relevant set of variables shows a systematic inter-dependence and the objective is to find out the latent factors that create a commonality.

The model attempts to explain a set of ${\displaystyle p}$ observations in each of ${\displaystyle n}$ individuals with a set of ${\displaystyle k}$ common factors (${\displaystyle f_{i,j}}$) where there are fewer factors per unit than observations per unit (${\displaystyle k<p}$). Each individual has ${\displaystyle k}$ of their own common factors, and these are related to the observations via the factor loading matrix (${\displaystyle L\in \mathbb {R} ^{p\times k}}$), for a single observation, according to

where

In matrix notation

where observation matrix ${\displaystyle X\in \mathbb {R} ^{p\times n}}$, loading matrix ${\displaystyle L\in \mathbb {R} ^{p\times k}}$, factor matrix ${\displaystyle F\in \mathbb {R} ^{k\times n}}$, error term matrix ${\displaystyle \varepsilon \in \mathbb {R} ^{p\times n}}$ and mean matrix ${\displaystyle \mathrm {M} \in \mathbb {R} ^{p\times n}}$ whereby the ${\displaystyle (i,m)}$th element is simply ${\displaystyle \mathrm {M} _{i,m}=\mu _{i}}$.

Also we will impose the following assumptions on ${\displaystyle F}$: