Identifiability

Let ${\displaystyle {\mathcal {P}}=\{P_{\theta }:\theta \in \Theta \}}$ be a statistical model with parameter space ${\displaystyle \Theta }$. We say that ${\displaystyle {\mathcal {P}}}$ is identifiable if the mapping ${\displaystyle \theta \mapsto P_{\theta }}$ is one-to-one:^[2]

${\displaystyle P_{\theta _{1}}=P_{\theta _{2}}\quad \Rightarrow \quad \theta _{1}=\theta _{2}\quad \ {\text{for all }}\theta _{1},\theta _{2}\in \Theta .}$

This definition means that distinct values of θ should correspond to distinct probability distributions: if θ₁≠θ₂, then also P_θ1≠P_θ2.^[3] If the distributions are defined in terms of the probability density functions (pdfs), then two pdfs should be considered distinct only if they differ on a set of non-zero measure (for example two functions ƒ₁(x) = 1_{0 ≤ x < 1} and ƒ₂(x) = 1_{0 ≤ x ≤ 1} differ only at a single point x = 1 — a set of measure zero — and thus cannot be considered as distinct pdfs).

Identifiability of the model in the sense of invertibility of the map ${\displaystyle \theta \mapsto P_{\theta }}$ is equivalent to being able to learn the model's true parameter if the model can be observed indefinitely long. Indeed, if {X_t} ⊆ S is the sequence of observations from the model, then by the strong law of large numbers,

${\displaystyle {\frac {1}{T}}\sum _{t=1}^{T}\mathbf {1} _{\{X_{t}\in A\}}\ {\xrightarrow {\text{a.s.}}}\ \Pr[X_{t}\in A],}$

for every measurable set A ⊆ S (here 1_{...} is the indicator function). Thus, with an infinite number of observations we will be able to find the true probability distribution P₀ in the model, and since the identifiability condition above requires that the map ${\displaystyle \theta \mapsto P_{\theta }}$ be invertible, we will also be able to find the true value of the parameter which generated given distribution P₀.

Let ${\displaystyle {\mathcal {P}}}$ be the normal location-scale family:

${\displaystyle {\mathcal {P}}={\Big \{}\ f_{\theta }(x)={\tfrac {1}{{\sqrt {2\pi }}\sigma }}e^{-{\frac {1}{2\sigma ^{2}}}(x-\mu )^{2}}\ {\Big |}\ \theta =(\mu ,\sigma ):\mu \in \mathbb {R} ,\,\sigma \!>0\ {\Big \}}.}$

Then

${\displaystyle {\begin{aligned}&f_{\theta _{1}}(x)=f_{\theta _{2}}(x)\\[6pt]\Longleftrightarrow {}&{\frac {1}{{\sqrt {2\pi }}\sigma _{1}}}\exp \left(-{\frac {1}{2\sigma _{1}^{2}}}(x-\mu _{1})^{2}\right)={\frac {1}{{\sqrt {2\pi }}\sigma _{2}}}\exp \left(-{\frac {1}{2\sigma _{2}^{2}}}(x-\mu _{2})^{2}\right)\\[6pt]\Longleftrightarrow {}&{\frac {1}{\sigma _{1}^{2}}}(x-\mu _{1})^{2}+\ln \sigma _{1}={\frac {1}{\sigma _{2}^{2}}}(x-\mu _{2})^{2}+\ln \sigma _{2}\\[6pt]\Longleftrightarrow {}&x^{2}\left({\frac {1}{\sigma _{1}^{2}}}-{\frac {1}{\sigma _{2}^{2}}}\right)-2x\left({\frac {\mu _{1}}{\sigma _{1}^{2}}}-{\frac {\mu _{2}}{\sigma _{2}^{2}}}\right)+\left({\frac {\mu _{1}^{2}}{\sigma _{1}^{2}}}-{\frac {\mu _{2}^{2}}{\sigma _{2}^{2}}}+\ln \sigma _{1}-\ln \sigma _{2}\right)=0\end{aligned}}}$

This expression is equal to zero for almost all x only when all its coefficients are equal to zero, which is only possible when |σ₁| = |σ₂| and μ₁ = μ₂. Since in the scale parameter σ is restricted to be greater than zero, we conclude that the model is identifiable: ƒ_θ1 = ƒ_θ2 ⇔ θ₁ = θ₂.

Let ${\displaystyle {\mathcal {P}}}$ be the standard linear regression model:

${\displaystyle y=\beta 'x+\varepsilon ,\quad \mathrm {E} [\,\varepsilon \mid x\,]=0}$

(where ′ denotes matrix transpose). Then the parameter β is identifiable if and only if the matrix ${\displaystyle \mathrm {E} [xx']}$ is invertible. Thus, this is the identification condition in the model.

Suppose ${\displaystyle {\mathcal {P}}}$ is the classical errors-in-variables linear model:

${\displaystyle {\begin{cases}y=\beta x^{*}+\varepsilon ,\\x=x^{*}+\eta ,\end{cases}}}$

where (ε,η,x*) are jointly normal independent random variables with zero expected value and unknown variances, and only the variables (x,y) are observed. Then this model is not identifiable,^[4] only the product βσ²_∗ is (where σ²_∗ is the variance of the latent regressor x*). This is also an example of a set identifiable model: although the exact value of β cannot be learned, we can guarantee that it must lie somewhere in the interval (β_yx, 1÷β_xy), where β_yx is the coefficient in OLS regression of y on x, and β_xy is the coefficient in OLS regression of x on y.^[5]

If we abandon the normality assumption and require that x* were not normally distributed, retaining only the independence condition ε ⊥ η ⊥ x*, then the model becomes identifiable.^[4]

• System identification
• Structural identifiability
• Observability
• Simultaneous equations model