Hubbry Logo
Minimum-variance unbiased estimatorMinimum-variance unbiased estimatorMain
Open search
Minimum-variance unbiased estimator
Community hub
Minimum-variance unbiased estimator
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Minimum-variance unbiased estimator
Minimum-variance unbiased estimator
Minimum-variance unbiased estimator
View on Wikipedia
from Wikipedia

In statistics a minimum-variance unbiased estimator (MVUE) or uniformly minimum-variance unbiased estimator (UMVUE) is an unbiased estimator that has lower variance than any other unbiased estimator for all possible values of the parameter.

For practical statistics problems, it is important to determine the MVUE if one exists, since less-than-optimal procedures would naturally be avoided, other things being equal. This has led to substantial development of statistical theory related to the problem of optimal estimation.

While combining the constraint of unbiasedness with the desirability metric of least variance leads to good results in most practical settings—making MVUE a natural starting point for a broad range of analyses—a targeted specification may perform better for a given problem; thus, MVUE is not always the best stopping point.

Definition

[edit]

Consider estimation of based on data i.i.d. from some member of a family of densities , where is the parameter space. An unbiased estimator of is UMVUE if ,

for any other unbiased estimator

If an unbiased estimator of exists, then one can prove there is an essentially unique MVUE.[1] Using the Rao–Blackwell theorem one can also prove that determining the MVUE is simply a matter of finding a complete sufficient statistic for the family and conditioning any unbiased estimator on it.

Further, by the Lehmann–Scheffé theorem, an unbiased estimator that is a function of a complete, sufficient statistic is the UMVUE estimator.

Put formally, suppose is unbiased for , and that is a complete sufficient statistic for the family of densities. Then

is the MVUE for

A Bayesian analog is a Bayes estimator, particularly with minimum mean square error (MMSE).

Estimator selection

[edit]

An efficient estimator need not exist, but if it does and if it is unbiased, it is the MVUE. Since the mean squared error (MSE) of an estimator δ is

the MVUE minimizes MSE among unbiased estimators. In some cases biased estimators have lower MSE because they have a smaller variance than does any unbiased estimator; see estimator bias.

Example

[edit]

Consider the data to be a single observation from an absolutely continuous distribution on with density

where θ > 0, and we wish to find the UMVU estimator of

First we recognize that the density can be written as

which is an exponential family with sufficient statistic . In fact this is a full rank exponential family, and therefore is complete sufficient. See exponential family for a derivation which shows

Therefore,

Here we use Lehmann–Scheffé theorem to get the MVUE. Clearly, is unbiased and is complete sufficient, thus the UMVU estimator is

This example illustrates that an unbiased function of the complete sufficient statistic will be UMVU, as Lehmann–Scheffé theorem states.

Other examples

[edit]
  • For a normal distribution with unknown mean and variance, the sample mean and (unbiased) sample variance are the MVUEs for the population mean and population variance.
    However, the sample standard deviation is not unbiased for the population standard deviation – see unbiased estimation of standard deviation.
    Further, for other distributions the sample mean and sample variance are not in general MVUEs – for a uniform distribution with unknown upper and lower bounds, the mid-range is the MVUE for the population mean.
  • If k exemplars are chosen (without replacement) from a discrete uniform distribution over the set {1, 2, ..., N} with unknown upper bound N, the MVUE for N is: where m is the sample maximum. This is a scaled and shifted (so unbiased) transform of the sample maximum, which is a sufficient and complete statistic. See German tank problem for details.

See also

[edit]

Bayesian analogs

[edit]

References

[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Minimum-variance unbiased estimator

View on Grokipedia
from Grokipedia
In , a minimum-variance unbiased (MVUE) is an unbiased of a that achieves the lowest possible variance among all unbiased of that . This property makes it a desirable for , as it balances lack of bias with optimal precision under squared error loss. The concept of an MVUE is fundamentally linked to the Cramér–Rao lower bound (CRLB), which establishes a theoretical minimum on the variance of any unbiased based on the in the data. An unbiased attains MVUE status if its variance equals the CRLB, rendering it efficient in the sense that no other unbiased can perform better. The CRLB was independently derived by in his 1945 paper on the accuracy attainable in and by Harald Cramér in his 1946 work on sufficient statistics and optimal . In practice, the uniformly minimum-variance unbiased (UMVUE) extends this idea by requiring minimum variance uniformly across the entire space, often leveraging complete sufficient statistics via the Lehmann–Scheffé theorem. Examples include the sample mean for the mean of a and the sample proportion for a binomial probability, both of which achieve the CRLB under regularity conditions.

Fundamentals of Estimation

Unbiased Estimators

An estimator is a function of the observed sample data designed to approximate an unknown parameter of the underlying probability distribution. An estimator θ^(X)\hat{\theta}(X) is unbiased for the parameter θ\theta if its expected value equals the true parameter value for all θ\theta in the parameter space, formally E[θ^(X)]=θE[\hat{\theta}(X)] = \theta, where XX denotes the random sample. The θ^\hat{\theta} is defined as Bias(θ^)=E[θ^(X)]−θ\text{Bias}(\hat{\theta}) = E[\hat{\theta}(X)] - \theta, measuring the systematic deviation from the true ; an estimator is unbiased precisely when this bias is zero. Due to the linearity of expectation, the sum or of unbiased estimators is also unbiased. The concept of unbiased estimation was introduced by in the early , in the context of estimation for astronomical data. For intuition, consider estimating the bias probability pp of a coin from nn independent flips, where heads is coded as 1 and tails as 0; the sample proportion p^=1n∑i=1nXi\hat{p} = \frac{1}{n} \sum_{i=1}^n X_i (the sample mean) is unbiased because E[p^]=pE[\hat{p}] = p.

Variance in Estimators

In statistics, the variance of an estimator θ^\hat{\theta} of a parameter θ\theta, based on data XX, is defined as Var⁡(θ^)=E[(θ^(X)−E[θ^(X)])2]\operatorname{Var}(\hat{\theta}) = E[(\hat{\theta}(X) - E[\hat{\theta}(X)])^2]. This quantity measures the expected squared deviation of the estimator from its own expected value, thereby quantifying the uncertainty or dispersion in the possible values that θ^\hat{\theta} can take across repeated sampling from the same distribution. Lower variance indicates higher precision, as the estimator's outcomes cluster more tightly around its mean. A key metric for evaluating estimator performance is the mean squared error (MSE), which decomposes as MSE⁡(θ^)=Var⁡(θ^)+[Bias⁡(θ^)]2\operatorname{MSE}(\hat{\theta}) = \operatorname{Var}(\hat{\theta}) + [\operatorname{Bias}(\hat{\theta})]^2, where Bias⁡(θ^)=E[θ^(X)]−θ\operatorname{Bias}(\hat{\theta}) = E[\hat{\theta}(X)] - \theta. This relationship highlights variance's contribution to total estimation error, distinct from systematic deviations captured by . The variance of any is inherently non-negative, as it represents a second moment of deviation that cannot be negative. For unbiased estimators, where is zero, MSE reduces exactly to the variance, making it the direct measure of accuracy. Although low variance is desirable for consistent and precise estimates, it alone does not guarantee good performance; an estimator with minimal variance but substantial bias can still produce systematically inaccurate results. Balancing low variance with low bias is thus critical for reliable . As an illustrative example, the sample mean Xˉn=1n∑i=1nXi\bar{X}_n = \frac{1}{n} \sum_{i=1}^n X_i serves as an of the population mean μ\mu for independent and identically distributed samples X1,…,XnX_1, \dots, X_n from a distribution with finite variance σ2\sigma^2. Its variance is Var⁡(Xˉn)=σ2n\operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n}, showing that estimation uncertainty decreases proportionally with increasing sample size nn.

Core Concepts

Definition of MVUE

In statistics, a minimum-variance unbiased estimator (MVUE) of a parameter θ\theta is an unbiased estimator θ^∗\hat{\theta}^* such that its variance satisfies Var⁡(θ^∗)≤Var⁡(θ^)\operatorname{Var}(\hat{\theta}^*) \leq \operatorname{Var}(\hat{\theta}) for every other unbiased estimator θ^\hat{\theta} of θ\theta, with the inequality holding for all θ\theta in the parameter space Θ\Theta. This property establishes θ^∗\hat{\theta}^* as optimal within the class of unbiased estimators, minimizing the dispersion of estimates around the true parameter value while preserving unbiasedness, defined as E[θ^]=θ\mathbb{E}[\hat{\theta}] = \theta. The term is often synonymous with uniformly minimum-variance unbiased estimator (UMVUE). Relative efficiency provides a measure of performance comparison among unbiased estimators, defined as Eff⁡(θ^,θ^∗)=Var⁡(θ^∗)Var⁡(θ^)\operatorname{Eff}(\hat{\theta}, \hat{\theta}^*) = \frac{\operatorname{Var}(\hat{\theta}^*)}{\operatorname{Var}(\hat{\theta})}, where the value is at most 1 for an MVUE θ^∗\hat{\theta}^*, indicating superior precision. Under certain regularity conditions, such as the existence of a complete sufficient statistic TT, the MVUE is unique up to sets of measure zero. To see this, suppose θ^1\hat{\theta}_1 and θ^2\hat{\theta}_2 are two unbiased estimators with minimum variance, both functions of TT. Their difference θ^1−θ^2\hat{\theta}_1 - \hat{\theta}_2 is then an unbiased estimator of zero that is a function of TT; completeness implies P(θ^1−θ^2=0)=1\mathbb{P}(\hat{\theta}_1 - \hat{\theta}_2 = 0) = 1, establishing . The related Cramér–Rao lower bound, providing a theoretical minimum variance for unbiased estimators, was derived by in 1945 and Harald Cramér in 1946. The MVUE concept is further developed through results like the Lehmann–Scheffé theorem.

Efficiency and the Cramer-Rao Bound

In , an unbiased θ^\hat{\theta} of a θ\theta is defined as efficient if its variance equals the Cramér–Rao lower bound (CRLB), that is, Var⁡(θ^)=1/I(θ)\operatorname{Var}(\hat{\theta}) = 1 / I(\theta), where I(θ)I(\theta) denotes the in the sample. This bound represents the theoretical minimum variance achievable by any unbiased , serving as a benchmark for the performance of the minimum-variance unbiased (MVUE). Efficiency thus indicates that the extracts all available about θ\theta from the data, with no room for improvement in variance among unbiased alternatives. The CRLB is derived for parametric families of distributions satisfying certain regularity conditions, such as the differentiability of the log-likelihood function with respect to θ\theta and the ability to interchange differentiation and integration. Specifically, for an unbiased θ^\hat{\theta} based on a sample from a distribution with probability f(x;θ)f(x; \theta), the bound states that Var⁡(θ^)≥1/I(θ)\operatorname{Var}(\hat{\theta}) \geq 1 / I(\theta), with equality holding if the is a function of the in a manner that satisfies the conditions for attainment. The derivation typically proceeds by noting that Cov⁡(θ^,∂/∂θlog⁡f(X;θ))=1\operatorname{Cov}(\hat{\theta}, \partial / \partial \theta \log f(X; \theta)) = 1 under regularity conditions, and applying the : [Cov⁡(θ^,score⁡)]2≤Var⁡(θ^)⋅I(θ)[\operatorname{Cov}(\hat{\theta}, \operatorname{score})]^2 \leq \operatorname{Var}(\hat{\theta}) \cdot I(\theta), yielding the variance lower bound. The Fisher information I(θ)I(\theta) quantifies the amount of information the sample carries about θ\theta and is given by I(θ)=E[(∂∂θlog⁡f(X;θ))2]=−E[∂2∂θ2log⁡f(X;θ)],I(\theta) = \mathbb{E}\left[ \left( \frac{\partial}{\partial \theta} \log f(X; \theta) \right)^2 \right] = -\mathbb{E}\left[ \frac{\partial^2}{\partial \theta^2} \log f(X; \theta) \right], where the expectations are taken with respect to the distribution parameterized by θ\theta. This measure, introduced by Ronald Fisher, captures the curvature of the log-likelihood and the sensitivity of the distribution to changes in θ\theta. For the CRLB to be attainable, the distribution must belong to the regular exponential family or satisfy similar conditions ensuring the existence of an estimator that achieves equality in the bound. In such cases, the MVUE coincides with an efficient estimator, fully realizing the theoretical variance minimum. In the multiparameter setting, where θ=(θ1,…,θk)⊤\theta = (\theta_1, \dots, \theta_k)^\top is a vector of unknown parameters, the CRLB extends to a matrix inequality: the of any unbiased θ^\hat{\theta} satisfies Cov⁡(θ^)≥I(θ)−1\operatorname{Cov}(\hat{\theta}) \geq I(\theta)^{-1}, where I(θ)I(\theta) is now the k×kk \times k matrix with elements Iij(θ)=E[∂∂θilog⁡f(X;θ)⋅∂∂θjlog⁡f(X;θ)]I_{ij}(\theta) = \mathbb{E}\left[ \frac{\partial}{\partial \theta_i} \log f(X; \theta) \cdot \frac{\partial}{\partial \theta_j} \log f(X; \theta) \right]. This form accounts for correlations between estimators of different parameters, providing bounds on variances and covariances via the inverse matrix elements. Equality holds componentwise under analogous regularity conditions, such as the information matrix being positive definite and the being asymptotically normal. When an MVUE exists in these multiparameter regular cases, it attains the CRLB, ensuring optimality across the parameter vector.

Construction Methods

Complete Sufficient Statistics

In statistical , a T(X)T(\mathbf{X}) is said to be sufficient for the θ\theta if the conditional distribution of the sample X\mathbf{X} given T(X)T(\mathbf{X}) does not depend on θ\theta. This means that T(X)T(\mathbf{X}) captures all the about θ\theta contained in the sample, allowing for reduction without loss of inferential content. The concept was formalized through the Neyman-Fisher factorization theorem, which provides a practical criterion: the joint probability density (or mass) function can be expressed as f(x;θ)=g(T(x);θ) h(x)f(\mathbf{x}; \theta) = g(T(\mathbf{x}); \theta) \, h(\mathbf{x}), where gg depends on θ\theta only through TT, and hh is independent of θ\theta. Among sufficient statistics, a minimal sufficient statistic represents the coarsest possible reduction of the while remaining sufficient. It is a sufficient statistic such that every other sufficient statistic can be expressed as a function of it, ensuring maximal data compression without sacrificing about θ\theta. This notion arises naturally in the context of partitioning the based on likelihood ratios and is essential for identifying the simplest form of sufficient reduction. Completeness is a property of the family of distributions of a statistic TT that complements sufficiency by preventing "wasted" information in unbiased estimation. Specifically, the family {Pθ:θ∈Θ}\{ P_\theta : \theta \in \Theta \} of the distribution of TT is complete if, for every measurable function gg such that Eθ[g(T)]=0E_\theta [g(T)] = 0 for all θ∈Θ\theta \in \Theta, it follows that Pθ(g(T)=0)=1P_\theta (g(T) = 0) = 1 for all θ\theta. This ensures that no non-trivial unbiased estimator of zero exists within the family, which is crucial for the uniqueness of minimum-variance unbiased estimators based on complete sufficient statistics. The Rao-Blackwell theorem leverages sufficiency to improve estimators. If δ(X)\delta(\mathbf{X}) is an unbiased estimator of θ\theta and TT is sufficient for θ\theta, then the refined estimator δ′(T)=E[δ(X)∣T]\delta'(T) = E[\delta(\mathbf{X}) \mid T] is also unbiased for θ\theta and satisfies Var(δ′(T))≤Var(δ(X))\mathrm{Var}(\delta'(T)) \leq \mathrm{Var}(\delta(\mathbf{X})), with equality if and only if δ(X)\delta(\mathbf{X}) is already a function of TT. This theorem provides a method to condition any initial unbiased estimator on a sufficient statistic, yielding a more efficient alternative that serves as a building block for achieving minimum variance. A related concept is bounded completeness, which relaxes the condition to bounded functions gg with ∣g(T)∣≤M|g(T)| \leq M for some finite MM. The family is boundedly complete if Eθ[g(T)]=0E_\theta [g(T)] = 0 for all θ\theta implies g(T)=0g(T) = 0 . This weaker property still guarantees uniqueness for certain classes of unbiased estimators and plays a key role in results like , where a boundedly complete is independent of ancillary statistics, aiding in the construction of optimal estimators.

Lehmann-Scheffe Theorem

The Lehmann-Scheffé theorem establishes a key result in theory, linking completeness and to the existence and uniqueness of minimum-variance unbiased estimators. Specifically, the theorem states that if TT is a complete sufficient statistic for a θ\theta, and θ^=g(T)\hat{\theta} = g(T) is any unbiased of θ\theta (i.e., Eθ[g(T)]=θE_\theta[g(T)] = \theta for all θ\theta in the parameter space), then g(T)g(T) is the unique uniformly minimum-variance unbiased estimator (UMVUE) of θ\theta. The proof outline proceeds in two main steps, building on prior results in . First, the Rao-Blackwell guarantees that for any unbiased estimator WW of θ\theta, the E(W∣T)E(W \mid T) is also unbiased and has variance no greater than that of WW, with equality only if WW is already a function of TT. Second, completeness of TT ensures uniqueness: suppose there exists another unbiased estimator θ~\tilde{\theta} with variance less than or equal to that of g(T)g(T); then E[(θ~−g(T))∣T]=0E[(\tilde{\theta} - g(T)) \mid T] = 0 for all θ\theta, implying by completeness that θ~=g(T)\tilde{\theta} = g(T) . The holds under regularity conditions that ensure the existence of a , such as those satisfied by full-rank with an open parameter space. In such families, the natural sufficient statistic (e.g., the sum of observations in a one-parameter ) is complete, allowing direct application of the theorem to derive UMVUEs. The result generalizes straightforwardly to estimating any function ψ(θ)\psi(\theta): if TT is complete sufficient and ψ^=h(T)\hat{\psi} = h(T) satisfies Eθ[h(T)]=ψ(θ)E_\theta[h(T)] = \psi(\theta) for all θ\theta, then h(T)h(T) is the unique UMVUE of ψ(θ)\psi(\theta). This extension is immediate from the original proof, replacing θ\theta with ψ(θ)\psi(\theta). Despite its utility, the theorem has limitations, as not all parametric families possess a complete . For instance, in the uniform distribution on [θ,θ+1][\theta, \theta + 1], the joint order statistics (X(1),X(n))(X_{(1)}, X_{(n)}) form a minimal , but it is not complete, precluding the guarantee of a unique UMVUE via this method.

Applications and Examples

Uniform Distribution Example

Consider an independent and identically distributed (i.i.d.) sample X1,…,XnX_1, \dots, X_n drawn from the uniform distribution on the interval [0,θ][0, \theta], where θ>0\theta > 0 is the unknown to be estimated. The for each XiX_i is f(x;θ)=1/θf(x; \theta) = 1/\theta for 0≤x≤θ0 \leq x \leq \theta, and 0 otherwise. The maximum likelihood estimator (MLE) for θ\theta is the sample maximum, θ^MLE=X(n)=max⁡{X1,…,Xn}\hat{\theta}_{\text{MLE}} = X_{(n)} = \max\{X_1, \dots, X_n\}. To derive this, the is L(θ)=θ−nL(\theta) = \theta^{-n} if X(n)≤θX_{(n)} \leq \theta and 0 otherwise; maximizing it yields θ^MLE=X(n)\hat{\theta}_{\text{MLE}} = X_{(n)}. However, this estimator is biased, as its is E[X(n)]=nθ/(n+1)E[X_{(n)}] = n\theta / (n+1). An unbiased estimator is obtained by adjusting for the bias: θ^=n+1nX(n)\hat{\theta} = \frac{n+1}{n} X_{(n)}. To verify unbiasedness, note that E[θ^]=n+1nE[X(n)]=n+1n⋅nθn+1=θE[\hat{\theta}] = \frac{n+1}{n} E[X_{(n)}] = \frac{n+1}{n} \cdot \frac{n\theta}{n+1} = \theta. The X(n)X_{(n)} is a for θ\theta by the factorization theorem, as the joint density factors into g(X(n);θ)⋅h(x)g(X_{(n)}; \theta) \cdot h(\mathbf{x}), where g(X(n);θ)=θ−nI(0≤X(n)≤θ)g(X_{(n)}; \theta) = \theta^{-n} I(0 \leq X_{(n)} \leq \theta) and h(x)h(\mathbf{x}) is independent of θ\theta. Moreover, X(n)X_{(n)} is complete because the uniform family satisfies the conditions for completeness of the maximal . By the Lehmann-Scheffé theorem, any unbiased function of a complete , such as θ^\hat{\theta}, is the minimum-variance unbiased estimator (MVUE) for θ\theta. To compute the variance, first find the distribution of X(n)X_{(n)}. The cumulative distribution function is FX(n)(x)=[x/θ]nF_{X_{(n)}}(x) = [x/\theta]^n for 0≤x≤θ0 \leq x \leq \theta, so the density is fX(n)(x)=nxn−1/θnf_{X_{(n)}}(x) = n x^{n-1} / \theta^n. Then, E[X(n)]=n∫0θxn/θn dx=nθ/(n+1)E[X_{(n)}] = n \int_0^\theta x^n / \theta^n \, dx = n \theta / (n+1), as before. For the second moment, E[X(n)2]=n∫0θxn+1/θn dx=nθ2/(n+2)E[X_{(n)}^2] = n \int_0^\theta x^{n+1} / \theta^n \, dx = n \theta^2 / (n+2). Thus, Var⁡(X(n))=E[X(n)2]−[E[X(n)]]2=nθ2/(n+2)−[nθ/(n+1)]2=nθ2/[(n+1)2(n+2)]\operatorname{Var}(X_{(n)}) = E[X_{(n)}^2] - [E[X_{(n)}]]^2 = n \theta^2 / (n+2) - [n \theta / (n+1)]^2 = n \theta^2 / [(n+1)^2 (n+2)]. Scaling for the unbiased estimator gives Var⁡(θ^)=(n+1n)2Var⁡(X(n))=θ2/[n(n+2)]\operatorname{Var}(\hat{\theta}) = \left( \frac{n+1}{n} \right)^2 \operatorname{Var}(X_{(n)}) = \theta^2 / [n(n+2)].

Exponential Distribution Example

Consider a random sample X1,X2,…,XnX_1, X_2, \dots, X_n from the exponential distribution with rate parameter λ>0\lambda > 0, where the pdf is given by f(x;λ)=λe−λxf(x; \lambda) = \lambda e^{-\lambda x} for x≥0x \ge 0. The parameter of interest is the rate λ\lambda, which represents the expected number of events per unit time. The sufficient statistic for λ\lambda is T=∑i=1nXiT = \sum_{i=1}^n X_i, and TT follows a Gamma distribution with shape parameter nn and rate parameter λ\lambda. An unbiased estimator for λ\lambda is λ^=n−1T\hat{\lambda} = \frac{n-1}{T}, derived by adjusting the method of moments estimator n/Tn/T (which is biased) to achieve unbiasedness, since E[1/T]=λ/(n−1)E[1/T] = \lambda / (n-1) for n>1n > 1. To establish that λ^\hat{\lambda} is the minimum-variance unbiased (MVUE), observe that TT is a complete for λ\lambda in this one-parameter . Since λ^\hat{\lambda} is an unbiased function of the complete sufficient statistic TT, the Lehmann-Scheffé theorem guarantees that λ^\hat{\lambda} is the unique MVUE for λ\lambda. The variance of this MVUE is \Var(λ^)=λ2n−2\Var(\hat{\lambda}) = \frac{\lambda^2}{n-2} for n>2n > 2. The Cramér-Rao lower bound (CRLB) provides a lower limit on the variance of any unbiased of λ\lambda, given by λ2n\frac{\lambda^2}{n}. The relative efficiency of λ^\hat{\lambda} is thus n−2n\frac{n-2}{n}, which approaches 1 as the sample size nn increases. Numerical simulations illustrate that the estimator's performance relative to the CRLB improves with larger nn, confirming its asymptotic efficiency.

Limitations and Extensions

Comparison to Biased Estimators

While minimum-variance unbiased estimators (MVUEs) minimize variance among unbiased estimators, their (MSE) equals their variance since is zero by definition. In contrast, biased estimators have MSE equal to variance plus squared , allowing for potentially lower overall MSE if the introduced is small and the reduction in variance is substantial. A prominent class of biased estimators that outperform MVUEs in MSE are shrinkage estimators, which pull estimates toward a central value to reduce variability. The James-Stein estimator exemplifies this for estimating the vector of a under quadratic loss; it shrinks the unbiased sample toward zero and dominates the sample in MSE for dimensions greater than two, regardless of the true . In the uniform distribution on [0, θ], the maximum likelihood estimator (MLE) θ̂ = max(X_i) is biased downward but achieves lower MSE than the MVUE ((n+1)/n) max(X_i) for finite sample sizes n, as the bias term is outweighed by the MLE's smaller variance. MVUEs remain preferable in large samples, where they often attain asymptotic and unbiasedness ensures reliable long-run average performance, or in regulatory contexts such as clinical trials where institutional constraints demand unbiased estimates to avoid systematic over- or underestimation. Asymptotically, MVUEs are typically under regularity conditions, matching the Cramér-Rao lower bound, but biased estimators like shrinkage methods can yield lower MSE in small samples by exploiting the bias-variance trade-off.

Bayesian and Generalized Analogs

In , point estimators serve as counterparts to frequentist minimum-variance unbiased estimators (MVUE), but prioritize minimizing the expected posterior loss rather than enforcing unbiasedness and minimizing variance within that constraint. The under squared error loss is the posterior mean, which achieves the minimum posterior expected squared error by integrating prior beliefs with observed data. This approach contrasts with MVUE by allowing bias in the frequentist sense to incorporate prior information, often resulting in lower overall risk when priors reflect substantive knowledge. Bayesian credible intervals provide an analog to frequentist confidence intervals, quantifying as the range containing the with a specified , such as 95%. Unlike confidence intervals, which achieve coverage in repeated sampling under the true , credible intervals directly interpret probability conditional on the and prior. Under squared error loss, Bayesian emphasizes minimizing posterior , offering a coherent measure of performance that aligns with decision-theoretic optimality. Empirical Bayes methods and hierarchical models introduce shrinkage estimators as biased analogs to MVUE, leveraging data-driven priors to reduce at the cost of unbiasedness. In hierarchical settings, parameters are treated as draws from a higher-level distribution, leading to estimators that pull individual estimates toward a . The James-Stein estimator exemplifies this for estimating multiple normal means under squared error loss, shrinking sample means toward their and dominating the unbiased maximum likelihood when the dimension exceeds two, with risk reduction up to 12% in high dimensions. Generalized analogs to MVUE arise in through estimators that maintain unbiasedness under group invariance, minimizing the over invariant spaces. These estimators achieve the property among invariant unbiased rules, extending MVUE to structured problems like location-scale families where transformations preserve the model. Such generalizations ensure robustness in invariant decision problems, often coinciding with Bayes rules under least favorable priors. From a Bayesian viewpoint, frequentist MVUE limitations stem from disregarding prior information, which can inflate posterior risk in scenarios with informative priors. For estimating the rate θ\theta of an exponential distribution based on i.i.d. samples X1,…,Xn∼exp⁡(θ)X_1, \dots, X_n \sim \exp(\theta), the MVUE is (n−1)/∑Xi(n-1) / \sum X_i, but with a gamma conjugate prior θ∼Γ(α,β)\theta \sim \Gamma(\alpha, \beta), the posterior is Γ(α+n,β+∑Xi)\Gamma(\alpha + n, \beta + \sum X_i), and the posterior mean (α+n)/(β+∑Xi)(\alpha + n) / (\beta + \sum X_i) shrinks the MLE θ^=n/∑Xi\hat{\theta} = n / \sum X_i toward the prior mean α/β\alpha / \beta, typically yielding lower mean squared error when the prior is accurate.

References

  1. https://www.stat.purdue.edu/~dasgupta/528-5.pdf
  2. https://www2.stat.duke.edu/~pdh10/Teaching/732/Notes/unbiasedness.pdf
  3. https://www.math.uri.edu/~merino/spring06/mth452/mth452Sec5.5%2B5.6%2B5.7print.pdf
  4. https://web.ma.utexas.edu/users/gordanz/notes/estimators_color.pdf
  5. https://online.stat.psu.edu/stat415/lesson/1/1.3
  6. https://web.stanford.edu/class/archive/cs/cs109/cs109.1218/files/student_drive/7.6.pdf
  7. https://link.springer.com/article/10.1007/s00407-022-00291-w
  8. https://math.arizona.edu/~jwatkins/N_unbiased.pdf
  9. https://www.stat.cmu.edu/~larry/=stat401/lecture-01.pdf
  10. https://www.khoury.northeastern.edu/home/vip/teach/MLcourse/4_boosting/materials/MSE_bias_var.pdf
  11. https://online.stat.psu.edu/stat414/lesson/24/24.4
  12. https://uregina.ca/~kozdron/Teaching/Regina/252Winter16/Handouts/ch3.pdf
  13. https://engineering.purdue.edu/ChanGroup/ECE645Notes/StudentLecture06.pdf
  14. https://www.stats.ox.ac.uk/~steffen/teaching/bs2siMT04/si3c.pdf
  15. https://www.researchgate.net/publication/302324470_Introduction_to_Rao_1945_Information_and_the_Accuracy_Attainable_in_the_Estimation_of_Statistical_Parameters
  16. https://gwern.net/doc/statistics/1992-rao.pdf
  17. https://jhanley.biostat.mcgill.ca/bios601/Likelihood/Fisher1922.pdf
  18. https://www.scirp.org/reference/referencespapers?referenceid=3208837
  19. https://parker.ad.siu.edu/Olive/ich4.pdf
  20. https://genome.sph.umich.edu/w/images/3/38/Bios602-wi13-lec06-handout.pdf
  21. https://pages.stat.wisc.edu/~shao/stat709/stat709-15.pdf
  22. https://www.ams.sunysb.edu/~zhu/ams570/Solutions-Casella-Berger.pdf
  23. https://link.springer.com/chapter/10.1007/978-1-4614-1412-4_23
  24. http://www.stat.yale.edu/~hz68/619/Stein-1961.pdf
  25. https://dept.stat.lsa.umich.edu/~moulib/StatNotes2.pdf
  26. https://www2.stat.duke.edu/~pdh10/Teaching/732/Notes/efficiency.pdf
  27. https://pmc.ncbi.nlm.nih.gov/articles/PMC3813981/
  28. https://www.princeton.edu/~umueller/medunbiased.pdf
  29. https://link.springer.com/book/10.1007/978-1-4757-4286-2
  30. https://pmc.ncbi.nlm.nih.gov/articles/PMC6630113/
  31. https://people.eecs.berkeley.edu/~jordan/courses/260-spring10/other-readings/chapter9.pdf
  32. https://math.stackexchange.com/questions/2034206/finding-umvue-of-theta-when-the-underlying-distribution-is-exponential-distri
Add your contribution
Related Hubs
User Avatar
No comments yet.