Hubbry Logo
Likelihood-ratio testLikelihood-ratio testMain
Open search
Likelihood-ratio test
Community hub
Likelihood-ratio test
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Likelihood-ratio test
Likelihood-ratio test
Likelihood-ratio test
View on Wikipedia
from Wikipedia

In statistics, the likelihood-ratio test is a hypothesis test that involves comparing the goodness of fit of two competing statistical models, typically one found by maximization over the entire parameter space and another found after imposing some constraint, based on the ratio of their likelihoods. If the more constrained model (i.e., the null hypothesis) is supported by the observed data, the two likelihoods should not differ by more than sampling error.[1] Thus the likelihood-ratio test tests whether this ratio is significantly different from one, or equivalently whether its natural logarithm is significantly different from zero.

The likelihood-ratio test, also known as Wilks test,[2] is the oldest of the three classical approaches to hypothesis testing, together with the Lagrange multiplier test and the Wald test.[3] In fact, the latter two can be conceptualized as approximations to the likelihood-ratio test, and are asymptotically equivalent.[4][5][6] In the case of comparing two models each of which has no unknown parameters, use of the likelihood-ratio test can be justified by the Neyman–Pearson lemma. The lemma demonstrates that the test has the highest power among all competitors.[7]

Definition

[edit]

General

[edit]

Suppose that we have a statistical model with parameter space . A null hypothesis is often stated by saying that the parameter lies in a specified subset of . The alternative hypothesis is thus that lies in the complement of , i.e. in , which is denoted by . The likelihood ratio test statistic for the null hypothesis is given by:[8]

where the quantity inside the brackets is called the likelihood ratio. Here, the notation refers to the supremum. As all likelihoods are positive, and as the constrained maximum cannot exceed the unconstrained maximum, the likelihood ratio is bounded between zero and one.

Often the likelihood-ratio test statistic is expressed as a difference between the log-likelihoods where is the logarithm of the maximized likelihood function , and is the maximal value in the special case that the null hypothesis is true (but not necessarily a value that maximizes for the sampled data) and denote the respective arguments of the maxima and the allowed ranges they're embedded in. Multiplying by −2 ensures mathematically that (by Wilks' theorem) converges asymptotically to being χ²-distributed if the null hypothesis happens to be true.[9] The finite-sample distributions of likelihood-ratio statistics are generally unknown.[10]

The likelihood-ratio test requires that the models be nested – i.e. the more complex model can be transformed into the simpler model by imposing constraints on the former's parameters. Many common test statistics are tests for nested models and can be phrased as log-likelihood ratios or approximations thereof: e.g. the Z-test, the F-test, the G-test, and Pearson's chi-squared test; for an illustration with the one-sample t-test, see below.

If the models are not nested, then instead of the likelihood-ratio test, there is a generalization of the test that can usually be used: for details, see relative likelihood.

Case of simple hypotheses

[edit]
Main article: Neyman–Pearson lemma

A simple-vs.-simple hypothesis test has completely specified models under both the null hypothesis and the alternative hypothesis, which for convenience are written in terms of fixed values of a notional parameter :

In this case, under either hypothesis, the distribution of the data is fully specified: there are no unknown parameters to estimate. For this case, a variant of the likelihood-ratio test is available:[11][12]

Some older references may use the reciprocal of the function above as the definition.[13] Thus, the likelihood ratio is small if the alternative model is better than the null model.

The likelihood-ratio test provides the decision rule as follows:

  • If , do not reject ;
  • If , reject ;
  • If , reject with probability .

The values and are usually chosen to obtain a specified significance level , via the relation The Neyman–Pearson lemma states that this likelihood-ratio test is the most powerful among all level tests for this case.[7][12]

Interpretation

[edit]

The likelihood ratio is a function of the data ; therefore, it is a statistic, although unusual in that the statistic's value depends on a parameter, . The likelihood-ratio test rejects the null hypothesis if the value of this statistic is too small. How small is too small depends on the significance level of the test, i.e. on what probability of Type I error is considered tolerable (Type I errors consist of the rejection of a null hypothesis that is true).

The numerator corresponds to the likelihood of an observed outcome under the null hypothesis. The denominator corresponds to the maximum likelihood of an observed outcome, varying parameters over the whole parameter space. The numerator of this ratio is less than the denominator; so, the likelihood ratio is between 0 and 1. Low values of the likelihood ratio mean that the observed result was much less likely to occur under the null hypothesis as compared to the alternative. High values of the statistic mean that the observed outcome was nearly as likely to occur under the null hypothesis as the alternative, and so the null hypothesis cannot be rejected.

An example

[edit]

The following example is adapted and abridged from Stuart, Ord & Arnold (1999, §22.2).

Suppose that we have a random sample, of size n, from a population that is normally-distributed. Both the mean, μ, and the standard deviation, σ, of the population are unknown. We want to test whether the mean is equal to a given value, μ0 .

Thus, our null hypothesis is H0:  μ = μ0  and our alternative hypothesis is H1:  μ ≠ μ0 . The likelihood function is

With some calculation (omitted here), it can then be shown that where t is the t-statistic with n − 1 degrees of freedom. Hence we may use the known exact distribution of tn−1 to draw inferences.

Asymptotic distribution: Wilks’ theorem

[edit]
Main article: Wilks' theorem

If the distribution of the likelihood ratio corresponding to a particular null and alternative hypothesis can be explicitly determined then it can directly be used to form decision regions (to sustain or reject the null hypothesis). In most cases, however, the exact distribution of the likelihood ratio corresponding to specific hypotheses is very difficult to determine.[citation needed]

Assuming H0 is true, there is a fundamental result by Samuel S. Wilks: As the sample size approaches , and if the null hypothesis lies strictly within the interior of the parameter space, the test statistic defined above will be asymptotically chi-squared distributed () with degrees of freedom equal to the difference in dimensionality of and .[14] This implies that for a great variety of hypotheses, we can calculate the likelihood ratio for the data and then compare the observed to the value corresponding to a desired statistical significance as an approximate statistical test. Other extensions exist.[which?]

See also

[edit]

References

[edit]

Further reading

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Likelihood-ratio test

View on Grokipedia
from Grokipedia
The likelihood-ratio test (LRT) is a fundamental statistical procedure in hypothesis testing that assesses the goodness-of-fit between observed and two competing parametric models—one corresponding to the and the other to the alternative—by computing the ratio of their maximized likelihood functions. The test rejects the if this ratio is below a critical threshold, indicating that the provides a substantially better explanation of the . Developed by and in their seminal 1933 paper, the LRT emerged as a response to the need for optimal tests in the Neyman-Pearson framework, which emphasizes controlling the type I error rate while maximizing power against specific alternatives. The Neyman-Pearson lemma establishes that, for simple null and alternative hypotheses (where both specify exact parameter values), the LRT yields the most powerful test of a given size, meaning it achieves the highest probability of correctly rejecting the null when it is false. This optimality property underpins its foundational role in classical statistics, distinguishing it from earlier approaches like those of , which focused more on p-values without explicit power considerations. For more general composite hypotheses (where hypotheses involve ranges of parameters), the test is extended to the generalized likelihood-ratio test, which compares the maximum likelihood under the full model to that under the restricted null model. Under standard regularity conditions—such as the models being identifiable and the data being independent and identically distributed—the test statistic, defined as -2 times the log of the likelihood ratio, converges asymptotically to a chi-squared distribution with degrees of freedom equal to the difference in the number of free parameters between the unrestricted and restricted models. This asymptotic property enables practical computation of p-values and critical values even for large samples, making the LRT robust and versatile across diverse applications.

Introduction

Overview

The (LRT) is a fundamental method in for comparing the relative fit of two competing models to a , particularly when one model is nested within the other, by evaluating the ratio of their maximized likelihood values. This approach assesses whether the additional parameters in the more complex model significantly improve the fit beyond what can be attributed to chance, aiding in and hypothesis evaluation. In the context of hypothesis testing, the LRT evaluates a H0H_0 (often corresponding to the simpler, nested model) against an HAH_A (the more general model), using the −2log⁡Λ-2 \log \Lambda, where Λ\Lambda is the defined as the supremum of the likelihood under H0H_0 divided by the supremum under HAH_A. The test rejects H0H_0 if this statistic exceeds a critical value determined by the desired significance level. The LRT derives its appeal from optimality properties under regularity conditions; for simple hypotheses (point null and alternative), the Neyman-Pearson lemma establishes it as the uniformly most powerful test of a given size, maximizing power while controlling the type I error rate. The LRT relies on key assumptions, including that the data consist of independent and identically distributed (i.i.d.) observations from the specified parametric family, and that the is correctly specified for both models. These ensure the validity of and the test's inferential properties. Under large sample sizes, the asymptotically follows a with equal to the difference in the number of parameters between the models.

Historical Development

The emerged from early 20th-century developments in , with precursors in Karl Pearson's work on goodness-of-fit testing. In 1900, Pearson introduced the chi-squared statistic as a measure for assessing whether observed frequencies in categorical data deviated significantly from expected values under a hypothesized distribution, laying foundational groundwork for ratio-based criteria in hypothesis testing that were later refined into likelihood frameworks. Building on this, Ronald A. Fisher formalized the method of maximum likelihood estimation in 1922, providing the probabilistic foundation essential for constructing likelihood ratios by maximizing the probability of observed data given parameters. Fisher's approach emphasized estimating parameters that render the data most probable, which directly influenced subsequent tests comparing likelihoods under competing hypotheses. The test itself was developed in the 1930s by Jerzy Neyman and Egon S. Pearson as a core component of their unified theory of hypothesis testing. In their seminal 1933 paper, they proposed the likelihood ratio as the optimal criterion for deciding between two simple hypotheses, rejecting the null when the ratio of likelihoods under the alternative versus null exceeds a threshold calibrated for a fixed significance level, thus establishing it as a uniformly most powerful test in that setting. Extensions to composite hypotheses followed soon after, with Samuel S. Wilks contributing key results on the asymptotic behavior of the in 1938. Wilks demonstrated that, under regularity conditions, minus twice the log-likelihood ratio converges in distribution to a chi-squared with equal to the difference in the number of free parameters between the full and restricted models, enabling practical for more complex scenarios. The likelihood-ratio test gained broader influence in modern statistics through its integration into generalized linear models, as outlined by John A. Nelder and Robert W. M. Wedderburn in 1972, where it underpins deviance-based assessments of model fit within distributions. Post-1980s advancements in further propelled its adoption, with software packages like GLIM implementing likelihood-ratio procedures for iterative model fitting and testing, facilitating widespread use in applied analyses across disciplines.

Mathematical Formulation

Likelihood Function and Hypotheses

The likelihood function serves as the foundational measure in the likelihood-ratio test, quantifying how well a parametric model explains observed data. For an independent and identically distributed (i.i.d.) sample x=(x1,…,xn)\mathbf{x} = (x_1, \dots, x_n) drawn from a probability density or mass function f(xi∣θ)f(x_i \mid \theta), where θ\theta is the parameter in the parameter space Θ\Theta, the likelihood function is defined as L(θ∣x)=∏i=1nf(xi∣θ).L(\theta \mid \mathbf{x}) = \prod_{i=1}^n f(x_i \mid \theta). This formulation treats the data as fixed and views the function as varying with respect to θ\theta, distinguishing it from the joint probability density of the sample. The assumption of i.i.d. observations ensures the product form of the likelihood, which simplifies subsequent maximizations and is a prerequisite for the test's theoretical properties. In the context of hypothesis testing, the likelihood-ratio test compares a H0:θ∈Θ0H_0: \theta \in \Theta_0 against an HA:θ∈ΘAH_A: \theta \in \Theta_A, where Θ0⊂ΘA\Theta_0 \subset \Theta_A ensures the models are nested, with Θ0\Theta_0 representing the restricted under the null. The is simple if Θ0\Theta_0 consists of a single point (e.g., a specific value of θ\theta) and composite otherwise, allowing for a range of parameter values. These hypotheses are formulated within identifiable parametric models, where distinct parameters yield distinct distributions, and the likelihood is well-defined and positive for all θ∈Θ\theta \in \Theta. Additionally, differentiability of the log-likelihood ℓ(θ∣x)=log⁡L(θ∣x)\ell(\theta \mid \mathbf{x}) = \log L(\theta \mid \mathbf{x}) with respect to θ\theta is assumed to facilitate and asymptotic approximations in later analyses. The test relies on the maximized likelihoods under each hypothesis: sup⁡θ∈Θ0L(θ∣x)\sup_{\theta \in \Theta_0} L(\theta \mid \mathbf{x}) for the null and sup⁡θ∈ΘAL(θ∣x)\sup_{\theta \in \Theta_A} L(\theta \mid \mathbf{x}) for the alternative, obtained by evaluating the likelihood at their respective maximum likelihood estimators. These suprema quantify the relative support for the hypotheses given the , assuming the models satisfy regularity conditions such as of the parameter spaces or interior maxima. For non-nested models where Θ0⊄ΘA\Theta_0 \not\subset \Theta_A, the standard likelihood-ratio test is invalid, as the ratio lacks a meaningful interpretation under the null; alternative procedures, such as the Vuong test, should be used instead. These elements form the basis for constructing the in subsequent formulations.

Test Statistic

The likelihood-ratio test statistic is defined as the ratio of the maximum likelihood under the to the maximum likelihood under the , given by Λ=sup⁡θ∈Θ0L(θ∣x)sup⁡θ∈ΘAL(θ∣x),\Lambda = \frac{\sup_{\theta \in \Theta_0} L(\theta \mid x)}{\sup_{\theta \in \Theta_A} L(\theta \mid x)}, where L(θ∣x)L(\theta \mid x) denotes the , Θ0\Theta_0 is the parameter space under the H0H_0, and ΘA\Theta_A encompasses Θ0\Theta_0 under the HAH_A. This formulation, introduced by Neyman and Pearson, measures the relative support for H0H_0 versus HAH_A based on the observed data xx. Under H0H_0, Λ≤1\Lambda \leq 1 because the supremum over the restricted space Θ0\Theta_0 cannot exceed that over the larger space ΘA\Theta_A, leading to a rejection region of the form Λ<k\Lambda < k for some critical value k<1k < 1. For computational convenience, the statistic is often transformed to λ=−2log⁡Λ\lambda = -2 \log \Lambda, which satisfies λ≥0\lambda \geq 0 under H0H_0 with rejection when λ>c\lambda > c for a suitable threshold cc. Equivalently, λ\lambda can be expressed in terms of the log-likelihood l(θ)=log⁡L(θ∣x)l(\theta) = \log L(\theta \mid x) as λ=2[l(θ^A)−l(θ^0)],\lambda = 2 \left[ l(\hat{\theta}_A) - l(\hat{\theta}_0) \right], where θ^0=arg⁡max⁡θ∈Θ0l(θ)\hat{\theta}_0 = \arg\max_{\theta \in \Theta_0} l(\theta) and θ^A=arg⁡max⁡θ∈ΘAl(θ)\hat{\theta}_A = \arg\max_{\theta \in \Theta_A} l(\theta). The maximizations required to compute λ\lambda involve maximum likelihood estimation, which for complex models typically relies on numerical optimization techniques such as the Newton-Raphson method or expectation-maximization algorithms. However, challenges arise when Θ0\Theta_0 lies on the boundary of ΘA\Theta_A, as in cases of testing for the presence of parameters (e.g., variance components equal to zero); here, the standard asymptotic chi-squared distribution for λ\lambda under H0H_0 may not hold, requiring instead a mixture of chi-squared distributions or other adjustments for valid inference. In large samples, λ\lambda facilitates inference by approximating known distributions when boundary issues are absent.

Distribution and Inference

Exact Distribution for Simple Hypotheses

When both the H0H_0 and the HAH_A are simple—meaning each fully specifies a single for the data—the likelihood-ratio test (LRT) possesses optimal properties. Specifically, the establishes that the LRT is the uniformly most powerful (UMP) test of any given significance level α\alpha for distinguishing between these two distributions. Under H0H_0, the exact of the likelihood ratio statistic Λ=L(θ0∣x)L(θA∣x)\Lambda = \frac{L(\theta_0 | \mathbf{x})}{L(\theta_A | \mathbf{x})}, where θ0\theta_0 and θA\theta_A are the fixed parameters under H0H_0 and HAH_A, respectively, is model-dependent and derived directly from the assumed data-generating process. This distribution governs the exact critical values and p-values for the test, but its form varies with the parametric family; for instance, in models from the , it can involve ratios of sufficient statistics whose distributions are known under the null (e.g., gamma or chi-squared for scale parameters in exponential distributions). Exact tests are particularly feasible in discrete models with small sample sizes, such as the binomial distribution for testing simple hypotheses about a success probability pp. Here, Λ\Lambda simplifies to a monotone function of the number of successes kk, and the null distribution is binomial with parameters nn (trials) and p=p0p = p_0, enabling precise computation of rejection regions by enumerating probabilities of outcomes yielding Λ≤c\Lambda \leq c for a chosen cc that controls the type I error at α\alpha. To obtain p-values, one calculates P(Λ≤Λobs∣H0)P(\Lambda \leq \Lambda_{\text{obs}} \mid H_0), where Λobs\Lambda_{\text{obs}} is the observed value; in discrete cases like the binomial, this is the sum of null probabilities over the critical region, often using precomputed tables for very small nn. For continuous or higher-dimensional models where analytical summation or integration is intractable, simulation approximates the by generating many samples from the null model, computing Λ\Lambda for each, and estimating the tail probability empirically. However, deriving the exact distribution remains challenging and highly specific to the model, often rendering it impractical for composite hypotheses, large samples, or non-standard families, which motivates reliance on asymptotic approximations for broader applicability.

Asymptotic Distribution

Under the H0H_0 and appropriate regularity conditions, states that the likelihood ratio test statistic −2log⁡λn-2 \log \lambda_n, where λn\lambda_n is the likelihood ratio, converges in distribution to a with dd as the sample size n→∞n \to \infty, with d=dim⁡(ΘA)−dim⁡(Θ0)d = \dim(\Theta_A) - \dim(\Theta_0) representing the difference in the dimensions of the parameter spaces under the alternative hypothesis ΘA\Theta_A and the Θ0\Theta_0. This asymptotic facilitates the computation of p-values and critical values for the test in large samples. The requires several regularity conditions to hold, including that the true value lies in the interior of the parameter space Θ0\Theta_0, ensuring of the parameters under H0H_0; that the log-likelihood function is twice continuously differentiable with respect to the parameters in an open neighborhood of the true value; that the of the outer product of the score function (or equivalently, the matrix) is positive definite; and that the support of the distribution does not depend on the parameters. These conditions guarantee the consistency and asymptotic normality of the maximum likelihood estimators under both hypotheses. A sketch of the proof begins with a second-order Taylor expansion of the log-likelihood around the maximum likelihood θ^0\hat{\theta}_0 under H0H_0, evaluated at the unrestricted θ^A\hat{\theta}_A, yielding −2log⁡λn≈(θ^A−θ^0)TI(θ^0)(θ^A−θ^0)-2 \log \lambda_n \approx (\hat{\theta}_A - \hat{\theta}_0)^T I(\hat{\theta}_0) (\hat{\theta}_A - \hat{\theta}_0), where I(⋅)I(\cdot) is the observed information matrix. Asymptotically, under H0H_0, θ^A−θ^0\hat{\theta}_A - \hat{\theta}_0 behaves like a normal random vector with covariance involving the inverse Fisher information restricted to the tangent space orthogonal to Θ0\Theta_0, resulting in the quadratic form converging to χd2\chi^2_d. For small samples or when regularity conditions are mildly violated, the chi-squared approximation can exhibit bias, leading to inflated type I error rates. Bartlett corrections address this by scaling the test statistic by a factor cc (typically c≈1−d(d+2)6nc \approx 1 - \frac{d(d+2)}{6n}) derived from higher-order expansions of the log-likelihood, improving the agreement with the . Alternatively, bootstrap methods, such as parametric or nonparametric resampling of the data under H0H_0, provide empirical approximations to the of −2log⁡λn-2 \log \lambda_n, enhancing accuracy without relying on asymptotic .

Examples and Applications

Binomial Proportion Test

The likelihood ratio test is commonly applied to assess hypotheses concerning a single binomial proportion pp, the probability of success in independent Bernoulli trials. Consider testing the null hypothesis H0:p=p0H_0: p = p_0 against the two-sided alternative HA:p≠p0H_A: p \neq p_0, based on nn trials yielding kk observed successes. The likelihood function under this model is given by L(p)=(nk)pk(1−p)n−k.L(p) = \binom{n}{k} p^k (1-p)^{n-k}. Under H0H_0, the maximized likelihood is L(p0)L(p_0), while under the full model (including HAH_A), the maximum likelihood estimator (MLE) is p^=k/n\hat{p} = k/n, yielding L(p^)L(\hat{p}). The likelihood ratio test statistic is then λ=2[log⁡L(p^)−log⁡L(p0)]=2[klog⁡(p^p0)+(n−k)log⁡(1−p^1−p0)],\lambda = 2 \left[ \log L(\hat{p}) - \log L(p_0) \right] = 2 \left[ k \log \left( \frac{\hat{p}}{p_0} \right) + (n - k) \log \left( \frac{1 - \hat{p}}{1 - p_0} \right) \right], where the binomial coefficient cancels out in the difference. To interpret the test, λ\lambda is compared to a critical value from the χ12\chi^2_1 distribution (referenced asymptotically for decision-making). For a numerical illustration, suppose n=100n = 100, k=45k = 45, and p0=0.5p_0 = 0.5, so p^=0.45\hat{p} = 0.45. Substituting into the formula gives λ≈1.00\lambda \approx 1.00. The corresponding p-value is approximately 0.32, indicating insufficient evidence to reject H0H_0 at common significance levels like α=0.05\alpha = 0.05. This step-by-step computation highlights the LRT's practicality for binomial data. A key advantage of the LRT in this binomial setting is its exact equivalence to Pearson's chi-squared goodness-of-fit test when framed as comparing observed frequencies (k successes, n-k failures) to expected frequencies under H0H_0 (np0n p_0, n(1−p0)n (1-p_0)). Furthermore, it aligns precisely with the statistic here, as all three tests reduce to similar forms under the simple for the proportion.

Linear Regression Model

In the model, the likelihood-ratio test (LRT) is applied to compare a full model, specified as Y=Xβ+ϵ\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} where ϵ∼N(0,σ2I)\boldsymbol{\epsilon} \sim N(\mathbf{0}, \sigma^2 \mathbf{I}), against a reduced model under the H0:Y=X0β0+ϵH_0: \mathbf{Y} = \mathbf{X}_0 \boldsymbol{\beta}_0 + \boldsymbol{\epsilon} with the same error assumption, where X0\mathbf{X}_0 is a subset of columns from X\mathbf{X} corresponding to the parameters restricted under H0H_0. This setup tests whether the additional predictors in the full model significantly improve the fit, such as when evaluating the inclusion of one or more regressors. The test statistic is computed as λ=nlog⁡(RSSreducedRSSfull)\lambda = n \log\left(\frac{\text{RSS}_\text{reduced}}{\text{RSS}_\text{full}}\right), where nn is the number of observations and RSS denotes the residual sum of squares for each model; under the assumption of normally distributed errors, this LRT is equivalent to the F-test for nested models. For instance, consider a dataset with n=20n=20 observations where a simple linear regression model (one predictor) is compared to a multiple regression model adding a second predictor to test the significance of that coefficient; the resulting λ≈5.4\lambda \approx 5.4 is then compared to a χ12\chi^2_1 distribution for inference at a 5% significance level, where the p-value indicates whether the added predictor is warranted. The LRT in connects directly to analysis of variance (ANOVA) frameworks, as the comparison of RSS between nested models mirrors the partitioning of variance in ANOVA tables for testing model adequacy. Regarding , the LRT remains applicable even when predictors in the full model are correlated, though high multicollinearity can inflate the variance of estimates and reduce the test's power to detect true differences between models. While the LRT leverages the full likelihood for generality and can extend beyond strict normality assumptions through asymptotic approximations, its reliability for non-normal errors depends on large-sample behavior. Under normality, the test relates exactly to the for finite samples.

Comparison with Other Tests

The likelihood-ratio test (LRT) is one of three classical large-sample tests for composite hypotheses, alongside the and the (also known as the Lagrange multiplier test). The is based on the maximum likelihood estimator (MLE) under the (HA) and its estimated , effectively testing whether the MLE is sufficiently far from the (H0) values. In contrast, the LRT compares the likelihoods under both H0 and HA by evaluating the ratio of the maximized likelihood under H0 to that under HA, utilizing information from both restricted and unrestricted models. The , meanwhile, relies solely on the score function (the of the log-likelihood) and observed information matrix evaluated under H0, making it computationally advantageous when the null model is simpler to fit than the alternative. Asymptotically, under H0, all three tests are equivalent and follow a with equal to the difference in the number of parameters between HA and H0, as established by for the LRT and corresponding results for the others. However, in finite samples, differences emerge: the can exhibit poorer performance, such as inflated Type I error rates or reduced power, particularly when the MLE under HA is near the boundary or under model misspecification, whereas the LRT tends to be more robust due to its balanced use of both models' information. Additionally, while the LRT and are invariant to reparameterization of the model, the is not, meaning its results can change depending on how parameters are scaled or transformed. The choice among these tests often depends on computational feasibility and model structure. The LRT is particularly preferred for nested models, where HA encompasses H0, as it directly measures the improvement in fit via the likelihood ratio and is the natural extension of exact tests for simple hypotheses. The score test is ideal when evaluating perturbations from H0 without needing to estimate the full HA model, such as in preliminary screening, while the Wald test suits scenarios where the unrestricted MLE is already available. In generalized linear models (GLMs), the LRT is operationalized through the deviance statistic, defined as twice the difference in log-likelihoods between the full and reduced models, which serves as the standard measure for model comparison and testing nested hypotheses.

Generalized Likelihood Ratio Test

The generalized likelihood ratio test (GLRT) is the extension of the classical likelihood ratio test to composite hypotheses within nested parametric models, where the restricts the parameter space of the alternative. Further extensions and modifications of the GLRT address more complex scenarios, such as non-nested hypotheses, high-dimensional settings, and situations where asymptotic approximations are unreliable. For instance, when comparing non-nested models, Vuong's test (1989) provides a framework by approximating the Kullback-Leibler divergence between models using a derived from normalized log-likelihood differences, enabling selection between competing specifications without assuming nesting. In high-dimensional contexts with numerous nuisance parameters, the profile likelihood approach adjusts the GLRT by maximizing the likelihood over nuisance parameters under each , often incorporating penalty terms to mitigate and improve finite-sample performance. This adjusted LRT, sometimes combined with regularization like in sparse settings, ensures the test remains valid even when the number of parameters approaches or exceeds the sample size. For cases where asymptotic chi-squared distributions fail, such as small sample sizes or boundary constraints on parameters, the bootstrap GLRT resamples the to empirically estimate p-values, offering a distribution-free alternative that enhances accuracy in boundary problems or non-regular cases. Applications of the GLRT in frequently involve for , where it evaluates the number of components by testing nested or generalized structures, serving as a foundation for information criteria like AIC and BIC that approximate the test for computational efficiency. In fitting, the GLRT helps detect the presence of multiple subpopulations, though it requires adjustments for issues.

References

  1. https://fiveable.me/theoretical-statistics/unit-8/likelihood-ratio-tests/study-guide/NaudFVNsZmoACSUa
  2. https://bookdown.org/egarpor/inference/hyptest-lik.html
  3. http://www.stats.org.uk/statistical-inference/NeymanPearson1933.pdf
  4. https://stat210a.berkeley.edu/fall-2024/reader/hypothesis-testing.html
  5. https://www.stat.rice.edu/~tsn4867/all_files/IMS_VOLUME/LEHMANN/Lehmann_hist_opt.pdf
  6. https://pages.stat.wisc.edu/~shao/stat610/stat610-09.pdf
  7. https://www.stat.berkeley.edu/~bartlett/courses/2013spring-stat210b/notes/26notes.pdf
  8. https://online.stat.psu.edu/stat415/book/export/html/847
  9. https://www.statlect.com/fundamentals-of-statistics/likelihood-ratio-test
  10. https://www.itl.nist.gov/div898/handbook/apr/section2/apr233.htm
  11. https://www.probabilitycourse.com/chapter8/8_4_5_likelihood_ratio_tests.php
  12. https://online.stat.psu.edu/stat415/lesson/26/26.1
  13. https://files.eric.ed.gov/fulltext/EJ1267545.pdf
  14. https://royalsocietypublishing.org/doi/10.1098/rsta.1922.0009
  15. https://royalsocietypublishing.org/doi/10.1098/rsta.1933.0009
  16. https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-9/issue-1/The-Large-Sample-Distribution-of-the-Likelihood-Ratio-for/10.1214/aoms/1177733277.full
  17. https://www.jstor.org/stable/2344614
  18. https://www.jstor.org/stable/48554500
  19. https://www.stat.umn.edu/geyer/5421/notes/likelihood.html
  20. https://www.stat.purdue.edu/~zhanghao/MAS/handout/Likelihood.pdf
  21. https://www2.stat.duke.edu/~sayan/SAMSI/lec/411notes03.pdf
  22. https://sites.stat.washington.edu/jaw/COURSES/580s/581/LECTNOTES/ch4-rev1.pdf
  23. https://www.scholars.northwestern.edu/en/publications/statistical-comparison-of-non-nested-probabilistic-discrete-choic/
  24. https://pages.stat.wisc.edu/~shao/stat610/Casella_Berger_Statistical_Inference.pdf
  25. https://pmc.ncbi.nlm.nih.gov/articles/PMC3693095/
  26. https://stats.libretexts.org/Bookshelves/Probability_Theory/Probability_Mathematical_Statistics_and_Stochastic_Processes_(Siegrist)/09%3A_Hypothesis_Testing/9.05%3A_Likelihood_Ratio_Tests
  27. https://faculty.washington.edu/lober/www.isdsjournal.org/htdocs/articles/269.pdf
  28. https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-9/issue-1/The-Large-Sample-Distribution-of-the-Likelihood-Ratio-for-Testing/10.1214/aoms/1177732360.full
  29. https://math.umd.edu/~slud/s701.S14/WilksThm.pdf
  30. https://www.tandfonline.com/doi/abs/10.1081/STA-100104749
  31. https://www.researchgate.net/publication/265716792_AnIntroduction_to_Bartlett_Correction_and_Bias_Reduction
  32. https://www.jstor.org/stable/2347790
  33. https://www.stat.umn.edu/geyer/5421/notes/binom.html
  34. https://blogs.sas.com/content/iml/2024/03/27/likelihood-ratio-test.html
  35. https://bookdown.org/mike/data_analysis/nested-model-tests.html
  36. https://online.stat.psu.edu/stat857/node/45/
  37. https://sites.warnercnr.colostate.edu/gwhite/wp-content/uploads/sites/73/2017/04/LikelihoodRatioTests.pdf
  38. https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faqhow-are-the-likelihood-ratio-wald-and-lagrange-multiplier-score-tests-different-andor-similar/
  39. https://www.cs.ubc.ca/~arnaud/stat461/lecture_stat461_WaldRaoLRtests_handouts_2008.pdf
  40. https://dept.stat.lsa.umich.edu/~moulib/powerjonmay1.pdf
  41. https://www.sfu.ca/sasdoc/sashtml/ets/chap14/sect44.htm
  42. https://online.stat.psu.edu/stat504/lesson/6/6.3/6.3.4
  43. https://engineering.purdue.edu/~mrb/resources/LecturesF/Session_18.pdf
  44. https://www.econometricsociety.org/publications/econometrica/1989/03/01/likelihood-ratio-tests-model-selection-and-non-nested
Add your contribution
Related Hubs
User Avatar
No comments yet.