Linear model

Linear modelMain

Community hub

Linear model

7 pages, 0 posts

0 subscribers

Recent from talks

Be the first to start a discussion here.

Recent from talks

Be the first to start a discussion here.

Contribute something

About hubMembersContent overviewUpdatesRules

Main reference articles

Linear model

View on Wikipedia

from Wikipedia

In statistics, the term linear model refers to any model which assumes linearity in the system. The most common occurrence is in connection with regression models and the term is often taken as synonymous with linear regression model. However, the term is also used in time series analysis with a different meaning. In each case, the designation "linear" is used to identify a subclass of models for which substantial reduction in the complexity of the related statistical theory is possible.

Linear regression models

[edit]

For the regression case, the statistical model is as follows. Given a (random) sample $(Y_{i},X_{i1},\ldots ,X_{ip}),\,i=1,\ldots ,n$ the relation between the observations $Y_{i}$ and the independent variables $X_{ij}$ is formulated as

Y_{i}=\beta _{0}+\beta _{1}\phi _{1}(X_{i1})+\cdots +\beta _{p}\phi _{p}(X_{ip})+\varepsilon _{i}\qquad i=1,\ldots ,n

where $\phi _{1},\ldots ,\phi _{p}$ may be nonlinear functions. In the above, the quantities $\varepsilon _{i}$ are random variables representing errors in the relationship. The "linear" part of the designation relates to the appearance of the regression coefficients, $\beta _{j}$ in a linear way in the above relationship. Alternatively, one may say that the predicted values corresponding to the above model, namely

{\hat {Y}}_{i}=\beta _{0}+\beta _{1}\phi _{1}(X_{i1})+\cdots +\beta _{p}\phi _{p}(X_{ip})\qquad (i=1,\ldots ,n),

are linear functions of the $\beta _{j}$ .

Given that estimation is undertaken on the basis of a least squares analysis, estimates of the unknown parameters $\beta _{j}$ are determined by minimising a sum of squares function

S=\sum _{i=1}^{n}\varepsilon _{i}^{2}=\sum _{i=1}^{n}\left(Y_{i}-\beta _{0}-\beta _{1}\phi _{1}(X_{i1})-\cdots -\beta _{p}\phi _{p}(X_{ip})\right)^{2}.

From this, it can readily be seen that the "linear" aspect of the model means the following:

the function to be minimised is a quadratic function of the $\beta _{j}$ for which minimisation is a relatively simple problem;
the derivatives of the function are linear functions of the $\beta _{j}$ making it easy to find the minimising values;
the minimising values $\beta _{j}$ are linear functions of the observations $Y_{i}$ ;
the minimising values $\beta _{j}$ are linear functions of the random errors $\varepsilon _{i}$ which makes it relatively easy to determine the statistical properties of the estimated values of $\beta _{j}$ .

Time series models

[edit]

An example of a linear time series model is an autoregressive moving average model. Here the model for values { $X_{t}$ } in a time series can be written in the form

X_{t}=c+\varepsilon _{t}+\sum _{i=1}^{p}\phi _{i}X_{t-i}+\sum _{i=1}^{q}\theta _{i}\varepsilon _{t-i}.\,

where again the quantities $\varepsilon _{i}$ are random variables representing innovations which are new random effects that appear at a certain time but also affect values of $X$ at later times. In this instance the use of the term "linear model" refers to the structure of the above relationship in representing $X_{t}$ as a linear function of past values of the same time series and of current and past values of the innovations.^[1] This particular aspect of the structure means that it is relatively simple to derive relations for the mean and covariance properties of the time series. Note that here the "linear" part of the term "linear model" is not referring to the coefficients $\phi _{i}$ and $\theta _{i}$ , as it would be in the case of a regression model, which looks structurally similar.

Other uses in statistics

[edit]

There are some other instances where "nonlinear model" is used to contrast with a linearly structured model, although the term "linear model" is not usually applied. One example of this is nonlinear dimensionality reduction.

References

[edit]

^ Priestley, M.B. (1988) Non-linear and Non-stationary time series analysis, Academic Press. ISBN 0-12-564911-8

Statistics

Descriptive statistics

Continuous data

Center	Mean Arithmetic Arithmetic-Geometric Contraharmonic Cubic Generalized/power Geometric Harmonic Heronian Heinz Lehmer Median Mode
Dispersion	Average absolute deviation Coefficient of variation Interquartile range Percentile Range Standard deviation Variance
Shape	Central limit theorem Moments Kurtosis L-moments Skewness

Count data

Index of dispersion

Summary tables

Dependence

Graphics

Data collection

Study design	Effect size Missing data Optimal design Population Replication Sample size determination Statistic Statistical power
Survey methodology	Sampling Cluster Stratified Opinion poll Questionnaire Standard error
Controlled experiments	Blocking Factorial experiment Interaction Random assignment Randomized controlled trial Randomized experiment Scientific control
Adaptive designs	Adaptive clinical trial Stochastic approximation Up-and-down designs
Observational studies	Cohort study Cross-sectional study Natural experiment Quasi-experiment

Statistical inference

Statistical theory

Frequentist inference

Point estimation	Estimating equations Maximum likelihood Method of moments M-estimator Minimum distance Unbiased estimators Mean-unbiased minimum-variance Rao–Blackwellization Lehmann–Scheffé theorem Median unbiased Plug-in
Interval estimation	Confidence interval Pivot Likelihood interval Prediction interval Tolerance interval Resampling Bootstrap Jackknife
Testing hypotheses	1- & 2-tails Power Uniformly most powerful test Permutation test Randomization test Multiple comparisons
Parametric tests	Likelihood-ratio Score/Lagrange multiplier Wald

Specific tests

Z-test (normal) Student's t-test F-test
Goodness of fit	Chi-squared G-test Kolmogorov–Smirnov Anderson–Darling Lilliefors Jarque–Bera Normality (Shapiro–Wilk) Likelihood-ratio test Model selection Cross validation AIC BIC
Rank statistics	Sign Sample median Signed rank (Wilcoxon) Hodges–Lehmann estimator Rank sum (Mann–Whitney) Nonparametric anova 1-way (Kruskal–Wallis) 2-way (Friedman) Ordered alternative (Jonckheere–Terpstra) Van der Waerden test

Bayesian inference

Correlation	Pearson product-moment Partial correlation Confounding variable Coefficient of determination
Regression analysis (see also Template:Least squares and regression analysis	Errors and residuals Regression validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines (MARS)
Linear regression	Simple linear regression Ordinary least squares General linear model Bayesian regression
Non-standard predictors	Nonlinear regression Nonparametric Semiparametric Isotonic Robust Homoscedasticity and Heteroscedasticity
Generalized linear model	Exponential families Logistic (Bernoulli) / Binomial / Poisson regressions
Partition of variance	Analysis of variance (ANOVA, anova) Analysis of covariance Multivariate ANOVA Degrees of freedom

Categorical / multivariate / time-series / survival analysis

Categorical

Multivariate

Time-series

General	Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality
Specific tests	Dickey–Fuller Johansen Q-statistic (Ljung–Box) Durbin–Watson Breusch–Godfrey
Time domain	Autocorrelation (ACF) partial (PACF) Cross-correlation (XCF) ARMA model ARIMA model (Box–Jenkins) Autoregressive conditional heteroskedasticity (ARCH) Vector autoregression (VAR) (Autoregressive model (AR))
Frequency domain	Spectral density estimation Fourier analysis Least-squares spectral analysis Wavelet Whittle likelihood

Survival

Survival function	Kaplan–Meier estimator (product limit) Proportional hazards models Accelerated failure time (AFT) model First hitting time
Hazard function	Nelson–Aalen estimator
Test	Log-rank test

Applications

Biostatistics	Bioinformatics Clinical trials / studies Epidemiology Medical statistics
Engineering statistics	Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification
Social statistics	Actuarial science Census Crime statistics Demography Econometrics Jurimetrics National accounts Official statistics Population statistics Psychometrics
Spatial statistics	Cartography Environmental statistics Geographic information system Geostatistics Kriging

Revisions and contributors Edit on Wikipedia Read on Wikipedia

View on Grokipedia

from Grokipedia

A linear model in statistics is a framework for modeling the relationship between a response variable and one or more predictor variables, assuming that the expected value of the response is a linear function of the predictors, typically expressed in matrix form as

Y = X\beta + \epsilon

, where

Y

is the

n \times 1

vector of observed responses,

X

is the

n \times p

design matrix incorporating the predictors,

\beta

is the

p \times 1

vector of unknown parameters, and

\epsilon

is the

n \times 1

vector of random errors with mean zero.^[1]^[2] The origins of linear models trace back to the late 19th century, when Sir Francis Galton developed the concept of regression while studying hereditary traits in sweet peas, introducing the idea of a linear relationship tending toward the mean, which Karl Pearson later formalized in the early 20th century through the development of the product-moment correlation and multiple regression techniques.^[3]^[4] Linear models encompass a wide range of applications, including simple and multiple linear regression for prediction and inference, analysis of variance (ANOVA) for comparing group means, and analysis of covariance (ANCOVA) for adjusting means across covariates.^[2]^[1] Under the Gauss-Markov assumptions—where errors have zero mean, constant variance

\sigma^2

, and are uncorrelated—the ordinary least squares estimator of

\beta

is the best linear unbiased estimator (BLUE), providing efficient parameter estimates via the solution to the normal equations

X'X\hat{\beta} = X'Y

.^[2] Extensions include generalized least squares for heteroscedastic or correlated errors, as in the Aitken model where

\text{cov}(\epsilon) = \sigma^2 V

with known

V

, and further generalizations to linear mixed models incorporating random effects for clustered or hierarchical data.^[2]^[5] These models are foundational in fields like economics, biology, and social sciences, enabling hypothesis testing via F-statistics and confidence intervals under normality assumptions.^[1]^[2]

Basic Concepts

Definition and Scope

In statistics, a linear model describes the relationship between a dependent variable and one or more independent variables as a linear function of the parameters, meaning the expected value of the dependent variable is a linear combination of the independent variables weighted by unknown coefficients.^[2] This linearity pertains specifically to the parameters rather than the variables themselves, allowing transformations of the variables (such as logarithms or polynomials) to maintain the linear structure in the coefficients.^[6] A canonical example is the simple linear regression model, where the dependent variable

Y

is modeled as

Y = \beta_0 + \beta_1 X + \epsilon

, with

\beta_0

and

\beta_1

as the intercept and slope parameters,

X

as the independent variable, and

\epsilon

as a random error term representing unexplained variation.^[7] This formulation emphasizes the additivity of effects, where the influence of each independent variable contributes independently to the outcome, aligning with the superposition principle that permits combining solutions through scaling and addition.^[8] The scope of linear models encompasses a broad range of applications in statistics, including regression analysis, analysis of variance, and experimental design, primarily for predicting outcomes and drawing inferences about relationships in fields such as economics, social sciences, and engineering.^[9] Unlike nonlinear models, where conditional and marginal effects may diverge, linear models ensure these effects coincide, simplifying interpretation and enabling properties like homogeneity and additivity that support scalable solutions.^[10] A key advantage is that linearity in parameters facilitates closed-form solutions for parameter estimation, making them computationally efficient and analytically tractable compared to nonlinear alternatives requiring iterative methods.^[11]

Historical Background

The development of linear models traces its roots to the early 19th century, when astronomers and mathematicians sought methods to fit observational data amid measurement errors. In 1805, Adrien-Marie Legendre introduced the method of least squares in his work Nouvelles méthodes pour la détermination des orbites des comètes, applying it to minimize the sum of squared residuals for predicting comet orbits based on astronomical observations. This deterministic approach marked a foundational step in handling overdetermined systems. Four years later, in 1809, Carl Friedrich Gauss published Theoria motus corporum coelestium in sectionibus conicis solem ambientum, where he claimed prior use of least squares since 1795 and provided a probabilistic justification by linking it to the normal distribution of errors, establishing it as a maximum-likelihood estimator under Gaussian assumptions. The concept of regression emerged in the late 19th century through studies of inheritance patterns. In 1886, Francis Galton coined the term "regression" in his paper "Regression Towards Mediocrity in Hereditary Stature," published in the Journal of the Anthropological Institute, while analyzing height data from parents and children.^[12] Galton observed that extreme parental heights tended to produce offspring heights closer to the population average, introducing the idea of linear relationships between variables and laying the groundwork for bivariate regression as a tool in biometrics. This work shifted focus from mere curve fitting to modeling dependencies, influencing subsequent statistical applications in natural sciences.^[13] Building on Galton's ideas, Karl Pearson formalized key aspects of linear regression in the late 19th and early 20th centuries. In 1895, Pearson developed the product-moment correlation coefficient to quantify the strength of linear relationships between variables. He further extended this to multiple regression techniques around 1900–1910, enabling the modeling of a dependent variable against several predictors, which provided a mathematical foundation for broader applications in biometrics and beyond.^[14] In the 1920s, Ronald A. Fisher advanced linear models into a unified framework for experimental design and analysis. Working at the Rothamsted Experimental Station, Fisher developed analysis of variance (ANOVA) in his 1925 book Statistical Methods for Research Workers, extending least squares to partition variance in designed experiments, such as agricultural trials. By the early 1930s, in works like The Design of Experiments (1935), Fisher synthesized regression, ANOVA, and covariance analysis into the general linear model, incorporating probabilistic error terms to enable inference from sample data.^[15] This evolution transformed linear models from deterministic tools to probabilistic frameworks essential for hypothesis testing in experimental sciences. The 1930s saw Jerzy Neyman and Egon Pearson formalize inference procedures for linear models through their Neyman-Pearson lemma, introduced in the 1933 paper "On the Problem of the Most Efficient Tests of Statistical Hypotheses" in Philosophical Transactions of the Royal Society.^[16] Their framework emphasized controlling error rates (Type I and Type II) and power in hypothesis testing, providing a rigorous basis for applying linear models to decision-making under uncertainty. Post-World War II computational advances, including early electronic computers like the ENIAC (1945) and statistical software developments in the 1950s, facilitated the widespread adoption of these methods by enabling efficient matrix computations for large datasets.^[17] This period marked the transition of linear models from theoretical constructs to practical tools in fields like economics and social sciences.

Mathematical Formulation

General Linear Model Equation

The general linear model expresses the relationship between a response variable and one or more predictor variables as a linear combination of parameters plus an error term. In its scalar form, for each observation

i = 1, \dots, n

, the model is given by

Y_i = \beta_0 + \beta_1 X_{i1} + \cdots + \beta_{p-1} X_{i,p-1} + \varepsilon_i,

where

Y_i

is the observed response,

\beta_0

is the intercept,

\beta_j

(for

j = 1, \dots, p-1

) are the slope coefficients associated with the predictors

X_{ij}

, and

\varepsilon_i

represents the random error for the

i

th observation.^[18]^[19] This formulation can be compactly represented in vector-matrix notation as

\mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon},

where

\mathbf{Y}

is an

n \times 1

vector of responses,

\mathbf{X}

is an

n \times p

design matrix with rows corresponding to observations,

\boldsymbol{\beta}

is a

p \times 1

vector of parameters (including the intercept), and

\boldsymbol{\varepsilon}

is an

n \times 1

vector of errors.^[19]^[20] The errors

\varepsilon_i

are typically assumed to be independently and identically distributed as normal with mean zero and constant variance

\sigma^2

, though full details on these assumptions appear in the relevant section.^[19] In this model, each coefficient

\beta_j

(for

j \geq 1

) represents the partial effect of the

j

th predictor on the response, interpreted as the expected change in

Y

for a one-unit increase in

X_j

, holding all other predictors constant.^[21]^[22] The design matrix

\mathbf{X}

structures the predictors for estimation; its first column consists entirely of 1s to accommodate the intercept term

\beta_0

.^[23] Categorical predictors are incorporated by creating dummy variables, where each category (except one reference category) is represented by a binary column in

\mathbf{X}

to avoid multicollinearity.^[24]

Matrix Representation

The design matrix

X

serves as the foundational structure in the matrix representation of linear models, typically an

n \times p

matrix where

n

denotes the number of observations and

p

the number of parameters (including the intercept). Its rows represent individual observations, while columns correspond to the predictor variables. For the model to be identifiable and to ensure unique parameter estimates,

X

must have full column rank, meaning its rank equals

p

, which implies that the columns are linearly independent and there is no perfect multicollinearity. This full column rank condition guarantees the invertibility of the matrix

X^\top X

, a key property that enables the explicit solution for model parameters. The parameter vector

\beta

, a

p \times 1

column vector containing the coefficients, is estimated in matrix form via the ordinary least squares (OLS) estimator

\hat{\beta} = (X^\top X)^{-1} X^\top Y

, where

Y

is the

n \times 1

response vector; this form previews the efficient algebraic solution without requiring iterative methods, with full details on its derivation provided in the section on ordinary least squares estimation.^[25] A central element in this representation is the projection matrix

H = X (X^\top X)^{-1} X^\top

, often termed the hat matrix, which orthogonally projects the response vector

Y

onto the column space of

X

to yield the fitted values

\hat{Y} = H Y

. This matrix

H

is symmetric (

H^\top = H

) and idempotent (

H^2 = H

), properties that reflect its role as an orthogonal projection operator and facilitate analytical manipulations such as variance computations. Complementing

H

is the residual maker matrix

M = I_n - H

, where

I_n

is the

n \times n

identity matrix, which produces the residuals

e = M Y

by projecting

Y

onto the orthogonal complement of the column space of

X

. The matrix

M

annihilates

X

such that

M X = 0

, ensuring residuals are uncorrelated with the predictors in the column space, and it is also symmetric (

M^\top = M

) and idempotent (

M^2 = M

). These properties underscore

M

's utility in isolating deviations unexplained by the model.^[25] The matrix formulation offers substantial computational advantages, particularly for large datasets, as it leverages optimized linear algebra algorithms in statistical software to perform operations like matrix inversion and multiplication efficiently, scaling to high-dimensional problems where scalar-based approaches would be prohibitive.^[26]

Assumptions and Diagnostics

Core Assumptions

The validity of the linear model, particularly in the context of ordinary least squares (OLS) estimation and statistical inference, hinges on several foundational assumptions that ensure the model's parameters are unbiased, consistent, and efficient. These assumptions pertain to the relationship between the response variable

Y

and predictors

X

, as well as the properties of the error terms

\epsilon

. Violations can lead to biased estimates or invalid inference, though some robustness holds under large samples via the central limit theorem (CLT). The core assumptions are linearity, independence, homoscedasticity, normality, no perfect multicollinearity, and exogeneity.^[27]^[28]^[29] Linearity requires that the conditional expectation of the response variable is a linear function of the predictors, expressed as

E(Y \mid X) = X\beta

, where

\beta

is the vector of parameters. This assumption implies that the effects of the predictors on the mean response are additive and that the relationship is straight-line in the parameters, holding the other variables fixed. It does not necessitate a linear relationship in the raw data but rather in the population model; nonlinearities can be addressed through transformations or additional terms, but the core model form must satisfy this for OLS to yield unbiased estimates.^[27]^[28]^[29] Independence assumes that the error terms

\epsilon_i

for different observations are statistically independent, meaning no correlation between residuals across observations, such as in time series where autocorrelation might occur. This ensures that the variance-covariance matrix of the errors is diagonal, supporting the unbiasedness of OLS estimators under the Gauss-Markov theorem. Independence is crucial for the standard errors and hypothesis tests to be valid, as dependence can inflate or deflate them.^[27]^[28] Homoscedasticity stipulates that the variance of the errors is constant across all levels of the predictors, i.e.,

\text{Var}(\epsilon_i \mid X) = \sigma^2

for some constant

\sigma^2

. This equal spread of residuals prevents heteroscedasticity, where variance changes with

X

, which could lead to inefficient OLS estimates and unreliable standard errors. Under this assumption, combined with exogeneity, OLS achieves the best linear unbiased estimator (BLUE) property.^[27]^[28]^[29] Normality posits that the errors follow a normal distribution,

\epsilon_i \sim N(0, \sigma^2)

, which is necessary for exact finite-sample inference, such as t-tests and F-tests on coefficients. However, this assumption is not required for consistency or unbiasedness of OLS; for large samples, the CLT ensures asymptotic normality of the estimators, making inference approximately valid even without it. Normality primarily affects the distribution of test statistics in small samples.^[27]^[28] No perfect multicollinearity requires that the predictors are not linearly dependent, meaning the design matrix

X

has full column rank, so no predictor is an exact linear combination of others. This ensures the parameter matrix

(\mathbf{X}^T\mathbf{X})^{-1}

exists and is unique, preventing infinite or undefined OLS estimates. While mild multicollinearity is tolerable, perfect collinearity renders coefficients non-identifiable.^[29] Exogeneity, or the zero conditional mean assumption, states that the errors are uncorrelated with the predictors,

E(\epsilon_i \mid X) = 0

, implying no omitted variables or endogeneity biasing the estimates. This strict exogeneity ensures OLS estimators are unbiased and consistent, as any correlation would violate the orthogonality condition essential for projection-based estimation.^[29]

Violation Detection and Remedies

Detecting violations of the linear model's assumptions is essential for ensuring the validity of inferences drawn from the analysis. Post-estimation diagnostics primarily focus on the residuals, which represent the differences between observed and predicted values. These tools help identify departures from linearity, independence, homoscedasticity, and normality, allowing practitioners to assess model adequacy before proceeding to remedies.^[30] To check for linearity, residual plots of residuals against fitted values are commonly used; a random scatter around zero indicates the assumption holds, while patterns such as curves or funnels suggest nonlinearity. For independence, particularly in time series contexts, the Durbin-Watson test detects first-order autocorrelation by computing a statistic that compares adjacent residuals; values near 2 indicate no autocorrelation, while deviations toward 0 or 4 signal positive or negative autocorrelation, respectively.^[31] Heteroscedasticity is assessed via the Breusch-Pagan test, which regresses squared residuals on the predictors and tests the significance of the resulting coefficients under a chi-squared distribution; a significant result rejects constant variance.^[32] Normality of residuals is evaluated using quantile-quantile (Q-Q) plots, where points aligning closely with a straight line support the assumption, and deviations indicate skewness or heavy tails.^[33] Multicollinearity among predictors can inflate variance estimates and destabilize coefficients, even if other assumptions hold. The variance inflation factor (VIF) measures this by quantifying how much the variance of a coefficient is increased due to correlation with other predictors; for each predictor, VIF is computed as 1 over (1 - R²) from regressing it on the others, with values exceeding 10 signaling severe multicollinearity requiring attention.^[34] Outliers and influential points can disproportionately affect model fit and parameter estimates. Cook's distance identifies influential observations by measuring the change in fitted values when a single data point is removed; values greater than 4/n (where n is the sample size) or exceeding an F-threshold flag potential issues, combining leverage and residual magnitude.^[35] Leverage plots, based on hat values from the projection matrix, highlight high-leverage points that lie far from the mean of predictors.^[35] Once violations are detected, targeted remedies can restore assumption validity without discarding the linear framework. For heteroscedasticity, logarithmic or Box-Cox transformations stabilize variance by adjusting the scale of the response or predictors; the Box-Cox family, parameterized by λ, applies y^λ for λ ≠ 0 or log(y) for λ = 0, with maximum likelihood estimating the optimal λ to achieve homoscedasticity.^[36] Robust standard errors, such as those proposed by White, adjust inference by estimating the covariance matrix accounting for heteroscedasticity, providing consistent standard errors without altering coefficients.^[37] Autocorrelation, often in temporal data, can be addressed by including lagged dependent variables as predictors, effectively modeling the serial dependence and reducing residual correlation.^[38] For multicollinearity, ridge regression introduces a penalty term (λ times the sum of squared coefficients) to the least squares objective, shrinking estimates toward zero and stabilizing them in correlated predictor spaces; λ is tuned via cross-validation or ridge traces.^[34] Outliers may be handled by removing influential points identified via Cook's distance if they are verifiable errors, or by robust regression methods that downweight them, though sensitivity analyses are recommended to confirm robustness.^[35] Transformations like those in the Box-Cox framework can also mitigate multiple violations simultaneously by improving overall distributional properties.^[36]

Estimation and Inference

Ordinary Least Squares Estimation

The ordinary least squares (OLS) estimation method seeks to find the parameter vector

\hat{\beta}

that minimizes the sum of squared residuals, given by

\sum_{i=1}^n (y_i - \mathbf{x}_i' \hat{\beta})^2

, where

y_i

is the observed response and

\mathbf{x}_i' \hat{\beta}

is the predicted value.^[39] This objective function measures the total deviation between observed and fitted values, weighted equally by the square of each residual to penalize larger errors more heavily.^[40] To derive the OLS estimator, differentiate the sum of squared residuals with respect to

\beta

and set the result to zero, yielding the normal equations

X'X \beta = X'y

, where

X

is the design matrix and

y

is the response vector.^[41] Assuming

X'X

is invertible (which requires no perfect multicollinearity), the solution is

\hat{\beta} = (X'X)^{-1} X'y

.^[39] In matrix representation, as detailed in the Mathematical Formulation section, this projection of

y

onto the column space of

X

ensures the residuals are orthogonal to the predictors.^[40] Under the assumptions of linearity in parameters, strict exogeneity of regressors, and no perfect multicollinearity, the OLS estimator is unbiased, satisfying

E[\hat{\beta}] = \beta

.^[42] Furthermore, by the Gauss-Markov theorem, under the additional assumptions of homoscedasticity of errors and uncorrelated errors,

\hat{\beta}

is the best linear unbiased estimator (BLUE), possessing the minimum variance among all linear unbiased estimators of

\beta

. The variance-covariance matrix of the estimator is

\operatorname{Var}(\hat{\beta}) = \sigma^2 (X'X)^{-1}

, where

\sigma^2

is the error variance, highlighting how the precision of

\hat{\beta}

improves with more informative data in

X

.^[43] OLS estimation is widely implemented in statistical software for practical application. In R, the lm() function from the base stats package fits linear models using OLS by default, accepting a formula interface for specifying predictors and responses. In Python, the statsmodels library provides the OLS class in statsmodels.regression.linear_model, which computes

\hat{\beta}

and related statistics via methods like fit().

Hypothesis Testing and Confidence Intervals

In the general linear model, hypothesis testing provides a framework for assessing the statistical significance of the estimated parameters, typically using the ordinary least squares (OLS) estimator. For individual coefficients, the null hypothesis

H_0: \beta_j = 0

tests whether the j-th predictor has no linear association with the response variable, assuming the model is otherwise correctly specified. The test statistic is the t-statistic, given by

t = \frac{\hat{\beta}_j}{\text{SE}(\hat{\beta}_j)}

, where

\hat{\beta}_j

is the OLS estimate and

\text{SE}(\hat{\beta}_j)

is its standard error, derived from the variance-covariance matrix of the estimator under the model's assumptions. This standard error is

\text{SE}(\hat{\beta}_j) = \sqrt{ \hat{\sigma}^2 (X^T X)^{-1}_{jj} }

SE(β^j)=σ^2(XTX)jj−1

, with $\hat{\sigma}^2$ as the estimated error variance. The t-statistic follows a t-distribution with $n - p - 1$ degrees of freedom under the null, where n is the sample size and p the number of predictors; rejection of $H_0$ at a chosen significance level (e.g., 5%) indicates the coefficient is significantly different from zero.^[44]^[45] For assessing the overall model fit, the F-test evaluates the joint null hypothesis $H_0: \beta_j = 0$ for all j ≥ 1 (i.e., no predictors contribute), comparing the full model to an intercept-only model. The F-statistic is $F = \frac{\text{SSR} / \text{df}_\text{reg}}{\text{SSE} / \text{df}_\text{err}}$ , where SSR is the regression sum of squares, SSE the error sum of squares, $\text{df}_\text{reg} = p$ , and $\text{df}_\text{err} = n - p - 1$ . Under $H_0$ , F follows an F-distribution with (p, n - p - 1) degrees of freedom; a large F-value or small p-value rejects the null, supporting the inclusion of predictors. This test is equivalent to the ANOVA F-test in simple cases and extends to general linear hypotheses via matrix formulations.^[46]^[45] Confidence intervals offer a range of plausible values for parameters, constructed as $\hat{\beta}_j \pm t_{\alpha/2, \, n-p-1} \cdot \text{SE}(\hat{\beta}_j)$ , where $t_{\alpha/2}$ is the critical value from the t-distribution for a (1 - α) coverage level (e.g., 95%). These intervals quantify estimation uncertainty and contain the true $\beta_j$ with the stated probability in repeated sampling. For predictions, confidence intervals estimate the mean response at a new covariate value, while prediction intervals account for both mean uncertainty and individual observation variability, making them wider: the prediction interval is $\hat{y}_0 \pm t_{\alpha/2, n-p-1} \sqrt{ s^2 \left(1 + \mathbf{x}_0^T (X^T X)^{-1} \mathbf{x}_0 \right) }$

, where $s$ is the residual standard error, $\mathbf{x}_0$ is the new covariate vector including the intercept term. Prediction intervals are essential for forecasting single outcomes but expand with distance from the data centroid.^[47]^[48]^[49] The coefficient of determination, R-squared ( $R^2$ ), measures model fit as $R^2 = 1 - \frac{\text{SSE}}{\text{SST}}$ , where SST is the total sum of squares; it represents the proportion of total variance explained by the predictors, ranging from 0 to 1. However, $R^2$ increases with added predictors regardless of relevance, so the adjusted R-squared penalizes complexity: $R^2_\text{adj} = 1 - \frac{(1 - R^2)(n - 1)}{n - p - 1}$ . This adjusted metric favors parsimonious models and is preferred for model comparison. Values above 0.8 indicate strong fit in many applications, though interpretability depends on context.^[50]^[51] Power analysis and sample size planning ensure reliable inference, as statistical power—the probability of rejecting a false null hypothesis—depends on effect size, significance level, and sample size n. Larger n reduces standard errors, narrows intervals, and boosts power (e.g., to 80%), improving precision; for small effects (e.g., Cohen's f² = 0.02), n > 300 may be needed, while medium effects (f² = 0.15) require n ≈ 60-70. Tools like G*Power facilitate these calculations, emphasizing that inadequate n risks Type II errors (failing to detect true effects).^[52]^[53]

Classical Applications

Linear Regression Models

Linear regression serves as the foundational application of linear models, enabling the prediction of a continuous response variable based on one or more predictors assumed to have a linear relationship. In simple linear regression, the model posits a straight-line relationship between a single predictor and the response, formulated as

Y_i = \beta_0 + \beta_1 X_i + \epsilon_i

, where

Y_i

is the observed response for the

i

-th observation,

X_i

is the predictor value,

\beta_0

is the y-intercept,

\beta_1

is the slope coefficient, and

\epsilon_i

is the random error term assumed to be normally distributed with mean zero and constant variance.^[54] The slope

\beta_1

quantifies the expected change in

Y

for a one-unit increase in

X

, providing a direct interpretation in the units of the variables; for instance, if

Y

represents crop yield in bushels per acre and

X

is fertilizer amount in pounds per acre, a

\beta_1

of 2 indicates an additional 2 bushels per pound of fertilizer applied.^[55] This formulation traces its estimation roots to the method of least squares, first systematically presented by Adrien-Marie Legendre in 1805 for orbit determination and further justified probabilistically by Carl Friedrich Gauss in 1809.^[56]^[57] Multiple linear regression extends this framework to incorporate several predictors, expressed as

Y_i = \beta_0 + \sum_{j=1}^p \beta_j X_{ij} + \epsilon_i

, allowing for the joint influence of multiple factors on the response while controlling for confounding effects.^[54] To address potential interactions between predictors, where the effect of one variable depends on the level of another, interaction terms such as

\beta_{jk} X_{ij} X_{ik}

are included, modifying the model to capture non-additive relationships; for example, the impact of advertising spend on sales might vary by market region.^[58] Similarly, polynomial terms like

\beta_2 X_i^2

or higher powers enable modeling nonlinearity specific to individual predictors without assuming a global nonlinear form, such as quadratic effects in dose-response relationships where benefits plateau or diminish.^[59] These extensions maintain the linearity in parameters, preserving the core estimability via least squares. Model selection in linear regression involves choosing the optimal subset of predictors to balance fit and parsimony, often using stepwise procedures. Forward stepwise selection begins with an intercept-only model and iteratively adds the predictor yielding the largest improvement in fit, typically assessed via F-tests or information criteria, until no further additions are significant.^[60] Backward stepwise selection starts from the full model and removes the least useful predictor step-by-step, while bidirectional approaches combine both.^[60] Common criteria include the Akaike Information Criterion (AIC), which penalizes model complexity as

AIC = -2 \log(L) + 2k

where

L

is the likelihood and

k

the number of parameters, favoring models with lower values to avoid overfitting; the Bayesian Information Criterion (BIC) applies a harsher penalty

BIC = -2 \log(L) + k \log(n)

with sample size

n

, often selecting sparser models.^[61]^[61] Interpreting coefficients in multiple linear regression requires distinguishing between raw and standardized forms, as well as partial and marginal effects. Standardized coefficients, obtained by scaling predictors and the response to unit standard deviation, facilitate comparison of relative importance across variables with different units, such as expressing effects in terms of standard deviations changed.^[62] The partial effect of a predictor is the coefficient

\beta_j

, representing the change in

Y

for a unit change in

X_j

while holding all other predictors constant at their observed values.^[63] In contrast, marginal effects average this partial effect over the distribution of other covariates, providing an overall average impact that accounts for variability in the controls; for nonlinear extensions like interactions, marginal effects are computed as partial derivatives evaluated at representative points.^[63] A practical example of multiple linear regression is predicting house prices based on structural size and location attributes, as explored in hedonic pricing models. In the Boston housing dataset, the median value of owner-occupied homes (in thousands of dollars) is regressed on features like the number of rooms per dwelling (a proxy for size) and accessibility to employment centers (a location measure), yielding coefficients that quantify how a additional room or closer proximity increases value while controlling for other neighborhood factors such as crime rates and pupil-teacher ratios. This approach highlights prediction for valuation and policy analysis, with size often showing a positive linear effect and location capturing spatial premiums through continuous or categorical encodings.

Analysis of Variance (ANOVA)

Analysis of variance (ANOVA) serves as a special case of the linear model framework specifically designed to compare means across multiple groups defined by one or more categorical factors, enabling researchers to assess whether observed differences are statistically significant beyond random variation. Developed by Ronald Fisher in the early 20th century for agricultural experimentation, ANOVA partitions the total variability in the response variable into components attributable to the factors of interest and residual error, providing a structured approach to hypothesis testing in experimental designs.^[64] In one-way ANOVA, the model tests the null hypothesis that all group means are equal for a single categorical factor with k levels, using n observations per group under balanced designs. The total sum of squares (SST), which measures overall variability, is decomposed into the sum of squares among groups (SSA), capturing between-group differences, and the sum of squares within groups (SSE), representing unexplained error:

SST = SSA + SSE

Here, SSA quantifies the variation due to group differences, while SSE reflects intra-group variability. The mean square among groups (MSA) is computed as SSA divided by its degrees of freedom (k-1), and the mean square error (MSE) as SSE divided by N - k degrees of freedom, where N is the total number of observations. The test statistic is the F-ratio:

F = \frac{MSA}{MSE}

This F-value follows an F-distribution under the null hypothesis, allowing evaluation of whether group means differ significantly.^[64] Factorial ANOVA extends this to multiple factors, examining main effects of each factor as well as their interactions, which indicate whether the effect of one factor depends on the levels of another. In a two-way factorial design with factors A (a levels) and B (b levels) and replication (r > 1 per cell), the model partitions SST into SSA, SSB, SSAB (interaction), and SSE. Main effects assess average differences across levels of A or B, while the interaction term SSAB / ((a-1)(b-1)) tests for non-additive effects. For instance, in a balanced two-way design, the ANOVA table includes separate F-tests for each source, enabling detection of significant interactions that might otherwise be overlooked in separate one-way analyses.^[64] Specific to ANOVA, key assumptions include homogeneity of variances across groups (equal error variances), in addition to the linearity and independence assumptions of the general linear model. Violation of homogeneity can inflate Type I error rates, and it is typically assessed using robust tests such as Levene's test, which compares absolute deviations from group medians to detect unequal spreads. If significant differences in variances are found, remedies like data transformation or Welch's ANOVA may be applied.^[65] When the overall F-test rejects the null hypothesis of equal means, post-hoc tests are essential to identify which specific group pairs differ, controlling for multiple comparisons to maintain family-wise error rates. The Tukey honestly significant difference (HSD) test is a widely used method, computing pairwise differences in means and comparing them to a critical value based on the studentized range distribution, suitable for balanced designs with equal sample sizes.^[66] ANOVA is mathematically equivalent to ordinary least squares regression where categorical factors are encoded using dummy (indicator) variables, with k-1 dummies per factor to avoid multicollinearity; the regression coefficients then represent mean differences from a reference category, and the overall F-test mirrors the ANOVA result. A classic example illustrates ANOVA's application in agriculture: comparing crop yields across different fertilizer types. In Ronald Fisher's analysis of potato yields from Rothamsted experiments, yields were measured under four manure treatments (no manure, nitrogenous fertilizers, sulphate of potash, and combinations), with replication across plots. The one-way ANOVA partitioned variability to reveal significant differences in mean yields (e.g., SSA highlighting treatment effects, F > critical value at α=0.05), demonstrating the superiority of combined fertilizers while accounting for soil error variance.^[64]

Time Series Models

Linear models for time series data extend classical regression frameworks to account for temporal dependencies, where observations are not independent but exhibit serial correlation. Unlike standard linear regression, which assumes independent errors, time series models incorporate lagged values of the dependent variable or predictors to capture autocorrelation, enabling better modeling of processes like economic trends or seasonal patterns. These models maintain the linear structure in parameters but adjust for non-stationarity and error structures through techniques like differencing and moving averages. Autoregressive (AR) models form a foundational class, where the current value of the series is expressed as a linear combination of its past values plus a white noise error term. For an AR(p) model of order p, the equation is

Y_t = \beta_0 + \beta_1 Y_{t-1} + \beta_2 Y_{t-2} + \dots + \beta_p Y_{t-p} + \varepsilon_t

, where

\varepsilon_t

is independent and identically distributed noise with mean zero and constant variance. Stationarity, essential for the model's validity and interpretability, requires that the roots of the characteristic polynomial

1 - \beta_1 z - \beta_2 z^2 - \dots - \beta_p z^p = 0

lie outside the unit circle in the complex plane, ensuring the process has constant mean and variance over time. This condition prevents explosive behavior and allows the autocovariance function to decay appropriately. ARIMA models generalize AR processes by integrating autoregression, differencing for non-stationarity, and moving averages. The ARIMA(p, d, q) model applies d levels of differencing to achieve stationarity, yielding

\Delta^d Y_t = \beta_0 + \sum_{i=1}^p \beta_i \Delta^d Y_{t-i} + \sum_{j=1}^q \theta_j \varepsilon_{t-j} + \varepsilon_t

, where

\Delta

denotes the first difference operator and

\theta_j

are moving average coefficients. The orders p, d, and q are selected based on patterns in the autocorrelation and partial autocorrelation functions, balancing model parsimony with fit. This framework, introduced by Box and Jenkins, addresses trends and cycles in data that violate the independence assumption of classical models. In regression with time series data, linear models include lagged predictors or the dependent variable to handle dynamics, such as

Y_t = \beta_0 + \sum_{k=1}^m \beta_k X_{t-k} + \sum_{i=1}^p \phi_i Y_{t-i} + \varepsilon_t

. Autocorrelation in residuals, a common violation, is detected using the Durbin-Watson statistic, which tests the null hypothesis of no first-order serial correlation by comparing the sum of squared differences in residuals to their variance; values near 2 indicate no autocorrelation, while deviations suggest correction. Serial correlation inflates standard errors in ordinary least squares, addressed by generalized least squares (GLS) estimation, which transforms the model to account for the error covariance structure, or by embedding ARIMA errors directly.^[67] Forecasting with these models generates point predictions and intervals that account for accumulating uncertainty. In ARIMA, h-step-ahead prediction intervals widen with the forecast horizon due to the propagation of error variance, often approximated as

\hat{Y}_{t+h} \pm z_{\alpha/2} \sqrt{\hat{\sigma}^2 (1 + \sum_{i=1}^{h-1} \psi_i^2)}

Y^t+h±zα/2σ^2(1+∑i=1h−1ψi2)

, where $\psi_i$ are infinite moving average coefficients. For instance, ARIMA models have been applied to forecast U.S. unemployment rates, an economic indicator, where an ARIMA(1,2,0) fit to monthly data from 1948–2015 captures cyclical patterns and provides reliable short-term projections with intervals reflecting economic volatility. This differs from classical linear models by explicitly modeling temporal dependence, improving accuracy for dependent data.^[68]

Generalized Linear Models

Generalized linear models (GLMs) extend the classical linear model framework to accommodate response variables that follow distributions other than the normal distribution, allowing for a broader range of data types such as binary outcomes, counts, and proportions.^[69] Introduced by Nelder and Wedderburn in 1972, GLMs unify various regression techniques under a single structure while relaxing the assumption of normally distributed errors inherent in ordinary least squares estimation.^[69] This flexibility makes GLMs particularly useful for modeling non-continuous or heteroscedastic data, where the mean of the response relates to predictors through a link function and the variance depends on the mean.^[69] The core components of a GLM consist of three elements: a random component specifying the distribution of the response variable

Y

from the exponential family, a linear predictor

\eta = X\beta

where

X

is the design matrix and

\beta

the parameter vector, and a link function

g(\mu) = \eta

that connects the expected value

\mu = E(Y)

to the linear predictor.^[69] Additionally, the variance of

Y

is modeled as

V(Y) = V(\mu) \phi / n

, where

V(\mu)

is the variance function specific to the distribution,

\phi

is a dispersion parameter (often 1 for canonical forms), and

n

accounts for sample size in grouped data.^[69] Common distributions include the binomial for binary or proportion data and the Poisson for count data, each paired with canonical link functions that simplify estimation.^[69] A prominent example is logistic regression, a GLM with a binomial random component and logit link function

g(\mu) = \log(\mu / (1 - \mu))

, used to model binary outcomes such as the presence or absence of a disease based on predictors like age and exposure risk.^[69] Similarly, Poisson regression employs a Poisson distribution with a log link

g(\mu) = \log(\mu)

to analyze count data, such as the number of events occurring in fixed intervals under varying conditions.^[69] Parameter estimation in GLMs proceeds by maximum likelihood, typically via iteratively reweighted least squares (IRLS), which iteratively solves weighted least squares problems to converge on the likelihood maximum.^[69] For model assessment and comparison, GLMs use deviance as an analog to the residual sum of squares in linear models, defined as twice the difference in log-likelihoods between the fitted and saturated models, enabling tests of goodness-of-fit and nested model comparisons.^[69] Model selection often employs the Akaike information criterion (AIC), which penalizes complexity by AIC = -2 \log L + 2p, where L is the maximized likelihood and p the number of parameters, balancing fit and parsimony.^[70] These tools facilitate practical application across fields like epidemiology and ecology, where response distributions deviate from normality.^[69]

Authority control databases
International	GND
National	United States France BnF data Czech Republic Latvia Israel
Other	Yale LUX

History

Linear model

Recent from talks

Recent from talks

Contribute something

Contribute something

Media Pages

Timelines

Articles

Notes collections

Notes

Notes

Days in Chronicle

Linear model

Linear regression models

Time series models

Other uses in statistics

See also

References

Linear model

Basic Concepts

Definition and Scope

Historical Background

Mathematical Formulation

General Linear Model Equation

Matrix Representation

Assumptions and Diagnostics

Core Assumptions

Violation Detection and Remedies

Estimation and Inference

Ordinary Least Squares Estimation

Hypothesis Testing and Confidence Intervals

Add your contribution

Related Hubs

Contribute something