Frequentist inference

Frequentism is based on the presumption that statistics represent probabilistic frequencies. This view was primarily developed by Ronald Fisher and the team of Jerzy Neyman and Egon Pearson. Ronald Fisher contributed to frequentist statistics by developing the frequentist concept of "significance testing", which is the study of the significance of a measure of a statistic when compared to the hypothesis.

Neyman-Pearson extended Fisher's ideas to apply to multiple hypotheses. They posed that the ratio of probabilities of two given hypotheses, when maximizing the difference between them, leads to a maximization of exceeding a given p-value. This relationship serves as the basis of type I and type II errors and confidence intervals.

For statistical inference, the statistic about which we want to make inferences is ${\displaystyle y\in Y}$, where the random vector ${\displaystyle Y}$ is a function of an unknown parameter, ${\displaystyle \theta }$.

The parameter ${\displaystyle \theta }$, in turn, is partitioned into (${\displaystyle \psi ,\lambda }$), where ${\displaystyle \psi }$ is the parameter of interest, and ${\displaystyle \lambda }$ is the nuisance parameter. For concreteness, ${\displaystyle \psi }$ might be the population mean, ${\displaystyle \mu }$, and the nuisance parameter ${\displaystyle \lambda }$ the standard deviation of the population mean, ${\displaystyle \sigma }$.^[1]

Thus, statistical inference is concerned with the expectation of random vector ${\displaystyle Y}$, ${\displaystyle E(Y)=E(Y;\theta )=\int yf_{Y}(y;\theta )dy}$.

To construct areas of uncertainty in frequentist inference, a pivot is used which defines the area around ${\displaystyle \psi }$ that can be used to provide an interval to estimate uncertainty. The pivot is a probability such that for a pivot, ${\displaystyle p}$, which is a function, that ${\displaystyle p(t,\psi )}$ is strictly increasing in ${\displaystyle \psi }$, where ${\displaystyle t\in T}$ is a random vector.

This allows that, for some ${\displaystyle 0<c<1}$, we can define ${\displaystyle P\{p(T,\psi )\leq p_{c}^{*}\}}$, which is the probability that the pivot function is less than some well-defined value. This implies ${\displaystyle P\{\psi \leq q(T,c)\}=1-c}$, where ${\displaystyle q(t,c)}$ is a ${\displaystyle 1-c}$ upper limit for ${\displaystyle \psi }$.

Note that ${\displaystyle 1-c}$ is a range of outcomes that define a one-sided limit for ${\displaystyle \psi }$, and that ${\displaystyle 1-2c}$ is a two-sided limit for ${\displaystyle \psi }$, when we want to estimate a range of outcomes where ${\displaystyle \psi }$ may occur. This rigorously defines the confidence interval, which is the range of outcomes about which we can make statistical inferences.

Two complementary concepts in frequentist inference are the Fisherian reduction and the Neyman-Pearson operational criteria. Together these concepts illustrate a way of constructing frequentist intervals that define the limits for ${\displaystyle \psi }$. The Fisherian reduction is a method of determining the interval within which the true value of ${\displaystyle \psi }$ may lie, while the Neyman-Pearson operational criteria is a decision rule about making a priori probability assumptions.

The Fisherian reduction is defined as follows:

• Determine the likelihood function (this is usually just gathering the data);
• Reduce to a sufficient statistic ${\displaystyle S}$ of the same dimension as ${\displaystyle \theta }$;
• Find the function of ${\displaystyle S}$ that has a distribution depending only on ${\displaystyle \psi }$;
• Invert that distribution (this yields a cumulative distribution function or CDF) to obtain limits for ${\displaystyle \psi }$ at an arbitrary set of probability levels;
• Use the conditional distribution of the data given ${\displaystyle S=s}$ informally or formally as to assess the adequacy of the formulation.^[2]

Essentially, the Fisherian reduction is design to find where the sufficient statistic can be used to determine the range of outcomes where ${\displaystyle \psi }$ may occur on a probability distribution that defines all the potential values of ${\displaystyle \psi }$. This is necessary to formulating confidence intervals, where we can find a range of outcomes over which ${\displaystyle \psi }$ is likely to occur in the long-run.

The Neyman-Pearon operational criteria is an even more specific understanding of the range of outcomes where the relevant statistic, ${\displaystyle \psi }$, can be said to occur in the long run. The Neyman-Pearson operational criteria defines the likelihood of that range actually being adequate or of the range being inadequate. The Neyman-Pearson criteria defines the range of the probability distribution that, if ${\displaystyle \psi }$ exists in this range, is still below the true population statistic. For example, if the distribution from the Fisherian reduction exceeds a threshold that we consider to be a priori implausible, then the Neyman-Pearson reduction's evaluation of that distribution can be used to infer where looking purely at the Fisherian reduction's distributions can give us inaccurate results. Thus, the Neyman-Pearson reduction is used to find the probability of type I and type II errors.^[3] As a point of reference, the complement to this in Bayesian statistics is the minimum Bayes risk criterion.

Because of the reliance of the Neyman-Pearson criteria on our ability to find a range of outcomes where ${\displaystyle \psi }$ is likely to occur, the Neyman-Pearson approach is only possible where a Fisherian reduction can be achieved.^[4]

Frequentist inferences are associated with the application frequentist probability to experimental design and interpretation, and specifically with the view that any given experiment can be considered one of an infinite sequence of possible repetitions of the same experiment, each capable of producing statistically independent results.^[5] In this view, the frequentist inference approach to drawing conclusions from data is effectively to require that the correct conclusion should be drawn with a given (high) probability, among this notional set of repetitions.

However, exactly the same procedures can be developed under a subtly different formulation. This is one where a pre-experiment point of view is taken. It can be argued that the design of an experiment should include, before undertaking the experiment, decisions about exactly what steps will be taken to reach a conclusion from the data yet to be obtained. These steps can be specified by the scientist so that there is a high probability of reaching a correct decision where, in this case, the probability relates to a yet to occur set of random events and hence does not rely on the frequency interpretation of probability. This formulation has been discussed by Neyman,^[6] among others. This is especially pertinent because the significance of a frequentist test can vary under model selection, a violation of the likelihood principle.

Frequentism is the study of probability with the assumption that results occur with a given frequency over some period of time or with repeated sampling. As such, frequentist analysis must be formulated with consideration to the assumptions of the problem frequentism attempts to analyze. This requires looking into whether the question at hand is concerned with understanding variety of a statistic or locating the true value of a statistic. The difference between these assumptions is critical for interpreting a hypothesis test.

There are broadly two camps of statistical inference, the epistemic approach and the epidemiological approach. The epistemic approach is the study of variability; namely, how often do we expect a statistic to deviate from some observed value. The epidemiological approach is concerned with the study of uncertainty; in this approach, the value of the statistic is fixed but our understanding of that statistic is incomplete.^[7] For concreteness, imagine trying to measure the stock market quote versus evaluating an asset's price. The stock market fluctuates so greatly that trying to find exactly where a stock price is going to be is not useful: the stock market is better understood using the epistemic approach, where we can try to quantify its fickle movements. Conversely, the price of an asset might not change that much from day to day: it is better to locate the true value of the asset rather than find a range of prices and thus the epidemiological approach is better. The difference between these approaches is non-trivial for the purposes of inference.

For the epistemic approach, we formulate the problem as if we want to attribute probability to a hypothesis. This can only be done with Bayesian statistics, where the interpretation of probability is straightforward because Bayesian statistics is conditional on the entire sample space, whereas frequentist testing is concerned with the whole experimental design. Frequentist statistics is conditioned not on solely the data but also on the experimental design.^[8] In frequentist statistics, the cutoff for understanding the frequency occurrence is derived from the family distribution used in the experiment design. For example, a binomial distribution and a negative binomial distribution can be used to analyze exactly the same data, but because their tail ends are different the frequentist analysis will realize different levels of statistical significance for the same data that assumes different probability distributions. This difference does not occur in Bayesian inference. For more, see the likelihood principle, which frequentist statistics inherently violates.^[9]

For the epidemiological approach, the central idea behind frequentist statistics must be discussed. Frequentist statistics is designed so that, in the long-run, the frequency of a statistic may be understood, and in the long-run the range of the true mean of a statistic can be inferred. This leads to the Fisherian reduction and the Neyman-Pearson operational criteria, discussed above. When we define the Fisherian reduction and the Neyman-Pearson operational criteria for any statistic, we are assessing, according to these authors, the likelihood that the true value of the statistic will occur within a given range of outcomes assuming a number of repetitions of our sampling method.^[8] This allows for inference where, in the long-run, we can define that the combined results of multiple frequentist inferences to mean that a 95% confidence interval literally means the true mean lies in the confidence interval 95% of the time, but not that the mean is in a particular confidence interval with 95% certainty. This is a popular misconception.

Very commonly the epistemic view and the epidemiological view are incorrectly regarded as interconvertible. First, the epistemic view is centered around Fisherian significance tests that are designed to provide inductive evidence against the null hypothesis, ${\displaystyle H_{0}}$, in a single experiment, and is defined by the Fisherian p-value. Conversely, the epidemiological view, conducted with Neyman-Pearson hypothesis testing, is designed to minimize the Type II false acceptance errors in the long-run by providing error minimizations that work in the long-run. The difference between the two is critical because the epistemic view stresses the conditions under which we might find one value to be statistically significant; meanwhile, the epidemiological view defines the conditions under which long-run results present valid results. These are extremely different inferences, because one-time, epistemic conclusions do not inform long-run errors, and long-run errors cannot be used to certify whether one-time experiments are sensical. The assumption of one-time experiments to long-run occurrences is a misattribution, and the assumption of long run trends to individuals experiments is an example of the ecological fallacy.^[10]

Frequentist inferences stand in contrast to other types of statistical inferences, such as Bayesian inferences and fiducial inferences. While the "Bayesian inference" is sometimes held to include the approach to inferences leading to optimal decisions, a more restricted view is taken here for simplicity.

Bayesian inference is based in Bayesian probability, which treats "probability" as equivalent with "certainty", and thus that the essential difference between the frequentist inference and the Bayesian inference is the same as the difference between the two interpretations of what a "probability" means. However, where appropriate, Bayesian inferences (meaning in this case an application of Bayes' theorem) are used by those employing frequency probability.

There are two major differences in the frequentist and Bayesian approaches to inference that are not included in the above consideration of the interpretation of probability:

1. In a frequentist approach to inference, unknown parameters are typically considered as being fixed, rather than as being random variates. In contrast, a Bayesian approach allows probabilities to be associated with unknown parameters, where these probabilities can sometimes have a frequency probability interpretation as well as a Bayesian one. The Bayesian approach allows these probabilities to have an interpretation as representing the scientist's belief that given values of the parameter are true (see Bayesian probability - Personal probabilities and objective methods for constructing priors).
2. The result of a Bayesian approach can be a probability distribution for what is known about the parameters given the results of the experiment or study. The result of a frequentist approach is either a decision from a significance test or a confidence interval.

• Intuitive statistics
• German tank problem