Robust measures of scale

Robust measures of scale can be used as estimators of properties of the population, either for parameter estimation or as estimators of their own expected value.

For example, robust estimators of scale are used to estimate the population standard deviation, generally by multiplying by a scale factor to make it an unbiased consistent estimator; see scale parameter: estimation. For example, the interquartile range may be rendered an unbiased, consistent estimator for the population standard deviation if the data follow a normal distribution and the measure is divided by: $${\displaystyle 1.349\approx 2{\sqrt {2}}\,\operatorname {erf} ^{-1}\!\left({\tfrac {1}{2}}\right)}$$ where ${\displaystyle \operatorname {erf} ^{-1}}$ is the inverse error function.

In other situations, it makes more sense to think of a robust measure of scale as an estimator of its own expected value, interpreted as an alternative to the population standard deviation as a measure of scale. For example, the median absolute deviation (MAD) of a sample from a standard Cauchy distribution is an estimator of the population MAD, which in this case is 1, whereas the population variance does not exist.

Robust estimators typically have inferior statistical efficiency compared to conventional estimators for data drawn from a distribution without outliers, such as a normal distribution. However, they have superior efficiency for data drawn from a mixture distribution or from a heavy-tailed distribution, for which non-robust measures such as the standard deviation should not be used.

For example, for data drawn from the normal distribution, the median absolute deviation is 37% as efficient as the sample standard deviation, while the Rousseeuw–Croux estimator Q_n is 88% as efficient as the sample standard deviation.

One of the most common robust measures of scale is the interquartile range (IQR), the difference between the 75th percentile and the 25th percentile of a sample; this is the 25% trimmed range, an example of an L-estimator. Other trimmed ranges, such as the interdecile range (10% trimmed range) can also be used.

For a Gaussian distribution, IQR is related to ${\displaystyle \sigma }$, the standard deviation, as:^[1] $${\displaystyle \sigma \approx 0.7413\operatorname {IQR} =\operatorname {IQR} /1.349}$$

Another commonly used robust measure of scale is the median absolute deviation (MAD), the median of the absolute values of the differences between the data values and the overall median of the data set; for a Gaussian distribution, MAD is related to ${\displaystyle \sigma }$ as:^[2] $${\displaystyle \sigma \approx 1.4826\operatorname {MAD} \approx \operatorname {MAD} /0.6745}$$ For details, visit the section on relation to standard deviation in the main article on MAD.

Rousseeuw and Croux^[3] proposed 2 alternatives to the Median Absolute Deviation, motivated by two of its weaknesses:

1. It is inefficient (37% efficiency) at Gaussian distributions.
2. it computes a symmetric statistic about a location estimate, thus not dealing with skewness.

They propose two alternative statistics based on pairwise differences: S_n and Q_n.

S_n is defined as: $${\displaystyle {\begin{aligned}\sigma \approx S_{n}&:=1.1926\,\operatorname {med} _{i}\left(\operatorname {med} _{j}(\,\left|x_{i}-x_{j}\right|\,)\right),\\\end{aligned}}}$$ Q_n is defined as:^[4]

$${\displaystyle \sigma \approx Q_{n}:=2.2219\;{\bigl \{}|x_{i}-x_{j}|:i<j{\bigr \}}_{(k)}}$$

Where:

• The factor 2.2219 is a consistency constant,
• The set ${\displaystyle \{\,|x_{i}-x_{j}|:i<j\,\}}$ consists of all pairwise absolute differences between the observations ${\displaystyle x_{i}}$and ${\displaystyle x_{j}}$, and
• The subscript ${\displaystyle (k)}$ represents the ${\displaystyle k}$th order statistic, or ${\displaystyle k\approx {\binom {n}{2}}/4}$

These can be computed in O(n log n) time and O(n) space.

Neither of these requires location estimation, as they are based only on differences between values. They are both more efficient than the MAD under a Gaussian distribution: S_n is 58% efficient, while Q_n is 82% efficient.

For a sample from a normal distribution, S_n is approximately unbiased for the population standard deviation even down to very modest sample sizes (<1% bias for n = 10).

For a large sample from a normal distribution, 2.22Q_n is approximately unbiased for the population standard deviation. For small or moderate samples, the expected value of Q_n under a normal distribution depends markedly on the sample size, so finite-sample correction factors (obtained from a table or from simulations) are used to calibrate the scale of Q_n.

Like S_n and Q_n, the biweight midvariance is intended to be robust without sacrificing too much efficiency. It is defined as:^[5]

$${\displaystyle {\frac {n\displaystyle \sum _{i=1}^{n}\left(x_{i}-Q\right)^{2}\left(1-u_{i}^{2}\right)^{4}I(|u_{i}|<1)}{\left(\displaystyle \sum _{i}\left(1-u_{i}^{2}\right)\left(1-5u_{i}^{2}\right)I(|u_{i}|<1)\right)^{2}}},}$$

where I is the indicator function, Q is the sample median of the X_i, and

$${\displaystyle u_{i}={\frac {x_{i}-Q}{9\cdot {\rm {MAD}}}}.}$$

Its square root is a robust estimator of scale, since data points are downweighted as their distance from the median increases, with points more than 9 MAD units from the median having no influence at all.

The biweight's efficiency has been estimated at around 84.7% for sets of 20 samples drawn from synthetically generated distributions with added excess kurtosis ("stretched tails"). For Gaussian distributions, its efficiency has been estimated at 98.2%.^[6]

Mizera and Müller extended the approach offered by Rousseeuw and Hubert by proposing a robust depth-based estimator for location and scale simultaneously, called location-scale depth. It is defined as follows:^[7]

$${\displaystyle d(\mu ,\sigma )={\begin{cases}\displaystyle \inf _{u\neq 0}\#{\Bigl \{}i:(u_{1},u_{2}){\Bigl (}{\begin{array}{c}\psi (\tau _{i})\\\chi (\tau _{i})-1\end{array}}{\Bigr )}\geq 0{\Bigr \}},&{\text{if }}\sigma >0,\\[1.5ex]\#\{i:y_{i}=\mu \},&{\text{if }}\sigma =0.\end{cases}}}$$

Where:

• ${\displaystyle \tau _{i}}$is a shorthand for ${\displaystyle (y_{i}-\mu )/\sigma }$,
• ${\displaystyle \psi }$ and ${\displaystyle \chi }$ depend on a fixed density ${\displaystyle f}$

They suggest that the most tractable version of location-scale depth is the one based on Student's t-distribution.

This section may need to be cleaned up. It has been merged from Robust confidence intervals.

A robust confidence interval is a robust modification of confidence intervals, meaning that one modifies the non-robust calculations of the confidence interval so that they are not badly affected by outlying or aberrant observations in a data-set.

In the process of weighing 1000 objects, under practical conditions, it is easy to believe that the operator might make a mistake in procedure and so report an incorrect mass (thereby making one type of systematic error). Suppose there were 100 objects and the operator weighed them all, one at a time, and repeated the whole process ten times. Then the operator can calculate a sample standard deviation for each object, and look for outliers. Any object with an unusually large standard deviation probably has an outlier in its data. These can be removed by various non-parametric techniques. If the operator repeated the process only three times, simply taking the median of the three measurements and using σ would give a confidence interval. The 200 extra weighings served only to detect and correct for operator error and did nothing to improve the confidence interval. With more repetitions, one could use a truncated mean, discarding the largest and smallest values and averaging the rest. A bootstrap calculation could be used to determine a confidence interval narrower than that calculated from σ, and so obtain some benefit from a large amount of extra work.

These procedures are robust against procedural errors which are not modeled by the assumption that the balance has a fixed known standard deviation σ. In practical applications where the occasional operator error can occur, or the balance can malfunction, the assumptions behind simple statistical calculations cannot be taken for granted. Before trusting the results of 100 objects weighed just three times each to have confidence intervals calculated from σ, it is necessary to test for and remove a reasonable number of outliers (testing the assumption that the operator is careful and correcting for the fact that he is not perfect), and to test the assumption that the data really have a normal distribution with standard deviation σ.

The theoretical analysis of such an experiment is complicated, but it is easy to set up a spreadsheet which draws random numbers from a normal distribution with standard deviation σ to simulate the situation; this can be done in Microsoft Excel using =NORMINV(RAND(),0,σ)), as discussed in ^[8] and the same techniques can be used in other spreadsheet programs such as in OpenOffice.org Calc and gnumeric.

After removing obvious outliers, one could subtract the median from the other two values for each object, and examine the distribution of the 200 resulting numbers. It should be normal with mean near zero and standard deviation a little larger than σ. A simple Monte Carlo spreadsheet calculation would reveal typical values for the standard deviation (around 105 to 115% of σ). Or, one could subtract the mean of each triplet from the values, and examine the distribution of 300 values. The mean is identically zero, but the standard deviation should be somewhat smaller (around 75 to 85% of σ).

• Heteroscedasticity-consistent standard errors
• Interquartile Range
• Mean Absolute Deviation