Mean Square Error

Definition 8..3
For a random sample $X_1, \ldots, X_n$ from $f(x;\theta)$ and a statistic
T $=t(X_1, \ldots, X_n$) which is an estimator of $\theta$, the mean square error (mse) is defined as

$$mse=E[(T-\theta)^2] .$$ (8.2)

The mse can be expressed alternatively as

$$\begin{eqnarray*} E[(T-\theta)^2]&=&E[(T-E(T))+(E(T)-\theta)]^2\\ &=&E[(T-E(T))^2] \quad + \quad [E(T)-\theta]^2. \end{eqnarray*}$$

So we have

$$\mbox{mse}=\mbox{Var}(T) \quad + \quad b_T^2(\theta).$$ (8.3)

Now from (8.3) we can see that the mse cannot usually be made equal to zero. It will only be small when both Var($T$) and the bias in $T$ are small. So rather than use unbiasedness and minimum variance to characterize ``goodness'' of a point estimator, we might employ the mean square error.

Example 8..1
Consider the problem of the choice of estimator of $\sigma^2$ based on a random sample of size $n$ from a $N(\mu , \sigma^2 )$ distribution. Recall that $S^2=\sum_{i=1}^n(X_i-\overline{X})^2/(n-1)$ is often called the sample variance and has the properties

$$\begin{eqnarray*} E(S^2)&=&\sigma^2, \quad \mbox{ (so $S^2$\ is unbiased)}\\ \mbox{Var}(S^2)&=&2 \sigma^4/(n-1). \end{eqnarray*}$$

[Note that this is not HC's use of $S^2$. See 4.1 Definition 3.]

Consider the mle of $\sigma^2$, $\sum_{i=1}^n(X_i-\overline{X}^2)/n$, which we'll denote by $\hat{\sigma}^2$. Now $\hat{\sigma}^2=\frac{n-1}{n}S^2$ and

$$\begin{eqnarray*} E(\hat{\sigma}^2)&=& \frac{n-1}{n}\sigma^2 = \left(1-\frac{1}{... ...=&\frac{(n-1)^2}{n^2}\mbox{Var}(S^2)=\frac{2(n-1)}{n^2}\sigma^4. \end{eqnarray*}$$

Why is $\hat \sigma^2$ biassed? To calculate $\hat \sigma^2$ we first have to extract the mean, consuming 1 degree of freedom. So we do not have $n$ independent estimates of dispersion about the mean; we have $(n-1)$.

Now $\hat{\sigma}^2$ is biased, but what about its mean square error? Using (2.3),

$$\begin{eqnarray*} \mbox{mse} \ \hat{\sigma}^2&=&\frac{2(n-1)\sigma^4}{n^2} \quad... ...\frac{1}{n})\sigma^2\right]^2\\ &=& \frac{\sigma^4(2n-1)}{n^2}. \end{eqnarray*}$$

Now for $S^2$,

$$\mbox{mse} \ S^2=\mbox{Var}(S^2)=\frac{2 \sigma^4}{n-1} > \left(\frac{2n-1}{n^2}\right)\sigma^4$$

since $\frac{2}{n-1} > \frac{2n-1}{n^2}$ for $n$ an integer greater than 1. So for the normal distribution the mle of $\sigma^2$ is better in the sense of mse than the sample variance.