Minimum Variance Estimation

The Theorem below gives a lower bound for the variance (or mse) of an estimator.

Theorem 8..1
Let $T=t({\bf X})$, based on a sample X from $f(x;\theta)$ be an estimator of $\theta$ (assumed to be one-dimensional). Then

$$\mbox{Var}(T) \geq \frac{[1+b_T'(\theta)]^2}{I_{\bf X}(\theta)}$$ (8.7)

and

$$\mbox{mse}(T) \geq \frac{[1+b_T'(\theta)]^2}{I_{\bf X}(\theta)} \quad + \quad b_T^2(\theta)$$ (8.8)

where $b_T(\theta)$ is given in (8.1) and $I_{\bf X}(\theta)$ is defined in (7.6).

Outline of Proof. [The validity depends on regularity conditions, where the interchange of integration and differentiation operations is permitted and on the existence and integrability of various partial derivatives.]

Define $V=\frac{\partial}{\partial \theta}\log f(X; \theta)=\frac{\partial}{\partial \theta}\log L(\theta)$ as in (7.7) and we note that $E(V)=0$ so Var(V)$=E(V^2)$ and

$$\begin{eqnarray*} % latex2html id marker 10298\mbox{cov}(V, T)&=&E(VT)\\ &=&E... ...mbox{ from (\ref{eq:bias})}\\ &=& 1 \quad + \quad b_T'(\theta). \end{eqnarray*}$$

Recall that the absolute value of the correlation coefficient, for measuring correlation between any two variables is less than or equal to $1$ and that

$$\rho_{V,T}=\mbox{cov}(V,T)/\sigma_V \sigma_T$$

so that we have

$$[\mbox{cov}(V, T)]^2 \leq \mbox{Var}(V)\mbox{Var}(T)$$

or

$$\begin{eqnarray*} \mbox{Var}(T) &\geq& [\mbox{cov}(V,T)]^2/\mbox{Var}(V)\\ &=& \frac{[1 + b_T'(\theta)]^2}{I_{\bf X}(\theta)} \end{eqnarray*}$$

thus proving (8.7). Now (8.8) follows using ( 8.3).

Corollary. For the class of unbiased estimators,

$$\mbox{Var}(T)=\mbox{mse}(T) \geq \frac{1}{I_{\bf X}(\theta)}.$$ (8.9)

Now inequality (8.9) is known as the Cramér-Rao lower bound, or sometimes the Information inequality. It provides (in ``regular estimation'' cases) a lower bound on the variance of an unbiased estimator, T. The inequality is generally attributed to Cramér's work in 1946 and Rao's work in 1945, though it was apparently first given by M. Frechet in 1937-38.

Definition 8..8
The (absolute) efficiency of an unbiased estimator T is defined as

$$e(T)=\frac{1/I_{\bf X}(\theta)}{\mbox{Var}(T)}.$$ (8.10)

Note that, because of (8.9), $e(T) \leq 1$, so we can think of $e(T)$ as a measure of efficiency of any given estimator, rather than the relative efficiency of one with respect to another as in Definition 8.1.

In the case where $e(T)=1$, so that the actual lower bound of Var($T$) is achieved, some texts refer to the estimator T as efficient. This terminology is not universally accepted. Some prefer to use the phrase minimum variance bound (MVB) for $1/I_{\bf X}(\theta)$, and an estimator which is unbiased and which attains this bound is called a minimum variance bound unbiased (MVBU) estimator.

Example 8..4
In the problem of estimating $\theta$ in a normal distribution with mean $\theta$ and known variance $\sigma^2$, find the MVB of an unbiased estimator.

The MVB is $1/I_{\bf X}(\theta)$ where $I_{\bf X}(\theta)=E\left(\frac{\partial \log L(\theta)}{\partial \theta}\right)^2$. For a sample $X_1, \ldots, X_n$ we have for the likelihood,

$$\begin{eqnarray*}L(\theta)&=& \prod_{i=1}^n \frac{1}{\sqrt{2 \pi}\sigma}e^{-\fra... ...2\\ &=& \frac{n^2}{\sigma^4}\mbox{Var}(\overline{X})=n/\sigma^2 \end{eqnarray*}$$

So the MVB is $\sigma^2/n$.