Power Function and Significance Level

Suppose that $T$ is the test statistic and $C$ the critical region for a test of a hypothesis concerning the value of a parameter $\theta$. Then the power function of the test is the probability that the test rejects $H_0$, when the actual parameter value is $\theta$. That is,

$$\pi(\theta)=P_{\theta}(\mbox{rejecting } H_0) = P_{\theta}(C).$$ (9.1)

Some texts interpret power as the probability of rejecting $H_0$ when it is false, but the more general interpretation is the probability that a test rejects $H_0$ for $\theta$ taking values given by $H_0$ or $H_1$.

Suppose we want to test the simple $H_0:\,\theta=\theta_0$, against the composite alternative $H_1:\theta \neq \theta_0$. Ideally we would like a test to detect a departure from $H_0$ with certainty; that is, we would like $\pi(\theta)$ to be $1$ for all $\theta$ in $H_1$, and $\pi(\theta)$ to be $0$ for $\theta$ in $H_0$. Since for a fixed sample size, P(rejecting $H_0\vert H_0$ is true) and P(not rejecting $H_0\vert H_0$ is false) cannot both be made arbitrarily small, the ideal test is not possible.

So long as $H_0$ is simple, it is possible to define P(Type I error), denoted by $\alpha$, as P(rejecting $H_0\vert H_0$ is true). But to allow for $H_0$ to be composite, we need the following definitions.

Definition 9..1
The size of a test (or of a critical region) is

$$\alpha = \max_{\theta \in H_0} P_{\theta}(\mbox{reject }H_0) = \max_{\theta \in H_0}\pi(\theta)$$ (9.2)

This is also known as the significance level.

Definition 9..2
The size of Type II error is

$$\beta = \max_{\theta \in H_1}[1-\pi(\theta)].$$ (9.3)

Some statisticians regard the formal approach above, of setting up a rejection region, as not the most appropriate, and prefer to compute a P-value. This involves the choice of a test statistic T, the extreme values of which provide evidence against $H_0$. The statistic T should be a good estimator of $\theta$ and its distribution under $H_0$ known. After experimentation, an observed value of T, t say, is examined to see whether it can be considered extreme in the sense of being unlikely to occur if $H_0$ were true. The computed P-value is the probability of observing $T=t$ or something more extreme. This is the ``$\alpha$'' at which the observed value $T=t$ is just significant.