Lecture 27: Power and Type II Error

We have seen that once the level $\alpha$ of a test is chosen, there can be many tests with this level. How then to choose the best among them. One way is via Bayes Tests and tests based on Likelihood Ratios. Another possiblility is to compare the probability of Type II error among competing tests. Recall that a Type II error is accepting the null hypothesis when it is false. To put a positive face on things, we define the power of a test as the probability that the test rejects the null hypothesis when the null hypothesis is false. Thus the power of a test is really a function, ${\cal P}:H_1\rightarrow [0,1]$ where for $F\in H_0$,

$${\cal P}(F) = \Pr(H_0\;{\rm is\;rejected\;if\;}F\;{is\rm\;the\;true\;distribution}) .$$

If the alternative hypothesis can be parametrized, then the power is usually regarded as a function of that parameter.

An example. Let us assume that the underlying population is normal with variance 1 and unknown mean $\mu$. $H_0 = \{\mu = 0\}$ and $H_1 = \{\mu \neq 0\}$. A likelyhood ratio test is constructed based on a random sample $(X_1,\dots,X_N)$ using the statistic

$$\frac{X_1 + \cdots + X_N}{\sqrt{N}}.$$

A symmetric test of the form ``accept H₀ if |T(X)| < 1.96'' is to be used. What is the power of this test? Note that under both the null and the alternative hypotheses, T(X) is normally distributed with variance 1. However, under the alternative hypothesis, if $\mu = \theta$, then ${\rm E}[T(X)] = \theta/\sqrt{N}$ and $T(X) - n{1/2}\theta$ has a standard normal distribution. Since it is always true that

$$\Pr(\vert T(X)\vert < 1.96) = \Pr(-1.96 - N^{1/2}\theta < T(X)-N^{-1/2}\theta < 1.96 - N^{1/2}\theta)$$

we see that

$${\cal P}(\theta) = 1-\int_{-1.96 - N^{1/2}\theta}^{1.96 - N^{1/2}\theta}\frac{\exp(-u^2/2)}{\sqrt{2\pi}}\;du.$$

This shows us two things.

• The power increases to 1 as $\vert\theta\vert\rightarrow\infty$.
• For any fixed $\theta\neq 0$, the power increases to 1 as the sample size $N\rightarrow\infty$.