Lecture 16: Student's t Statistic

We want to consider a very important special case of an orthogonal transformation of a random sample, V, of size k from a mean zero normal population. Suppose that the first column of my orthogonal matrix O has all entries equal to $1/\sqrt{k}$. Put $R \equiv (R_1,\dots,R_k) = VO$. Then R is also a random sample of size k from the same normal distribution,

$$R_1^2 + \cdots + R_k^2 = V_1^2 + \cdots + V_k^2$$

and

$$R_1 = \frac{V_1 + \cdots +V_k}{\sqrt{k}} = \sqrt{k}\frac{V_1 + \cdots + V_k}{k},$$

while

$$\begin{array} {rcl} R_2^2 + \cdots + R_k^2 & = & V_1^2 + \c... ...{$\displaystyle{\frac{V_1+\dots+V_k}{k}}$}\right)^2.\end{array}$$

Observe that since R₁ and the vector $(R_2, \dots, R_k)$ are independent, we have shown that for a random sample from a normal distribution with mean 0 the maximum likelihood estimate of the mean and the maximum likelihood estimate of the variance are independent. In fact, if the random sample were from a normal population with arbitrary mean $\mu$, since $V_i - \mu$ is normally distributed with mean 0, and the maximum likelihood estimate of the variance is unchanged when the a constant is subtracted from the observations, we have shown that the maximum likelihood estimates of the mean and the variance are independent whenever the population is normally distributed.

What is more, the maximum likelihood estimate of the mean is a random variable with a normal distribution, and the maximum likelihood estimate of the variance has a gamma distribution. This makes it possible to compute the density of the ratio of these two maximum likelihood ratios. Before we do so, let us see why we might want to bother.

Suppose that we had a population that was known to have a normal distribution, but we knew neither the mean nor the variance of this distribution. In fact, our problem was to try to determine if the mean of this normal population was 0 or not. To do this, we initially might try to use collect a random sample of size k, compute the maximum likelihood estimate of the mean,

$$\mu_{MLE}(V) \equiv \frac{V_1 + \cdots + V_k}{k}$$

and decide that if $\mu_{MLE}(V)$ is near to 0 then the population mean is 0 and if $\mu_{MLE}(V)$ is not near to 0 then the population mean is not 0. We would quantify this by picking a set A containing 0, and saying

• Accept the assertion that the mean is 0 if $\mu_{MLE}(V) \in A$.
• Reject the assertion that the mean is 0 if $\mu_{MLE}(V) \in A^c$.

The natural questions to ask are then

• If the mean really is 0, how likely is it that our test will tell us to reject the assertion that the mean is 0? In other words, what is

  $$\Pr(\mu_{MLE}(V) \in A^c)$$

  if the mean is 0?
• If the mean really is not 0, how likely is it that our test will tell us to accept the assertion that the mean is 0? In other words, what is

  $$\Pr(\mu_{MLE}(V) \in A)$$

  if the mean is 0?

The problem is that we cannot answer even the first question, for although we know that $\mu_{MLE}(V)$ has a normal distribution with mean 0 we do not know its variance. If the variance were known we could divide by its square root, the standard deviation, and get a random variable with a standard normal distribution. Of course, then A would depend on this standard deviation. What to do?

The obvious thing to try would be to try to divide by an estimate of the standard deviation. One convenient estimate would be the maximum likelihood estimate of the standard deviation,

$$\sigma_{MLE}(V) \equiv \frac{1}{k}\sum_{j=1}^k \left(V_j - \mbox{$\displaystyle{\frac{V_1+\dots+V_k}{k}}$}\right)^2,$$

especially since we know that it is independent of $\mu_{MLE}(V)$! At first glance this seems like it won't quite work as the distribution of $\sigma_{MLE}(V)$ also depends on the standard deviation of the sample, $\sigma$. However, all is not lost, since

$$\frac{\mu_{MLE}(V)}{\sigma_{MLE}(V)} = \frac{\sqrt{k}\mu_{MLE}(V)/\sigma}{\sqrt{k}\sigma_{MLE}(V)/\sigma}.$$

Now notice that if the population mean is really 0, then $\sqrt{k}\sigma_{MLE}(V)/\sigma$ is the square root of the sum of the squares of k-1 independent, normally distributed random variables, each with mean 0 and variance 1, and $\sqrt{k}\mu_{MLE}(V)/\sigma$ is a random variable which is normally distributed with mean and variance 1. Thus if V is a random sample from a normal distribution with mean 0, the distribution of

$$T(V) \equiv \frac{\sqrt{k}\mu_{MLE}(V)}{\sigma_{MLE}(V)}$$

is the same as the distribution of the ratio of two random variables R and S, R/S, where

• R and S are independent;
• R has the standard normal distribution,
• S² has a gamma distribution with $\alpha = (k-1)/2$ and $\beta = 2$,that is, S² has a chi-square distribution with k-1 degrees of freedom.

It is straightforward to find the density of R/S, so we shall do so below. Right now, it is important to realize that we can formulate our test about whether or not the population mean of our normal population is 0 in terms of $t(V) = \sqrt{k}T(V)$, and the all probabilities involving t(V) can be computed so long as the sample size is known. t(V) is called Student's t statistic and its density (really a family of densities depending on the sample size k) is called the t density.

A formula for the t density

Let R and S be as above. Then since R and S are independent and R has a density which is symmetric about 0, R/S also will have a density which is symmetric about 0. Thus we will first try to find $\Pr( -a \leq R/S \leq a)$ for any positive number a. From here it is easy to get the density of R/S. Now

$$\{-a \leq R/S \leq a\} = \{R^2/S^2 \leq a^2\} = \left\{\frac{S^2}{R^2+S^2} \geq \frac{1}{1+a^2}\right\}$$

and we know that since R² and S² are independent gamma random variables with $\beta = 2$ and $\alpha$ parameters equal to 1/2 and (k-1)/2 respectively, that S²/(R² + S²) has a B((k-1)/2,1/2) distribution. Therefore,

$$\Pr(-a \leq R/S \leq a) = \int_{1/(1+a^2)}^1 \frac{x^{(k-3)/2}(1-x)^{-1/2}}{B((k-1)/2,1/2)}$$

and so

$$\Pr(-\infty < R/S \leq a) = \frac{1}{2} + \frac{1}{2}\int_{1/(1+a^2)}^1 \frac{x^{(k-3)/2}(1-x)^{-1/2}}{B((k-1)/2,1/2)}.$$

If we differentiate this expression we obtain the density of R/S for a > 0:

$$\frac{1}{B((k-1)/2,1/2)}\left(\frac{1}{1+a^2}\right)^{k/2} =... ...{\sqrt{pi}\Gamma((k-1)/2)} \left(\frac{1}{1+a^2}\right)^{k/2}.$$

However, since the density of R/S is symmetric about 0, the preceding formula gives the density of R/S, and, therefore, T(V), for all a. It then follows immediately from the definition of t(V) that the density of t(V) is given by

$$f_{t(V)}(x) = \frac{\Gamma((k/2))}{\sqrt{\pi(k-1)}\Gamma((k... ...\left(1+\frac{x^2}{k-1}\right)^{-k/2}\;\; x\in(-\infty,\infty).$$

This density is called the t density with k-1 degrees of freedom. Notice that as $k\rightarrow\infty$ that f_t(V)(x) converges to the standard normal density. This is one justification for multiplying T(V) by the factor $\sqrt{k-1}$.

More generally, if we have a random sample, V, of size $k \geq 2$ from a normal population with mean $\mu$ and variance $\sigma^2$, then

$$t(V) \equiv \sqrt{k-1}\frac{\mu_{MLE}(V)}{\sigma_{MLE}(V)}$$

has a t distribution with k-1 degrees of freedom.