Lecture 17: Limiting Distributions

We have seen that the distribution of a statistics based on a random sample of size k depends on k. In some cases we have been fortunate to be able to compute the distribution of the statistic in terms of k. As we have noted, these are rare circumstances indeed, and frequently depended on very delicate calculations. Alas, in most cases such calculations are not feasible. However, frequently we can make approximations to the exact distributions. These approximations frequently take the following form: Suppose that F_k is the distribution of the statistic based on a random sample of size k. Then there is a distribution function F so that for every x,

$$\lim_{k\rightarrow\infty}F_k(x) = F(x)$$

You should be somewhat concerned that, as we have put it, there is no description of how large k should be to guarantee that F_k(x) is ``close enough'' to F(x). On the other hand we cannot do everything at once. As an analogy, remember that in calculus, we content for quite a while to say that the fact that

$$\lim_{\theta\rightarrow 0}\frac{\sin(\theta)}{\theta} = 1$$

was good enough to say that $sin(\theta) \approx \theta$ for small $\theta$before deriving Taylor's theorem to quantify ``$\approx$'' and ``small''.

The most famous of these theorems is the Central Limit Theorem. It says:
Central Limit Theorem: Suppose that F is a distribution with finite mean $\mu$ and finite variance $\sigma^2$. Let V^(k) be a random sample of size k from the distribution F. Let x be any real number. Then  

$$\lim_{k\rightarrow\infty} \Pr\left(\frac{(V_1^{(k)} + \cdots... ...\right) = \int_{-\infty}^x \frac{\exp(-u^2/2)}{\sqrt{2\pi}}\;du$$ (1)

Since the sum of expectations is the expectation of the sum, and for random samples, the variance of the sum is the sum of the variances, we can rewrite (1) is a couple of ways:  

$$\lim_{k\rightarrow\infty} \Pr\left(\frac{(V_1^{(k)} + \cdots... ...\right) = \int_{-\infty}^x \frac{\exp(-u^2/2)}{\sqrt{2\pi}}\;du$$ (2)

and  

$$\lim_{k\rightarrow\infty} \Pr\left( \frac{\sqrt{k}}{\sigma} ... ...right) = \int_{-\infty}^x \frac{\exp(-u^2/2)}{\sqrt{2\pi}}\;du.$$ (3)

It is important to realize that (1), (2) and (3) say exactly the same thing, and they are derivable from one another by algebra alone.

We have seen various examples in class and on the homework which suggest that the Central Limit Theorem is true. We will give some further evidence a little while later, and give a proof in the very special case where the random samples are from binomial distributions with p = 1/2 and N=1. This turns out to be a very important case both for theoretical and practical reasons.

At this juncture it would be good to point out that we do have an error approximation theorem, akin to the error term in Taylor's theorem:
The Berry-Esseen Theorem: Suppose that $R_1, \dots, R_k$ are independent identically distributed random variables with

• $E[R_1] = \mu$;
• $E[(R_1-\mu)^2] = \sigma^2$;
• $E[\vert R_1-\mu\vert^3] = \rho$.

Let x be any real number. Then  

$$\left\vert\Pr\left(\sqrt{k}\frac{(V_1^{(k)} + \cdots + V_k^{... ...sqrt{2\pi}}\;du.\right\vert \leq \frac{3\rho}{\sigma^3\sqrt{k}}$$ (4)

Proofs of the Central Limit Theorem and the Berry-Esseen Theorem may be found in advanced texts on probability theory, such as An Introduction to Probability Theory and Its Applications, Volume II by William Feller.

An example

Suppose that we are sampling from the continuous uniform distribution on [-1, 1]. Then

$$\mu = \int_{-1}^1 u (1/2)\;du = 0,$$

$$\sigma^2 = \int_{-1}^1 (u-0)^2 (1/2)\;du = 1/3;$$

and

$$\rho = \int_{-1}^1 \vert u-0\vert^3 (1/2)\;du = \int_0^1 u^3\;du = 1/4;$$

and the Berry-Esseen Theorem says  

$$\left\vert\Pr\left(\sqrt{k}\frac{(V_1^{(k)} + \cdots + V_k^{... ...2\pi}}\;du.\right\vert \leq \frac{3(1/4)}{(1/3)^{3/2}\sqrt{k}}.$$ (5)

Suppose now, for the sake of convenience, that k = 100. Since

$$\Pr\left(\frac{V_1 + \cdots + V_{100}}{100} \in [0,1/2]\righ... ...\left[0,\frac{\sqrt{100}}{\sqrt{1/3}}\frac{1}{2}\right]\right),$$

it follows from the central limit theorem, in the form of (3), that following approximation:

$$\Pr\left(\frac{V_1 + \cdots + V_{100}}{100} \in [0,1/2]\righ... ...\approx \int_0^{5\sqrt{3}}\frac{1}{\sqrt{2\pi}}\exp(-u^2/2)\;du$$

There are many other limiting distribution theorems. We have observed that as the degrees of freedom goes to infinity, the t-distribution converges to the standard normal distribution as well.

Here is one more example. Suppose that V^(k) is a random sample of size k from a distribution which is concentrated on the positive real numbers. Now let the the statistic T(V) = max(V₁^(k),...,V_k^(k)). We have already seen that if the commond distribution function for these random variables is F that

$$\Pr(T(V^{(k)}) \leq x) = F(x)^k.$$

We immediately see that if F(x) < 1 then

$$\lim_{k\rightarrow\infty}\Pr(T(V^{(k)}) \leq x) = \lim_{k\rightarrow\infty}F(x)^k = 0.$$

The problem is to find a sequence a_k and a distribution function G so that

$$\lim_{k\rightarrow\infty}\Pr\left(\frac{T(V^{(k)})}{a_k} \leq x\right) = \lim_{k\rightarrow\infty}F(a_kx)^k = G(x).$$

It is difficult to answer this question in general, so we will content ourselves with an examples.

First suppose that F(x) = 1 - (1/x), $x\geq 1$ and 0 otherwise. Then if x > 1/a_k, we would have

$$F(a_kx)^k = \left(1 - \frac{1}{a_kx}\right)^k$$

which suggests that we should take a_k = k to get for all x > 0

$$\lim_{k\rightarrow\infty}F(kx)^k = \left(1 - \frac{1}{kx}\right)^k = \exp(-1/x).$$

It is straightforward to check that if we define $G(x) = \exp(-1/x)$ for x > 0 and 0 otherwise that G is a distribution function.

In the case of a sample from an exponential distribution rescaling is not sufficient. In this case we might have $F(x) = 1-\exp(-x)$ for x > 0 and 0 otherwise. Then we can easily check that for $x \gt -\log(k)$ we have

$$\Pr(T(V^{(k)}) \gt x + \log(k)) = \left(1 - \frac{\exp(-x)}{k}\right)^k$$

so for any $x\in (-\infty,\infty)$,

$$\lim_{k\rightarrow\infty}\Pr(T(V^{(k)}) \gt x + \log(k)) = \exp(-\exp(-x))$$

and $G(x) = \exp(-\exp(-x))$ is a distribtution function.