Lecture 18: Laws of Large Numbers

Recall that if V^(k) is a random sample of size k from a population with distribution function F, and $\theta$ is a constant, we say that the statistic T(V^(k)) is unbiased for $\theta$ if

$$E[T(V^{(k)})] = \theta.$$

We have seen, for example, that the sample mean is unbiased for the mean of the population distribution. While it is comforting to know that a statistic is unbiased, it does not tell the whole story, since we don't observe the mean of the statistic, but only the statistic itself. We would like to know in what sense a sequence of statistics which are unbiased for $\theta$ converge to $\theta$ as the sample size goes to infinity.

We have already proved the following result:
Theorem: Suppose that $R_1, R_2, \dots$ is an infinite sequence of independent and identically distributed random variables with mean $\mu$ and variance $\sigma^2$. Then for each x > 0,

$$\Pr\left(\left\vert\frac{R_1 + \cdots + R_n}{n} - \mu \right\vert \gt x\right) \leq \frac{\sigma^2}{nx^2}.$$

In particular, for each x > 0,

$$\lim_{n\rightarrow\infty}\Pr\left(\left\vert\frac{R_1 + \cdots + R_n}{n} - \mu \right\vert \gt x\right) = 0.$$

This, too, is good news, but not the whole story, since it does not tell us directly the behaviour of the sample mean, but rather the behaviour of its distribution function.

Recall too, that once $\omega\in\Omega$ is chosen, $R_1(\omega), R_2(\omega),\dots$ is just a sequence of real numbers. For each such sequence we want to know whether the sequence

$$\frac{R_1(\omega) + \cdots + R_n(\omega)}{n}$$

converges to $\mu$. In particular, if $\Omega_g$ represents the set of $\omega$ where convergence occurs, we want to know $\Pr(\Omega_g)$.

Observe that

$$\lim_{n\rightarrow\infty}\frac{R_1(\omega) + \cdots + R_n(\omega)}{n} = \mu$$

is equivalent to

$$\lim_{n\rightarrow\infty}\left\vert\frac{R_1(\omega) + \cdots + R_n(\omega)}{n} -\mu \right\vert = 0.$$

Thus convergence fails to occur if for every x > 0 there are infinitely many positive integers N so that

$$\left\vert\frac{R_1(\omega) + \cdots + R_N(\omega)}{N} -\mu \right\vert \geq x.$$

Fix now some x > 0 and let

$$A_N = \left\{\omega: \left\vert\frac{R_1(\omega) + \cdots + R_N(\omega)}{N} -\mu \right\vert \geq x\right\}.$$

What we would like to do is to estimate $\Pr(A_k \cup A_{k+1} \cdots)$ for any positive integer k. We can start with

$$\Pr(A_k \cup A_{k+1} \cdots) \leq \sum_{N=k}^\infty\Pr(A_N)$$

If we apply Chebychev's inequality, we get nothing useful, since it would tell us that

$$\Pr(A_N) \leq \frac{\sigma^2}{x^2N}$$

and the righthand side of this inequality is not an summable sequence! Back to the drawing board, or more accurately, back to Markov's inequality.