Lecture 11: Random Samples

A random sample of size n is a vector valued random variable $V = (V_1,\dots, V_n)$ where

• The component random variables $V_1,\dots V_n$ are mutually independent;
• The component random variables $V_1,\dots V_n$ have the same distribution.

Usually these two properties are lumped together and referred to as independent and identically distributed, which is abbreviated as iid.

If V is a random sample of size n, it is common to refer to the distribution of the components as the common distribution, as it is the distribution they have in common. At other times you will hear it referred to as the marginal distribution. They are one in the same thing, as a random variable can have only one distribution function.

Suppose that V is a random sample of size n and the common distribution has distribution function F and density f. Then the distribution function of V at $(t_1,\dots,t_n)$ is given by

$$F_V(t_1,\dots,t_n) = \Pr(V_1\leq t_1,\dots,V_n\leq t_n) = \prod_{j=1}^n\Pr(V_j\leq t_j) = \prod_{j=1}^nF(t_j)$$

and the density function of V is (by repeated integration)

$$f_V(x_1,\dots,x_n) = \prod_{j=1}^nf(x_j)$$

Another example of Maximum Likelihood Estimation

If V is a random sample of size N where the common distribution has a density f which depends on a parameter $\theta$, one common way to estimate $\theta$is to try to maximize the density function f_V as a function of $\theta$.Here is a standard example.

Suppose that the common density is normal, that is

$$\Pr(V_i \leq t) = \int_{-\infty}^t \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\;dx.$$

where $\mu$ is any real number, and $\sigma \gt 0$. The maximum likelihood estimate of $(\mu,\sigma)$ is the ordered pair which maximizes

$$f_V(x_1,\dots,x_N) = \prod_{j=1}^N \frac{1}{\sqrt{2\pi\sigma... ...-N\log\sigma -\sum_{j=1}^N\frac{(x_j-\mu)^2}{2\sigma^2}\right).$$

This last expression is clearly maximized at the value of $(\mu,\sigma)$ where

$$L(\mu,\sigma)\equiv -N\log\sigma -\sum_{j=1}^N\frac{(x_j-\mu)^2}{2\sigma^2}$$

is maximized. The gradient of L is

$$\left[\frac{\partial L}{\partial \mu},\frac{\partial L}{\par... ...rac{N}{\sigma}+\frac{1}{\sigma^3}\sum_{j=1}^N(x_j-\mu)^2\right]$$

which is [0,0] when

$$\mu = \frac{1}{N}\sum_{j=1}^N x_j \equiv \overline{x}_N$$

and

$$\sigma^2 = \frac{1}{N}\sum_{j=1}^N (x_j-\overline{x}_N)^2 \equiv \sigma^2_{MLE}$$

With a little extra work we can show that this indeed gives the maximum of L, and, therefore, the maximum of $f_V(x_1,\dots,x_N)$.

We may rewrite $\sigma^2_{MLE}$ as follows:

$$\frac{1}{N}\sum_{j=1}^N (x_j-\overline{x}_N)^2 = \frac{1}{N... ... \left(\frac{1}{N}\sum_{j=1}^N x_j^2\right) -(\overline{x}_N)^2$$