Lecture 32: The Information Inequality

We have seen now that in theory one can construct an uniformly minimum variance unbiased estimator. However, if the details are too messy, it may not be practical to construct this estimator. If this is the case, it would be useful to know a lower bound on the variance of estimators of a given quantity. If the parameter space is one dimensional, we may proceed as follows.

Suppose that we have a regular model with two additional assumptions:

(I):

The set $A \equiv \{\vec{x}:p(\vec{x},\theta) \gt 0\}$ does not depend on the parameter $\theta$, and for all $\vec{x}\in A$ and $\theta\in\Theta$ we have

$$\frac{\partial}{\partial \theta}\log(p(\vec{x},\theta)) \in (-\infty,\infty);$$

(II):

If T is any statistic such that ${\rm E}[\vert T\vert] < \infty$ for all $\theta\in\Theta$ then

$$\frac{\partial}{\partial \theta}\int_{A}T(\vec{x})p(\vec{x},... ...x})\frac{\partial}{\partial \theta} p(\vec{x},\theta)\;d\vec{x}$$

or

$$\frac{\partial}{\partial \theta}\sum_{A}T(\vec{x})p(\vec{x},... ...A}T(\vec{x})\frac{\partial}{\partial \theta} p(\vec{x},\theta).$$

Properties (I) and (II) hold if $p(\cdot,\theta)$ is an exponetial family and $C(\theta)$ has a continuous derivative on $\Theta$ which is never zero.

Theorem 166 (Information Inequality)

Suppose that T is any statistic with finite variance for all $\theta\in\Theta$. Let $\Psi(\theta) = {\rm E}_\theta[T(\vec{X})]$. Then $\Psi$ is differentiable on $\Theta$ and

$${\rm Var}_\theta[T(\vec{X})] \geq \frac{(\Psi'(\theta))^2}{I(\theta)}$$

where

$$I(\theta) = {\rm E}_\theta\left[\left(\frac{\partial}{\partial \theta}\log(p(\vec{X},\theta))\right)^2\right].$$