Lecture 25: Exponential Families of Distributions

We will examine here a class of probability distributions for which the construction of Likelihood Ratio Tests is particularly easy.

We will first consider the case of absolutely continuous distributions. We will assume that we have one unknown parameter $\theta \in \Theta$ upon which our density $f_\theta$ depends, and that $f_{\theta}:R^d\rightarrow[0,\infty)$.(Typically, $f_\theta$ is the joint density for a random sample of size d.) If we can find

• A set $A\subset R^d$;
• A pair of functions $C:\Theta\rightarrow (-\infty,\infty)$ and $D:\Theta\rightarrow(-\infty,\infty)$;
• A pair of functions $T:R^d\rightarrow(-\infty,\infty)$ and $S:R^d\rightarrow (-\infty,\infty)$;

so that we have

$$f_\theta(\vec{x}) = \exp[C(\theta)T(\vec{x})+ D(\theta) + S(\vec{x})]I_A(\vec{x})$$

then we say that $\{f_\theta,\theta\in\Theta\}$ is an exponential family of densities.

In the discrete case, density is replaced by probabilility function, and we say that the set of probability functions $\{p_\theta, \theta\in\Theta\}$ is an exponential family of probability functions if

$$p_\theta(\vec{x}) = \exp[C(\theta)T(\vec{x})+ D(\theta) + S(\vec{x})]I_A(\vec{x}).$$

In each of these cases, I_A represents the indicator of A. This factor is crucial, and is sometimes overlooked. It does not depend on $\theta$.

This definition can be extended to the case where there are multiple parameters. Suppose that $\Theta\subset R^k$. The set of densities $\{f_{\vec{\theta}}, \vec{theta}\in\Theta\}$ is called an exponential family if we can write

$$f_{\vec{\theta}}(\vec{x}) = \exp[C_1(\vec{\theta})T_1(\vec{x... ...\vec{\theta})T_k(\vec{x})+ D(\theta) + S(\vec{x})]I_A(\vec{x}).$$

Of course, we may have any of the C_j or T_k identically . The case of probability mass functions is similar.

Many common distributions are exponential families. More importantly, densities and mass functions for random samples from exponential families are again exponential families.

Here are some examples.

Poisson:

First, take d = 1. Let $A = \{0, 1, \dots\}$. If X has a Poisson distribution with mean $\lambda$ then

$$\Pr(X = x) = \frac{\lambda^x}{x!}\exp(-\lambda)I_A(x) = \exp[x\log(\lambda) - \lambda + \log(x!)]I_A(x).$$

We see that T(x) = x, $C(\lambda) = \lambda$, $D(\lambda) = -\lambda$, and $S(x) = \log(x!)$. We have to take a little care with S here because it technically isn't defined for $x\neq A$. Since $\Pr(X = x) = 0$ for $x\in A^c$, we can define S(x) = 0 for $x\in A^c$.