The Exponential Family of Distributions

[Read HC 7.5 where we will use $B(\theta)$ for their $e^{q(\theta)}$ and $h(x)$ for their $e^{S(x)}$.]

Definition 7..4
The exponential family of distributions is a one-parameter family that can be written in the form

$$f(x;\theta)=B(\theta)h(x)e^{[p(\theta)K(x)]}, \, a< x < b,$$ (7.3)

where $\gamma<\theta<\delta$. If, in addition,

(a)

neither $a$ nor $b$ depends on $\theta$,

(b)

$p(\theta)$ is a non-trivial continuous function of $\theta$,

(c)

each of $K'(x) \not\equiv 0$ and $h(x)$ is a continuous function of $x$, $a<x<b$,

we say that we have a regular case of the exponential family.
Most of the well-known distributions can be put into this form, for example, binomial, Poisson, geometric, gamma and normal. The joint density function of a random sample X from such a distribution can be written as

$$f({\bf x};\theta)=B^n(\theta)\prod_{i=1}^nh(x_i) e^{p(\theta)\sum_{i=1}^nK(x_i)}, \, a<x_i<b.$$ (7.4)

Putting

$$T=t({\bf X})=\sum_{i=1}^nK(X_i) \mbox{ and }t({\bf x})=\sum_{i=1}^nK(x_i),$$

we see that $f({\bf x};\theta)$ can be written as

$$\left[B^n(\theta)e^{p(\theta)t(x)}\right]\left[\prod_{i=1}^nh(x_i)\right]= g(t({\bf x};\theta)h({\bf x}),$$

so that Theorem 1.1 applies and t(X) is a sufficient statistic for $\theta$.

Example 7..6
Let X $\sim$ U[0,$\theta$]. Then $f(x)=1/\theta$, x $\in$ [0,$\theta$]. We see that f(x) cannot be written in the form (1.1). We could write B($\theta$)=1/$\theta$, p($\theta$)$=0$, but then we would need

$$h(x)=\left\{\begin{array}{ll} 1, &0 \leq x \leq \theta\\ 0 & \mbox{ otherwise} \end{array} \right.$$

which makes h(x) depend on $\theta$ and the condition of (1.1) would not be satisfied.
[We already know that $\max X_i$ is sufficient for $\theta$ here, and note that $\max X_i$ is not of the form $\sum_{i=1}^nK(X_i)$.]

Example 7..7
Consider the normal distribution with mean $\theta$ and variance $1$. The density function can be written in the form (1.1) where

$$\frac{1}{\sqrt{2 \pi}}e^{-(x-\theta)^2/2}= \underbrace{{\frac... ...2}}}_{B(\theta)}.\underbrace{(e^{-x^2/2})}_{h(x)}.e^{\theta x},$$

and $p(\theta)=\theta, K(x)=x$. So $T=\sum K(X_i)=\sum X_i$ is minimal sufficient for $\theta$.

Note that we could have defined $p(\theta)=n\theta$ and $K(x)=x/n$, so that $T=\sum X_i/n=\overline{X}$ is also sufficient for $\theta$.

A distribution from the exponential family arises from tilting a simple density ,

$$\begin{eqnarray*} f({\bf x};\theta) & = & f(x) \times e^{\theta x - K(\theta)} ... ...eta x}) \\ \mu & = & K'(\theta) \\ \sigma^2 & =& K''(\theta) \end{eqnarray*}$$

and $\theta$ is termed the natural parameter.