Multivariate Normal (MVN) Distribution

The Multivariate Normal distribution has a prominent role in statistics as a consequence of the Central Limit Theorem. For example, estimates of regression parameters are asymptotically Normal. (Some people prefer to call it a Gaussian distribution).

We will extend the notation of section  4.1 to $p$ dimensions, so E(X) $=\mbox{\boldmath$\mu$}$ is the vector whose components are $E(X_1), \ldots, E(X_p)$ or $\mu_1, \ldots, \mu_p$, and $\mbox{\boldmath$\Sigma$}=$ cov(X) is the variance-covariance matrix ($p \times p$) whose diagonal terms are variances and off-diagonal terms are covariances, and

$$\mbox{\boldmath$\Sigma$}=\mbox{cov}({\bf X})=E[({\bf X}- \mbox{\boldmath$\mu$})({\bf X}-\mbox{\boldmath$\mu$})'].$$

Definition 4..1
The random $p$-vector ${\bf X}$ is said to be multivariate normal if and only if the linear function

$${\bf a}'{\bf X} = a_1X_1 + \dots + a_pX_p$$

is normal for all ${\bf a}$, where ${\bf a}'=(a_1,a_2,\ldots,a_p)$.

In loose statistical jargon, the terms `linear' and `Normal' are sometimes interchangeable. Where we have random variables that are `normal', we can think of the components as additive.

Theorem 4..1

If ${\bf X}$ is $p$-variate normal with mean $\mbox{\boldmath$\mu$}$ and covariance matrix $\Sigma$ (non-singular), then ${\bf X}$ has a pdf given by

$$f_{\bf X}({\bf x})= \frac{1}{(2\pi)^{p/2}\vert\mbox{\boldmat... ...ox{\boldmath$\Sigma$}^{-1}({\bf x} - \mbox{\boldmath$\mu$})}$$ (4.1)

Proof
We are given that E(X) $=\mbox{\boldmath$\mu$}$, E(X $-\mbox{\boldmath$\mu$})({\bf X} - \mbox{\boldmath$\mu$})'= \mbox{\boldmath$\Sigma$}$.

Since $\mbox{\boldmath$\Sigma$}$ is positive definite, there is a non-singular matrix P such that $\mbox{\boldmath$\Sigma$}=$PP$'$. [Chapter 1, sec 1.2, 6(b)(ii).] Consider the transformation ${\bf Y}={\bf P}^{-1}({\bf X}- \mbox{\boldmath$\mu$})$. By Definition 4.1, the components of Y are normal and

$$E({\bf Y})=E({\bf P}^{-1}({\bf X}-\mbox{\boldmath$\mu$}))={\b... ...th$\mu$})={\bf0}\mbox{ since }E({\bf X})=\mbox{\boldmath$\mu$},$$

$$\mbox{cov}({\bf Y})=E({\bf YY}') = {\bf P}^{-1}E[({\bf X}- \m... ...({\bf P}')^{-1}= {\bf P}^{-1}{\bf PP}'({\bf P}')^{-1} ={\bf I},$$

So $Y_1$,...,$Y_p$ are iid $N(0,1)$ and their joint pdf is given by

$$f_{\bf Y}({\bf y})= \frac{1}{(2 \pi)^{p/2}}e^{-\frac{1}{2}{\bf y}'{\bf y}}.$$

Using (3.6), the density of X is

$$f_{\bf X}({\bf x})=f_{\bf Y}\left({\bf P}^{-1}({\bf x}-\mbox{\boldmath$\mu$})\right)\mbox{abs}\vert{\bf P}^{-1}\vert,$$

where

$$\vert{\bf P}^{-1}\vert=\frac{1}{\vert{\bf P}\vert}=\frac{1}{\... ...vert^{1/2}}= \frac{1}{\vert\mbox{\boldmath$\Sigma$}\vert^{1/2}}$$

and the result follows.

Comments

1. Note that the transformation $\bf {Y}={\bf P}^{-1}({\bf X}-\mbox{ \boldmath$\mu$})$ is used to standardize ${\bf X}$, in the same way as $Z=\frac{X-\mu}{\sigma}$ was used in univariate theory.
2. Note that when $p=1$, (4.1) reduces to the pdf of the univariate normal.
3. The covariance matrix is symmetric, since $\mbox{cov}(X_i,\,X_j)=\mbox{cov}(X_j,\,X_i)$.
4. It is often convenient to write ${\bf X} \sim N_p(\mbox{\boldmath$\mu$}, \mbox{\boldmath$\Sigma$})$.
5. Note that
   

   $$\int^\infty_{-\infty}\dots \int^\infty_{-\infty}\frac{1}{(2... ...1\dots dx_p = \vert\mbox{\boldmath$\Sigma$}\vert^{\frac 12}.$$ (4.2)