Independence of Quadratic Forms

We will consider here some useful results involving quadratic forms in normal random variables.

Theorem 4..3

Suppose $X_1,\,X_2,\,\dots ,\,X_p$ are identically and independently distributed as $N(0,1)$ and let ${\bf X}'=(X_1,\,X_2,\,\dots ,\,X_p)$. Define $Q_1$ and $Q_2$ by

$$Q_1 = {\bf X}'{\bf BX}, \ \ Q_2 = {\bf X}'{\bf CX},$$

where ${\bf B}$ and ${\bf C}$ are $p \times p$ symmetric matrices with ranks less than or equal to $p$. Then $Q_1$ and $Q_2$ are independent if and only if ${\bf BC} = {\bf0}$.

Proof
Firstly note that ${\bf X}'{\bf BX}$ and ${\bf X}'{\bf CX}$ are scalars so that $Q_1$ and $Q_2$ each have univariate distributions. We will find the joint mgf of $Q_1$ and $Q_2$. Note that the pdf of ${\bf X}$ is given by (4.1) with $\mbox{\boldmath$\mu$} ={\bf0}$ and $\mbox{\boldmath$\Sigma$} = {\bf I}$, so we have

$$\begin{eqnarray*} % latex2html id marker 3358M_{Q_1,Q_2}(t_1,\,t_2) & = & E(e... ... C}\vert^{-\frac 12} \ , \ \ \mbox{using (\ref{eq:multint}),} \end{eqnarray*}$$

for values of $t_1,\,t_2$ which make ${\bf I}-2t_1{\bf B}-2t_2{\bf C}$ positive definite. Now the mgf's of the marginal distributions of $Q_1$ and $Q_2$ are $M_{Q_1,Q_2}(t_1,0), \ M_{Q_1,Q_2}(0,t_2)$ respectively. That is,

$$\begin{eqnarray*} M_{Q_1}(t_1) & = & \vert{\bf I}-2t_1{\bf B}\vert^{-\frac 12},\\ M_{Q_2}(t_2) & = & \vert{\bf I}-2t_2{\bf C}\vert^{-\frac 12}. \end{eqnarray*}$$

Now $Q_1$ and $Q_2$ are independent if and only if

$$M_{Q_1,Q_2}(t_1,t_2)=M_{Q_1}(t_1)M_{Q_2}(t_2) \ .$$

That is, if

$$\begin{eqnarray*} \vert{\bf I}-2t_1 {\bf B} -2t_2{\bf C}\vert & = & \vert{\bf I... ... = & \vert{\bf I}-2t_1{\bf B}-2t_2{\bf C}+4t_1t_2{\bf BC}\vert. \end{eqnarray*}$$

This is true if and only if ${\bf BC} = {\bf0}$.

[Note that the ${\bf0}$ here is a $p \times p$ matrix with every entry zero.]

The matrices $B,\ C$ are projection matrices. Q1 is the shadow of $(X'X)$ in the $B$ plane and $Q_2$ is the shadow of $X'X$ in the $C$ plane. $Q_1$ and $Q_2$ will be independent if $B \perp C$ since in that case, none of the information in $Q_1$ is contained in $Q_2$.

Example 4..1

If $X_1,\,X_2,\,\dots ,\,X_p$ are iid $N(0,1)$ random variables and $\overline{X}$ and $S^2$ are defined by

$$\begin{eqnarray*} \overline{X} & = & \sum^{p}_{i=1} X_i/p\\ S^2 & = & \sum^p_{i=1} (X_i - \overline{X})^2/(p-1), \end{eqnarray*}$$

show that $S^2$ and $p\overline{X}^2$ are independent.

We need to write both $S^2$ and $p\overline{X}^2$ as quadratic forms. It is easy to verify that $(p-1)S^2={\bf X}'{\bf BX}$ where

$${\bf B} = \left[ \begin{array}{cccc} 1-\frac 1p & -\frac 1p ... ...-\frac 1p & -\frac 1p & \dots & 1-\frac 1p \end{array}\right]$$

and that $p\overline{X}^2={\bf X}'{\bf CX}$ where

$${\bf C} = \left[ \begin{array}{cccc} \frac 1p & \frac 1p & \... ... \frac 1p & \frac 1p & \dots & \frac 1p \end{array} \right]$$

and that ${\bf BC}={\bf CB} = {\bf0}$.