Theorem: Suppose $\y = \X\bb^* + \w$, where $w_i \iid (0, \sigma^2)$. Suppose $\tfrac{1}{n} \X \Tr \X \to \bS$, where $\bS$ is positive definite, and let $\x_i$ denote the $d \times 1$ vector of covariates for subject $i$ (taken to be fixed, not random). If $\norm{\x_i}$ is uniformly bounded, then
\[\tfrac{1}{\sigma}(\X \Tr \X)^{1/2}(\bbh - \bb^*) \inD \Norm(\zero, \I).\]Proof:
I’ll split this theorem up into three parts:
- Set up the triangular array
- Apply the Lindeberg-Feller CLT
- Relate this result back to linear regression
Part 1: Set up the triangular array
The triangular array we will consider is $\x_i W_i$, where $\x_i$ is a fixed vector and $W_i$ a random scalar. Note that this satisfies the definition of a triangular array: (i) Independent since $W_i \ind W_j$, (ii) $\Ex \x_i W_i = \zero$, and (iii) $\Var \x_i W_i = \sigma^2 \x_i \x_i \Tr$, which is finite.
Expressing the row-wise means and average variances into standard regression notation with $\X$ representing the design matrix ($\x_i$ is the $i$th row of $\X$), we have
\[\begin{aligned} \z_n &= \frac{1}{n} \sum_i \x_i W_i = \frac{1}{n} \X \Tr \w \\ \V_n &= \frac{1}{n} \sum_i \Var (\x_i W_i) = \frac{1}{n} \sum_i \sigma^2 \x_i \x_i \Tr = \frac{\sigma^2}{n} \X \Tr \X \to \sigma^2 \Sigma \end{aligned}\]Part 2: Apply the Lindeberg-Feller CLT
We’ve already established that $\x_i W_i$ forms a triangular array, so the only other thing we need is to prove the Lindeberg condition. Letting $\eps > 0$ and noting that there exists an upper bound $M: \norm{\x_i} < M$ for all $i$,
\[\begin{alignat*}{2} \frac{1}{n} \sum_i &\Ex \{ \norm{\x_i W_i}^2 1(\norm{\x_i W_i} \ge \eps \sqrt{n}) \} \\ &= \frac{1}{n} \sum_i \Ex \{ W_i^2 \norm{\x_i}^2 1(\abs{W_i}\norm{\x_i} \ge \eps \sqrt{n}) \} &\hspace{8em}& \href{norm.html}{\text{Homogeneity}} \\ &< \frac{1}{n} \sum_i \Ex \{ W_i^2 M^2 1(\abs{W_i}M \ge \eps \sqrt{n}) \} && \textnormal{$\norm{x_i}$ bounded by $M$} \\ &= \Ex \{ W^2 M^2 1(\abs{W}M \ge \eps \sqrt{n}) \} && \textnormal{$W_i$ iid} \\ &= \Ex T_n && \textnormal{Defining $T_n$ here} \\ T_n &\inP 0 && \eps \sqrt{n}/M \to \infty \\ \Ex T_n &\to 0 && \href{dominated-convergence-theorem.html}{\text{DCT}} \textnormal{ with } Z = M^2 W^2; \Ex Z = M^2 \sigma^2 < \infty \end{alignat*}\]Thus, by the Lindeberg-Feller CLT, we have
\[\frac{1}{\sqrt{n}} \X \Tr \w \inD \Norm(\zero, \sigma^2 \Sigma)\]or
\[\frac{1}{\sigma} (\X \Tr \X)^{-1/2} \X \Tr \w \inD \Norm(\zero, \I).\]Part 3: Relate this result back to linear regression
The least squares regression estimator is
\[\begin{aligned}\bbh &= (\X \Tr \X)^{-1}\X \Tr \y \\ &= (\X \Tr \X)^{-1}\X \Tr (\X \bb^* + \w) \\ &= \bb^* + (\X \Tr \X)^{-1} \X \Tr \w \end{aligned}\]Thus,
\[\begin{aligned} \tfrac{1}{\sigma}(\X \Tr \X)^{1/2}(\bbh - \bb^*) &= \tfrac{1}{\sigma}(\X \Tr \X)^{-1/2} \X \Tr \w \\ &\inD \Norm(\zero, \I) \end{aligned}\]by our conclusion at the end of Part 2.