The Bernstein-von Mises Theorem is the name given to a variety of results that are essentially Bayesian versions of the central limit theorem. Two such results are presented below.
Posterior mean
The following regularity conditions are involved in the theorem:
- Standard likelihood regularity conditions (A), (B), and (C) are met.
-
For all $\eps > 0$, there exist $\delta > 0$ such that in the expansion
\[\as{ \ell(\bt) = \, &\ell(\bt^*) + (\bt - \bts) \Tr \u(\bts) - \\ &\tfrac{1}{2}(\bt - \bts) \Tr [\oI(\bts) + \R(\bts)] (\bt - \bts),}\]the probability of the event
\[\sup_{\norm{\bt - \bts} < \delta} \abs{\tfrac{1}{n}R_{ij}(\bt)} > \eps\]tends to 0 as $n \to \infty$ for all $i$ and $j$.
-
For all $\eps > 0$, there exist $\delta > 0$ such that the probability of the event
\[\sup_{\norm{\bt - \bts} \ge \delta} \tfrac{1}{n}\{\ell(\bt) - \ell(\bts)\} < -\eps\]tends to 1 as $n \to \infty$.
- The prior density $p(\bt)$ is continuous and positive for all $\bt \in \bT$.
- The prior expectation exists: $\int \norm{\bt} dP(\bt) < \infty$.
Theorem (Bernstein-von Mises): Let $\bth$ denote the posterior mean. If (1)-(5) hold, then
\[\sqrt{n}(\bth-\bts) \inD \Norm(\zero, \fI^{-1}(\bts)).\]Proof: See Lehmann’s Theory of Point Estimation (Theorem 8.3), which he attributes to Peter Bickel.
Posterior density
A central limit theorem-type result also hols for the posterior density and states that the posterior density of $\bd=\sqrt{n}(\bt-\bth)$ converges to the $\Norm(\zero, \fI(\bts)^{-1})$ density.
Theorem (Bernstein-von Mises): Suppose $p(\bt)$ is continuous with $p(\bt)>0$ for all $\bt \in \bT$. Under regularity conditions (A)-(D),
\[p(\bth + \bd/\sqrt{n}|\x) / p(\bth|\x) \inAS \exp\{-\tfrac{1}{2}\bd\Tr\fI(\bts)\bd\}.\]If, in addition,
\[\int p(\bth + \bd/\sqrt{n}|\x) / p(\bth|\x) \,d\bd \inAS \int \exp\{-\tfrac{1}{2}\bd\Tr\fI(\bts)\bd\} \,d\bd,\]then
\[\int\abs{p(\bd|\x) - \phi(\bd)}\,d\bd \inAS 0,\]where $\phi(\cdot)$ is the $\Norm(\zero, \fI(\bts)^{-1})$ density.