The variance of the score is called the Fisher information:

I (θ) = V u (θ | X) .

On its surface, this would seem to have nothing to do with information. However, the connection between the variance of the score and the curvature of the log-likelihood is made clear in the following theorem.

Theorem: If the likelihood allows all second-order partial derivatives to be passed under the integral sign, then

I (θ^{*}) = E I (θ^{*} | X) .

Proof: Letting $P$ denote the distribution function and $p$ the density function,

\begin{aligned} E I (θ^{*} | X) & = - \int \nabla_{θ}^{2} \log p (x | θ^{*}) d P (x | θ^{*}) \\ = - \int \nabla_{θ} \frac{\nabla_{θ} p (x | θ^{*})}{p (x | θ^{*})} d P (x | θ^{*}) & CR; \nabla \log x = 1 / x \\ = - \int \frac{\nabla_{θ}^{2} p (x | θ^{*}) p (x | θ^{*}) - \nabla_{θ} p (x | θ^{*}) \nabla_{θ} p (x | θ^{*})^{⊤}}{p (x | θ^{*})^{2}} d P (x | θ^{*}) & Chain rule \\ = - \nabla_{θ}^{2} \int d P + \int (\frac{\nabla_{θ} p (x | θ^{*})}{p (x | θ^{*})}) {(\frac{\nabla_{θ} p (x | θ^{*})}{p (x | θ^{*})})}^{⊤} d P (x | θ^{*}) & If \nabla_{θ}^{2} can be passed \\ = - \nabla_{θ}^{2} \int d P + \int u u^{⊤} d P & Def score \\ = \int u u^{⊤} d P & \int d P = 1 \\ = V u & E u = 0 \end{aligned}

In the final step, note that $\nabla_{θ}^{2}$ passable $⟹ \nabla_{θ}$ is passable.

Multiple observations

The above definition and proof assumes that we have a single observation. In cases where we have repeated observations $X_{1}, \dots, X_{n}$ , the symbol $I (θ)$ is unchanged: it still represents the expected information from a single observation. To indicate the information for the entire sample, we use the symbols $I_{n} (θ)$ and $I_{n} (θ)$ .

If observations are iid, then each observation has the same Fisher information and $E I_{n} = n I = I_{n}$ . If observations are independent, but not necessarily identically distributed, then $I (θ)$ requires subscripts and

E I_{n} = \sum_{i = 1}^{n} I_{i} = I_{n} .

If observations are not independent, then the Fisher information is more difficult to calculate as it involves the multivariate (joint) distribution of $x$ .