Definition: For a function $f : R^{d} \to R$ , its gradient is the $d \times 1$ column vector

\nabla f (x) = {[\frac{\partial f}{\partial x_{1}} \dots \frac{\partial f}{\partial x_{d}}]}^{⊤} .

For a function $f : R^{d} \to R^{k}$ , its gradient is the $d \times k$ matrix with $i j$ th element

\nabla f (x)_{i j} = \frac{\partial f_{j} (x)}{\partial x_{i}} .

Identities

\begin{aligned} Inner product: & \nabla_{x} (A^{⊤} x) & = A \\ Quadratic form: & \nabla_{x} (x^{⊤} A^{⊤} x) & = (A + A^{⊤}) x \\ Chain rule: & \nabla_{x} f (y) & = \nabla_{x} y \nabla_{y} f \\ Product rule: & \nabla (f^{⊤} g) & = (\nabla f) g + (\nabla g) f \\ Inverse function theorem: & \nabla_{x} y & = {(\nabla_{y} x)}^{- 1} \end{aligned}

Note that for the inverse function theorem to apply, the gradient must be invertible.

Usage

The above convention (that taking derivatives produces column vectors) is the one most often encountered in the statistics literature. However, this convention is arbitrary and it is also possible to adopt the alternative convention that derivative produce row vectors. Most authors stick to one convention and use the terms “gradient” and “derivative” interchangeably, although some authors reserve “gradient” specifically for the above convention and “derivative” for the alternate convention.