This article describes the row-orientation or “derivative” convention; see here for the column-orientation or “gradient” convention most often used in statistics.

Definition: For a function $f : R^{d} \to R$ , its derivative is the $1 \times d$ row vector

\dot{f} (x) = [\frac{\partial f}{\partial x_{1}} \dots \frac{\partial f}{\partial x_{d}}] .

For a function $f : R^{d} \to R^{k}$ , its derivative is the $k \times d$ matrix with $i j$ th element

\dot{f} (x)_{i j} = \frac{\partial f_{i} (x)}{\partial x_{j}} .

Identities

\begin{aligned} Inner product: & D_{x} (A x) & = A \\ Quadratic form: & D_{x} (x^{⊤} A^{⊤} x) & = x^{⊤} (A + A^{⊤}) \\ Chain rule: & D_{x} f (y) & = D_{y} f D_{x} y \\ Product rule: & D (f^{⊤} g) & = g^{⊤} \dot{f} + f^{⊤} \dot{g} \\ Inverse function theorem: & D_{x} y & = {(D_{y} x)}^{- 1} \end{aligned}

Note that for the inverse function theorem to apply, the derivative must be invertible

Relation to gradient form

The derivatives given above are the transposes of the gradients defined here:

\nabla f (x) = \dot{f} (x)^{⊤} .

Most authors stick to one convention and use the terms “gradient” and “derivative” interchangeably, although some authors reserve “derivative” specifically for the above convention and “gradient” for the column-oriented convention.