Non-probabilistic version
Theorem (Jensen’s inequality): For \(\a,\x \in \real^d\) with \(a_i > 0\) for all \(i\), if \(g\) is a convex function, then
\[g\left( \frac{\sum_i a_i x_i}{\sum_i a_i} \right) \leq \frac{\sum\nolimits_i a_i g(x_i)}{\sum_i a_i}\]Proof: The Cauchy-Schwarz Master Class (2004), Steele JM. Cambridge University Press.
The inequality is reversed if \(g\) is concave.
Probabilistic version
Theorem: If \(g\) is a convex function defined over an interval \(I\), and \(X\) is a random variable with \(\Pr(X \in I) = 1\) and finite expectation, then \(g(\Ex(X)) \leq \Ex(g(X)).\) If \(g\) is strictly convex, the inequality is strict unless \(X\) is a constant with probability 1.
Proof: Since \(\Ex(X)\) is finite and \(\Pr(X \in I) = 1\), we have \(\Ex(X) \in I\). Now, consider the point \(\{\Ex(X), g(\Ex(X))\}\). Since \(g\) is convex, there exists a line \(L(x)\) tangent to this point such that \(L(x) \leq g(x)\) for all \(x \in I\). Then
\[\begin{align*} \Ex\{g(X)\} &\geq \Ex\{L(X)\} \\ &= L(\Ex(X)) \\ &= g(\Ex(X)), \end{align*}\]with the inequality becoming strict if convexity is strict, unless \(X\) is a constant with probability 1.
The inequality is reversed if \(g\) is concave.