differentiability

Jacobian

Def (Jacobian)

Let $F : R^{d} \to R^{p}$ be a differentiable vector valued function, and $x \in R^{d}$ some input. Then, $D F (x)$ is called the Jacobian of $F$ at $x$ , and is the matrix:

[D F (x)]_{i j} = \frac{\partial f_{i}}{\partial x_{j}} (x)

where:

$i = 1 \dots p$
$j = 1 \dots d$
$f_{i}$ are the component functions, i.e. $F = (f_{1} (x) \dots f_{p} (x))$

NOTE: $D F (x)$ is a $p \times d$ matrix

Gradient

Def (Gradient)

Let $f : R^{d} \to R$ be a scalar function and $x \in R^{d}$ some input. Then, the Jacobian of this function will be a row vector, i.e. $D F (x) \in R^{1 \times d}$ . The transpose of the Jacobian is called the gradient i.e.

\nabla f (x) = [D F (x)]^{⊤} \in R^{d \times 1}

Hessian

Def (Hessian)

Let $f : R^{d} \to R$ be a twice differentiable function and $x \in R^{d}$ some input. Then the Hessian of $f$ at $x$ is defined as:

[\nabla^{2} f (x)]_{i j} = \frac{\partial^{2} f}{\partial x_{i} \partial x_{j}} (x) i, j = 1 \dots d

NOTE: $\nabla^{2} f (x)$ is a $d \times d$ matrix.