Gradient

2.3 Gradient

The derivative of function in inner-product space

Theorem 2.3.1 (Riesz representative theorem) Assuming $L : R^{m} \to R$ is a linear function, $⟨ \cdot, \cdot ⟩$ is the inner product on $R^{m}$ , then there exists a unique $v_{L} \in R^{m}$ such that $L (h) = ⟨ h, v_{L} ⟩$ , for any $h \in R^{m}$ .

Proof If $L = 0$ , then take $v_{L} = 0$ , it is unique.
When $L \neq 0$ , consider $\ker L = {h \in R^{m} | L (h) = 0} ⊊ R^{m}$ . Take $u_{1} \in (\ker L)^{⊥}$ and $∥ u_{1} ∥ = 1$ , then for $h \in R^{m}$ , there exists a unique $λ \in R$ such that

L (h) L (h − λu1 ) = 0 ⇔ λ = ------ L(u1)

Here $\begin{aligned} h - λ u_{1} & = h - \frac{L (h)}{L (u_{1})} u_{1} \in \ker L \\ 0 & = ⟨ h - λ u_{1}, u_{1} ⟩ = ⟨ h, u_{1} ⟩ - λ ⟨ u_{1}, u_{1} ⟩ \\ ⟨ h, u_{1} ⟩ & = λ = \frac{L (h)}{L (u_{1})} \\ L (h) & = ⟨ h, L (u_{1}) u_{1} ⟩ \end{aligned}$

Therefore, $v_{1} = L (u_{1}) u_{1}$ . The uniqueness is proved via contradiction. $◻$

v_{L}

is called the gradient of

L

. Given

f : R^{m} \to R

, the gradient of

d f (x_{0})

is called the gradient of

f

x_{0}

, marked as

grad f (x_{0})

\nabla f (x_{0})

, i.e.

df(x0 )(v ) = ⟨v, ∇f (x0)⟩

Example 2.3.1 Several examples for the gradient.

1.

$A : R^{m} \to R^{m}$ is linear, $f (x) = ⟨ A x, x ⟩$ , seek $\nabla f (x_{0})$ . $\begin{aligned} f (x_{0} + v) - f (x_{0}) & = ⟨ A (x_{0} + v), (x_{0} + v) ⟩ - ⟨ A x_{0}, x_{0} ⟩ \\ = \underset{d f (x_{0}) (v)}{\underset{⏟}{⟨ A x_{0}, v ⟩ + ⟨ A v, x_{0} ⟩}} + ⟨ A v, v ⟩ \\ d f (x_{0}) (v) & = ⟨ A x_{0}, v ⟩ + ⟨ A v, x_{0} ⟩ \\ = ⟨ A x_{0}, v ⟩ + ⟨ v, A^{T} x_{0} ⟩ = ⟨ v, (A + A^{T}) x_{0} ⟩ \\ \nabla f (x_{0}) & = (A + A^{T}) x_{0} \end{aligned}$

2.

$det : M_{n} \to R$ , define the inner product between matrices as

T ⟨A, B ⟩ = tr(A B ), ∀A, B ∈ ℳn

Therefore

ddet(A )(B) = trA ∗TB = ⟨B, A∗⟩

meaning $\nabla det (A) = A^{*}$ .

Assuming

v_{1}, . . ., v_{m}

is a set of bases in

R^{m}

, let

x = x_{1} v_{1} + . . . + x_{m} v_{m}

v = ξ_{1} v_{1} + . . . + ξ_{m} v_{m}

, then the form of

\nabla f (x_{0})

under coordinate

(x_{1}, . . ., x_{m})

could be expressed as

\begin{aligned} \nabla f (x_{0}) & = c_{1} v_{1} + . . . + c_{m} v_{m} \\ d f (x_{0}) (v) & = ⟨ v, \nabla f (x_{0}) ⟩ = ⟨ \sum_{i = 1}^{m} ξ_{i} v_{i}, \sum_{j = 1}^{m} c_{j} v_{j} ⟩ = \sum_{i, j = 1}^{n} ξ_{i} ⟨ v_{i}, v_{j} ⟩ c_{j} \\ = (\begin{array}{c} ξ_{1} & \dots & ξ_{m} \end{array}) {(⟨ v_{i}, v_{j} ⟩)}_{i, j} (\begin{array}{c} c_{1} \\ ⋮ \\ c_{m} \end{array}) \\ d f (x_{0}) (v) & = \sum_{i = 1}^{m} \frac{\partial f}{\partial x_{i}} (x_{0}) ξ_{i} = (\begin{array}{c} ξ_{1} & \dots & ξ_{m} \end{array}) (\begin{array}{c} \frac{\partial f}{\partial x_{1}} (x_{0}) \\ ⋮ \\ \frac{\partial f}{\partial x_{m}} (x_{0}) \end{array}) \end{aligned}

Therefore, $\begin{aligned} (\begin{array}{c} \frac{\partial f}{\partial x_{1}} (x_{0}) \\ ⋮ \\ \frac{\partial f}{\partial x_{m}} (x_{0}) \end{array}) & = {(⟨ v_{i}, v_{j} ⟩)}_{i, j} (\begin{array}{c} c_{1} \\ ⋮ \\ c_{m} \end{array}) \\ \nabla f (x_{0}) & = (\begin{array}{c} c_{1} \\ ⋮ \\ c_{m} \end{array}) = {(⟨ v_{i}, v_{j} ⟩)}_{i, j}^{- 1} (\begin{array}{c} \frac{\partial f}{\partial x_{1}} (x_{0}) \\ ⋮ \\ \frac{\partial f}{\partial x_{m}} (x_{0}) \end{array}) \\ \nabla & = {(⟨ v_{i}, v_{j} ⟩)}_{i, j}^{- 1} (\begin{array}{c} \frac{\partial}{\partial x_{1}} \\ ⋮ \\ \frac{\partial}{\partial x_{m}} \end{array}) \end{aligned}$

If $v_{1}, . . ., v_{m}$ is a set of normalized (standard) orthogonal bases, then ${(⟨ v_{i}, v_{j} ⟩)}_{i, j} = I$ , we have

∇f (x0 ) =

Moreover,

∖left|∂f-(x0)∖right| = |df (x0)(v)| = |⟨v, ∇f (x0)⟩| ≤ ∥∇f (x0)∥∥v ∥ ∂v

It takes $=$ if and only if $v, \nabla f (x_{0})$ are parallel. If we assume $∥ v ∥ = 1$ , the gradient could also be defined without inner product: $\begin{aligned} ∥ \nabla f (x_{0}) ∥ & = max_{∥ v ∥ = 1} | \frac{\partial f}{\partial v} (x_{0}) | = | \frac{\partial f}{\partial v^{*}} (x_{0}) | \\ \nabla f (x_{0}) & = ∥ \nabla f (x_{0}) ∥ v^{*} \end{aligned}$

Although this so-called gradient could be defined based on any kind of norm, yet without the backup of inner product, it does not have good properties.