Gradient Descent - MATH - Foundation

1. Directional Derivative, D_u f(a). The rate of change of function f at point a, when move in the direction of a unit vector u.

2. Gradient of f at a, ∇f(a).

3. When a function, e.g. cost function, is at point a, moving in any direction from a may change the value of the function. If it is to move in a direction u = ∇f(a), the increase of f will be maximized; If it is to move in an opposite direction u = -∇f(a), the decrease will be maximized.

4. Further, the max change in magnitude will be ||∇f(a)||, the norm of gradient of f at a.

5. A quick proof is as follows.