Gradient, Hessian, and Jacobian: All About Change
The gradient, Hessian, and Jacobian are not three unrelated equations. They describe three layers of local change.
Gradient: Which way should I move to increase one value fastest?
Hessian: How does the gradient change as I move?
Jacobian: If I change the inputs a little, how do all the outputs change?
Once you start with these questions, the notation has a job. It stops looking like a wall of symbols.
The Gradient: Which Way Is Uphill?
Take a scalar function with several inputs:
It may map a position to height, or a set of model weights to loss. Its gradient collects one partial derivative for each input.
Imagine a mountain. At your current position, the gradient points along the steepest path uphill. Its length tells you how steep that path is. The negative gradient points downhill.
This is local information. It does not reveal the whole mountain or promise the best route to its highest peak. It tells you what looks steepest from where you stand.
Press or drag anywhere to choose a starting point. The orange arrow shows the local gradient; the white path follows gradient descent.
- Position
- (0.80, -0.50)
- Gradient
- (1.12, -1.00)
- Height
- 0.70
This is enough for gradient descent. When minimizing a loss, the algorithm takes a step against the gradient:
The Hessian: How Is the Slope Changing?
The gradient changes as you move. The Hessian matrix records that change by taking the derivatives of the gradient:
Put another way: the Hessian is the Jacobian of the gradient. The gradient collects the first derivatives into a vector; taking that vector's Jacobian collects all second derivatives into a matrix.
Some row-vector conventions write a transpose here. For a sufficiently smooth function, the Hessian is symmetric, so the entries agree either way.
On a landscape, the Hessian describes the local curvature: whether the ground forms a peak, valley, ridge, or saddle, and which directions bend most strongly. Unlike the gradient, it describes how the slope itself changes.
Press or drag anywhere to choose a starting point. The crossed lines show the Hessian's principal curvature directions; the white path follows damped Newton's method.
- Gradient
- (0.00, 0.00)
- Hessian
- [[0.00, 0.00], [0.00, 0.00]]
- Local shape
- —
Blue and green show the directions of greatest and least second-order change. The optimizer adds damping where the surface is not locally bowl-shaped, caps long steps, and backtracks until height falls.
The Hessian gives a local quadratic model:
The gradient supplies the local slope. The Hessian tells us how that slope bends. Second-order methods use this curvature to choose better steps. Newton's method in optimization, for example, scales the gradient using the inverse Hessian.
Near a suitable minimum, it can converge in fewer steps than plain gradient descent. The tradeoff is cost: a full Hessian becomes expensive when a model has many parameters.
The Jacobian: How Do All Outputs Respond?
Now let the function have several inputs and several outputs:
Think of a machine with n knobs and m gauges. Turn one knob a little. Several gauges may move. The Jacobian matrix records every input to output response.
Each row follows one output. Each column follows one input. The Jacobian is the best local linear map between input space and output space.
One input can affect both outputs. The fixed Jacobian below maps the blue input vector to the orange output vector.
- Input Δx
- (0.70, 0.30)
- Jacobian J
- [[1, 0.6], [-0.4, 0.8]]
- Output ΔF
- (0.88, -0.04)
This matters in robot kinematics. A robot arm maps joint angles to the pose of its end effector. Its Jacobian maps joint velocities to the end effector's velocity.
A gradient is closely related to a Jacobian. For a function with one scalar output, the Jacobian has one row. The gradient usually writes the same derivatives as a column.
One Function, Three Views
Consider the bowl f(x, y) = x² + 2y². Its gradient is (2x, 4y). At (1, 1), the gradient is (2, 4). The function rises faster in the y direction.
Because this function has one output, its Jacobian contains the same first derivatives. Its Hessian is [[2, 0], [0, 4]]. The curvature stays constant everywhere.
| Object | Function shape | What it describes |
|---|---|---|
| Gradient | ℝn → ℝ | The local slope of one output |
| Hessian | ℝn → ℝ | The local change of the gradient, or curvature |
| Jacobian | ℝn → ℝm | Every output's local response to every input |
Why Engineers Keep Meeting Them
- Machine learning uses gradients to update model weights.
- Robotics uses Jacobians to connect joint motion with tool motion.
- Computer vision uses Jacobians to adjust camera poses, scene points, and image errors.
- Second order optimization uses Hessians, or estimates of them, to account for curvature.
They are not separate mathematical tricks. They are different questions about change.
The gradient asks how one value changes. The Jacobian asks how many outputs change. The Hessian asks how the rate of change itself changes.