f = x², x₀ = 3, η = 0.1
x₁ = 3 − 0.1 × 6 = 2.4
xₙ₊₁ = xₙ − η f′(xₙ).
Gradient descent minimises by moving against the gradient: , where is the step size (learning rate).
This page is easier if you are comfortable with:
For , and the update is . With and : , ; the error shrinks by 0.8 each step (the derivative of a power, the gradient). For with from 0: , the distance to 3 halving each step.
The step size matters. For (curvature up to ), the update multiplies by ; this has size below 1 only if . At the -coordinate flips sign without shrinking, and beyond it the iteration diverges. The slow direction (curvature 2) then limits progress, so the convergence rate depends on the ratio of curvatures, the condition number (condition number).
Gradient descent is the workhorse of machine learning (logistic regression, ridge regression). Second-order information speeds it up: Newton's method for optimisation and the conjugate gradient step. For convex problems it converges to the global minimum (convex functions); constrained versions use the KKT conditions.
Gradient descent takes repeated downhill steps proportional to the slope.
Each step multiplies the distance from the minimum by 0.8.
These examples are worked by hand, step by step, so you can compare your own working.
Find from .
Find from .
Find the largest stable for .
A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.
x₁ = 3 − 0.1 × 6 = 2.4
xₙ₊₁ = xₙ − η f′(xₙ).
2.4 − 0.1 × 4.8 = 1.92
Factor 0.8 each step.
x₁ = 3 − 0.5 × 6 = 0
Lands on the minimum.
factor 1 − 2.2 = −1.2: diverges
Step too large.
Gradient descent moves against the gradient: xₙ₊₁ = xₙ − η f′(xₙ). For f = x² the iterates shrink by (1 − 2η) each step, converging if 0 < η < 1.
Minimise f = x² from x₀ = 3 with η = 0.1. Find x₁.
Find x₂.
Show xₙ₊₁ = 0.8 xₙ.
What happens with η = 1.1?
xₙ + η f′ ✗
xₙ − η f′ ✓
That is ascent.
η = 1.1 ✗
need 0 < η < 1 for x² ✓
It overshoots.
x − η f(x) ✗
x − η f′(x) ✓
Use the gradient.
one step enough ✗
iterate to tolerance ✓
Watch f′ → 0.
Try each one on paper, then open the answer. Use the solver above to see the full working.
4
1
0.4
2.56
1.5
0.512
Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.
Solve: …
Repeatedly stepping against the gradient.
The step size η.
When 0 < η < 1.
It diverges.
Newton’s method for optimisation.
convex functions.
More in Numerical methods and optimisation and the formula sheets.