Gradient Descent Explained with Worked Examples

Gradient descent minimises by moving against the gradient: , where is the step size (learning rate).

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

This page is easier if you are comfortable with:

What Gradient descent means

For , and the update is . With and : , ; the error shrinks by 0.8 each step (the derivative of a power, the gradient). For with from 0: , the distance to 3 halving each step.

The step size matters. For (curvature up to ), the update multiplies by ; this has size below 1 only if . At the -coordinate flips sign without shrinking, and beyond it the iteration diverges. The slow direction (curvature 2) then limits progress, so the convergence rate depends on the ratio of curvatures, the condition number (condition number).

Gradient descent is the workhorse of machine learning (logistic regression, ridge regression). Second-order information speeds it up: Newton's method for optimisation and the conjugate gradient step. For convex problems it converges to the global minimum (convex functions); constrained versions use the KKT conditions.

What is it used for?

Gradient descent takes repeated downhill steps proportional to the slope.

How to use gradient descent

  1. Compute the gradient .
  2. Choose a step size .
  3. Update .
  4. Stop when the gradient is small.

Graph

Gradient descent on f = x² with α = 0.1 from x = 4: xₖ = 4(0.8)ᵏstep 5: 1.310701234567891000.511.522.533.54Gradient descent on f = x² with α = 0.1 from x = 4: xₖ = 4(0.8)ᵏstep kx_kx_k = 4(0.8)^kGraph with x from 0 to 10 and y from 0 to 4.2. Shows the curve x_k = 4(0.8)^k; point step 5: 1.3107 at (5, 1.31072).

Each step multiplies the distance from the minimum by 0.8.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: f = x², α = 0.1

Find from .

Worked example
Answer
1.310721.31072
  1. Factor
  2. x₅

Example 2: f = (x − 3)², α = 0.25

Find from .

Worked example
Answer
2.252.25
  1. x₁
  2. x₂

Example 3: the stability limit

Find the largest stable for .

Worked example
Answer
0.10.1
  1. L
  2. Limit

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

f = x², x₀ = 3, η = 0.1

x₁ = 3 − 0.1 × 6 = 2.4

xₙ₊₁ = xₙ − η f′(xₙ).

x₂

2.4 − 0.1 × 4.8 = 1.92

Factor 0.8 each step.

η = 0.5

x₁ = 3 − 0.5 × 6 = 0

Lands on the minimum.

η = 1.1

factor 1 − 2.2 = −1.2: diverges

Step too large.

Word problems

Gradient descent moves against the gradient: xₙ₊₁ = xₙ − η f′(xₙ). For f = x² the iterates shrink by (1 − 2η) each step, converging if 0 < η < 1.

First step

Minimise f = x² from x₀ = 3 with η = 0.1. Find x₁.

  1. f′(3)6
  2. x₁3 − 0.6
  3. Check: 2.4 ✓2.4.

Second step

Find x₂.

  1. f′(2.4)4.8
  2. x₂2.4 − 0.48
  3. Check: 1.92 ✓1.92.

The factor Higher

Show xₙ₊₁ = 0.8 xₙ.

  1. 1 − 2η0.8
  2. Check: ✓0.8.

Divergence Higher

What happens with η = 1.1?

  1. Factor1 − 2.2 = −1.2
  2. Check: |−1.2| > 1, diverges ✓Diverges.

Common mistakes

Adding the gradient

xₙ + η f′ ✗

xₙ − η f′ ✓

That is ascent.

Too large a step

η = 1.1 ✗

need 0 < η < 1 for x² ✓

It overshoots.

Using f instead of f′

x − η f(x) ✗

x − η f′(x) ✓

Use the gradient.

Stopping too early

one step enough ✗

iterate to tolerance ✓

Watch f′ → 0.

Exam tips

  1. x − η f′(x).
  2. Choose η small enough.
  3. Factor 1 − 2η for x².
  4. Stop when |f′| is small.
  5. Learning rate tuning matters.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. x₁ for f = x², x₀ = 5, η = 0.1 Foundation

4

2. x₁ for f = x², x₀ = 2, η = 0.25 Foundation

1

3. Factor 1 − 2η for η = 0.3 Foundation

0.4

4. x₂ for x₀ = 4, η = 0.1 Higher

2.56

5. f = (x − 3)², x₀ = 0, η = 0.25: x₁ Higher

1.5

6. x₃ for x₀ = 1, η = 0.1 Higher

0.512

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Gradient descent
Iterative minimisation.
Learning rate (η)
Step size.
Gradient
Direction of steepest rise.
Minimum
Lowest value.
Convergence
Approaching the minimum.
Overshoot
Passing the minimum.

Frequently asked questions

What is gradient descent?

Repeatedly stepping against the gradient.

What is the learning rate?

The step size η.

When does it converge for x²?

When 0 < η < 1.

What if η is too big?

It diverges.

Which page covers Newton’s method for optimisation?

Newton’s method for optimisation.

Which page covers convex functions?

convex functions.

More in Numerical methods and optimisation and the formula sheets.