Newton’s Method for Optimisation Explained with Worked Examples

Newton's method for minimisation updates , where is the Hessian. In one variable: . It is Newton–Raphson applied to .

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

This page is easier if you are comfortable with:

What Newton’s method for optimisation means

For a quadratic, one step lands on the minimum: , from : , , . In two variables, has Hessian , and from the gradient is , so (the Hessian). For (minimum at ): , , , ; then , , , approaching 0.6931.

Near a minimum the convergence is quadratic (Newton's quadratic convergence, Newton–Raphson), far faster than gradient descent (gradient descent), but each step needs the Hessian and a linear solve, costing (Gaussian elimination cost). It may head to a maximum or saddle if the Hessian is not positive definite, so safeguards are used.

Quasi-Newton methods (BFGS) approximate cheaply. For convex functions (convex functions) the Newton step is a descent direction; with constraints, see the KKT conditions. The stationary point conditions are in stationary points.

What is it used for?

Newton's method for optimisation uses the curvature to jump straight towards a minimum.

How to use newton’s method for optimisation

  1. Compute the gradient and Hessian at .
  2. Solve .
  3. Update .
  4. Check that is positive definite.

Graph

f(x) = eˣ − 2x with its minimum at ln 2minimum at 0.6931x1 = 1-0.4-0.200.20.40.60.811.21.400.511.522.53f(x) = eˣ − 2x with its minimum at ln 2xff(x) = e^x − 2xGraph with x from -0.5 to 1.5 and y from 0 to 3.2. Shows the curve f(x) = e^x − 2x; point minimum at 0.6931 at (0.6931, 0.6137); point x1 = 1 at (1, 0.7183).

Newton's iterates 0, 1, 0.7358 close in on the minimum.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: a quadratic

Find for from .

Worked example
Answer
22
  1. f′, f″
  2. x₁

Example 2: e^x − 2x

Find from .

Worked example
Answer
0.73580.7358
  1. x₁
  2. x₂

Example 3: two variables

Find for from .

Worked example
Answer
(0, 0)(0,\ 0)
  1. H⁻¹∇f

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

f = x⁴ from 1

x₁ = 1 − f′/f″ = 1 − 4/12 = 0.6667

Newton on f′.

Next step

x₂ = 0.4444

Ratio 2/3 each step.

f = eˣ − 2x from 0

x₁ = 1, x₂ = 0.7358

Minimum at ln 2 = 0.6931.

A quadratic

f = x² − 4x + 5 from any x₀ reaches 2 in one step

Exact for quadratics.

Word problems

Newton’s method for minimising f applies the root-finding iteration to f′: xₙ₊₁ = xₙ − f′(xₙ)/f″(xₙ). It converges quadratically near a minimum with f″ > 0 and is exact in one step for quadratics.

f = x⁴

Apply one step from x₀ = 1.

  1. f′4x³ = 4
  2. f″12x² = 12
  3. x₁1 − 1/3
  4. Check: 0.6667 ✓0.6667.

A quadratic

Minimise x² − 4x + 5 from x₀ = 0 in one step.

  1. f′2x − 4 = −4
  2. f″2
  3. x₁0 + 2
  4. Check: minimum at 2 ✓2.

eˣ − 2x Higher

Take two steps from x₀ = 0. Give x₂ to 4 decimal places.

  1. x₁0 − (1 − 2)/1 = 1
  2. x₂1 − (e − 2)/e
  3. Check: 0.7358 ✓0.7358.

The true minimum Higher

Where is the minimum of eˣ − 2x?

  1. eˣ = 2
  2. Check: ln 2 ✓0.6931.

Common mistakes

Adding instead of subtracting

x + f′/f″ ✗

x − f′/f″ ✓

Step opposes the gradient.

Using f instead of f′

f/f′ ✗

f′/f″ ✓

Optimising uses the derivative.

Ignoring f″ < 0

finds a maximum ✗

check f″ > 0 ✓

Newton finds stationary points.

Dividing by zero

f″ = 0 ✗

step undefined ✓

Use a different method.

Exam tips

  1. x − f′/f″.
  2. Needs f″ ≠ 0.
  3. Exact for quadratics.
  4. Check f″ > 0.
  5. Quadratic convergence.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. f = x² from 5: x₁ Foundation

0

2. f = x⁴ from 2: x₁ Foundation

1.3333

3. f = x⁴ from 1.5: x₁ Foundation

1

4. f = (x − 3)² from 0: x₁ Higher

3

5. f″ of x⁴ at 2 Higher

48

6. f′ of x⁴ at 2 Higher

32

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Newton’s method (optimisation)
xₙ₊₁ = xₙ − f′/f″.
Hessian
Matrix of second derivatives.
Stationary point
f′ = 0.
Minimum
f″ > 0.
Quadratic convergence
Error squares.
Step
xₙ₊₁ − xₙ.

Frequently asked questions

What is Newton’s method for optimisation?

Newton’s root-finding applied to f′.

What is the update?

x − f′(x)/f″(x).

How fast is it?

Quadratic near a minimum.

Is it exact for quadratics?

Yes, in one step.

Which page covers gradient descent?

gradient descent.

Which page covers convexity?

convex functions.

More in Numerical methods and optimisation and the formula sheets.