Least Squares Explained with Worked Examples

The least-squares solution of minimises and satisfies the normal equations . It gives the line or curve of best fit through data.

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

What Least squares (normal equations) means

When there are more equations than unknowns, no exact solution may exist. The best approximation makes the error vector perpendicular to every column of , which is the orthogonal projection of onto the column space. That condition is the normal equations.

For a straight line the equations reduce to the familiar formulas for the regression line. Solving the normal equations is an application of solving a system with an inverse.

When the columns are dependent, use the pseudoinverse, and for numerical stability the QR factorisation.

What is it used for?

Least squares finds the line or curve that comes closest to a set of points, measured by the sum of squared vertical errors.

How to use least squares (normal equations)

  1. Build the matrix with a column of ones and a column of -values.
  2. Compute and .
  3. Solve the normal equations.
  4. Read off the intercept and gradient.

Graph

Best-fit line y = 7/6 + x/2 through three points(0, 1)(1, 2)(2, 2)-0.500.511.522.500.511.522.53Best-fit line y = 7/6 + x/2 through three pointsxyleast-squares lineGraph with x from -0.5 to 2.5 and y from 0 to 3. Shows the curve least-squares line; data (3); point (0, 1) at (0, 1); point (1, 2) at (1, 2); point (2, 2) at (2, 2).

The line is not through every point; it makes the total of the squared gaps as small as possible.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: three points

Fit a line to , , .

Worked example
Answer
y=76+12xy=\tfrac76+\tfrac12x
  1. Normal equations
  2. Solve

Example 2: four points

Fit a line to , , , .

Worked example
Answer
y=0.7+2.2xy=0.7+2.2x
  1. Sums
  2. Gradient
  3. Intercept

Example 3: check the residual

For the first fit, show the residuals sum to zero.

Worked example
Answer
−16+13−16=0-\tfrac16+\tfrac13-\tfrac16=0
  1. Fitted values at x = 0, 1, 2
  2. Residuals

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

points (1,2), (2,3), (3,5)

slope b = (nΣxy − ΣxΣy)/(nΣx² − (Σx)²)

1.5.

same

intercept a = (Σy − bΣx)/n

0.3333.

fitted value at x = 2

a + 2b

3.3333.

residual at x = 3

5 − (a + 3b)

0.1667.

Word problems

The least-squares solution of Ax ≈ b minimises |Ax − b|² and satisfies the normal equations AᵀAx = Aᵀb. For a line y = a + bx it gives the usual regression formulas.

Slope

n = 3, Σx = 6, Σy = 10, Σx² = 14, Σxy = 23.

  1. (69 − 60)/(42 − 36)
  2. Check: ✓1.5

Intercept

(10 − 9)/3.

  1. 1/3
  2. Check: ✓0.3333

Fitted at x = 2 Higher

a + 2b.

  1. 0.3333 + 3
  2. Check: ✓3.3333

Residual at x = 3 Higher

y − fitted.

  1. 5 − 4.8333
  2. Check: ✓0.1667

Common mistakes

Minimising |residuals| sum

absolute ✗

sum of squares ✓

Least squares.

Solving Ax = b exactly

exact ✗

normal equations ✓

Overdetermined.

Residuals add to a positive number

positive ✗

they sum to 0 with an intercept ✓

Property.

AᵀA singular is fine

fine ✗

columns must be independent ✓

Else non-unique.

Exam tips

  1. AᵀAx = Aᵀb.
  2. Minimise squared error.
  3. Residual ⟂ columns.
  4. Independent columns ⇒ unique.
  5. Check residuals sum to 0.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. Slope for (0,1), (1,3), (2,5) Foundation

2

2. Intercept for (0,1), (1,3), (2,5) Foundation

1

3. Slope for (1,1), (2,2), (3,2) Foundation

0.5

4. Intercept for (1,1), (2,2), (3,2) Higher

0.6667

5. Slope for (0,0), (1,1), (2,4), 4 dp Higher

2

6. Intercept for (0,0), (1,1), (2,4), 4 dp Higher

-0.3333

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Least squares
Minimises
Normal equations
AᵀAx = Aᵀb.
Residual
b − Ax.
Overdetermined system
More equations than unknowns.
Projection
Fitted values.
Regression line
Best-fit line.

Frequently asked questions

What is the least-squares solution?

The x minimising |Ax − b|².

What are the normal equations?

AᵀAx = Aᵀb.

How does it relate to projection?

Ax is the projection of b onto the column space; see orthogonal projection.

When is it unique?

When the columns of A are independent.

Which page covers the pseudoinverse?

Moore–Penrose pseudoinverse.

Which page covers QR?

QR and Cholesky factorisations.

More in Vectors and matrices and the formula sheets.