Multiple Regression Explained with Worked Examples

In multiple regression , the least-squares coefficients are , where the first column of the design matrix is all ones for the intercept.

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

This page is easier if you are comfortable with:

What Multiple regression (normal equations) means

Start with simple regression: , . Then , , and (the inverse of a 2 × 2 matrix), so : the line , with prediction 6.667 at . The matrix solve is explained in solving a linear system with an inverse.

With two predictors, take the data points = , , , . They lie exactly on , and the normal equations return . Each coefficient is the change in for a unit change in that predictor with the other held fixed. The product entries are those of matrix multiplication and determinants of 2 × 2 matrices.

The fitted values are the projection of onto the column space of , a geometric view related to the rank–nullity theorem. Quality is measured by the coefficient of determination and related quantities and information criteria; correlated predictors make nearly singular, the reason for ridge regression.

What is it used for?

Multiple regression fits a linear model with several predictors by solving the normal equations.

How to use multiple regression (normal equations)

  1. Build the design matrix (a column of ones plus the predictors).
  2. Compute and .
  3. Solve .
  4. Predict with .

Graph

Fitted line y = 0.667 + 1.5x for the data (1,2), (2,4), (3,5)x = 4: 6.66700.511.522.533.544.55012345678Fitted line y = 0.667 + 1.5x for the data (1,2), (2,4), (3,5)xyfitted lineGraph with x from 0 to 5 and y from 0 to 8. Shows data (3); the curve fitted line; point x = 4: 6.667 at (4, 6.6667).

The line is the least-squares fit from (XᵀX)⁻¹Xᵀy.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: simple regression

Fit to , .

Worked example
Answer
β^=(0.6667, 1.5)\hat\beta=(0.6667,\ 1.5)
  1. X^T X
  2. X^T y
  3. β

Example 2: a prediction

Predict at using that line.

Worked example
Answer
6.66676.6667
  1. Formula

Example 3: two predictors

Find for the points , , , .

Worked example
Answer
(1, 2, 3)(1,\ 2,\ 3)
  1. Exact fit

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

Data (1,1), (2,2), (3,4)

slope Sxy/Sxx = 3/2

1.5.

Intercept

ȳ − slope·x̄ = 7/3 − 3

−0.6667.

Prediction at x = 4

−0.6667 + 6

5.3333.

R²

1 − SSE/SST

0.9643.

Word problems

β̂ = (XᵀX)⁻¹Xᵀy gives least-squares coefficients; the design matrix X has a column of ones for the intercept. With one predictor this reduces to slope = Sxy/Sxx.

Slope

x = (1, 2, 3), y = (1, 2, 4). Find Sxy/Sxx.

  1. 3/2
  2. Check: ✓1.5

Intercept

Find ȳ − slope·x̄.

  1. 2.3333 − 3
  2. Check: ✓-0.6667

Prediction Higher

Predict y at x = 4.

  1. −0.6667 + 1.5 × 4
  2. Check: ✓5.3333

R² Higher

SSE = 1/6, SST = 14/3. Find R².

  1. 1 − 0.0357
  2. Check: ✓0.9643

Common mistakes

Forgetting the column of ones

no intercept ✗

add 1s ✓

Design matrix.

XXᵀ

wrong order ✗

XᵀX ✓

p × p matrix.

Collinear predictors

invertible ✗

XᵀX singular ✓

Multicollinearity.

Extrapolating

trust far outside ✗

stay within data ✓

Risky.

Exam tips

  1. β̂ = (XᵀX)⁻¹Xᵀy.
  2. Column of ones.
  3. Check residuals.
  4. R² always rises with predictors.
  5. Use adjusted R².

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. Slope for Sxy = 6, Sxx = 4 Foundation

1.5

2. Intercept ȳ = 5, slope 2, x̄ = 2 Foundation

1

3. Prediction at x = 5 with y = 1 + 2x Foundation

11

4. Residual y = 7, ŷ = 6.5 Higher

0.5

5. Slope for Sxy = −3, Sxx = 6 Higher

-0.5

6. R² for SSE 2, SST 20 Higher

0.9

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Multiple regression
Several predictors.
Design matrix (X)
Predictors with a 1s column.
Normal equations
XᵀXβ = Xᵀy.
Coefficient
Effect size.
Multicollinearity
Correlated predictors.
Adjusted R²
Penalised fit.

Frequently asked questions

What is multiple regression?

Regression with several predictors.

What is the formula?

β̂ = (XᵀX)⁻¹Xᵀy.

What is the design matrix?

Predictors plus a column of ones.

What is multicollinearity?

Highly correlated predictors.

Which page covers R²?

coefficient of determination.

Which page covers correlation?

correlation coefficient.

More in Advanced statistics and stochastic processes and the formula sheets.