Ridge Regression Explained with Worked Examples

Ridge regression adds a penalty to least squares. The solution is , and it shrinks the coefficients towards zero as grows.

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

This page is easier if you are comfortable with:

What Ridge regression means

Ordinary least squares gives . When predictors are nearly collinear, is almost singular (a large condition number) and the coefficients are unstable. Adding makes the matrix well conditioned. For one predictor without intercept, : for , , and , so at , at , and at .

With (two orthogonal predictors) and the solution is , which is for : every coefficient shrinks by the same factor. The bias grows with while the variance falls: see the bias–variance decomposition. The inverse uses matrix inversion.

From a Bayesian viewpoint ridge regression is the posterior mode with a normal prior on the coefficients (posterior ∝ likelihood × prior). The version, LASSO, sets coefficients exactly to zero, see soft-thresholding. Compare also multiple regression and the regression line.

What is it used for?

Ridge regression shrinks coefficients to make them stable when predictors are correlated or data are scarce.

How to use ridge regression

  1. Form and .
  2. Add to the diagonal of .
  3. Solve .
  4. Choose by cross-validation or a criterion such as AIC.

Graph

The ridge slope 25/(14 + λ) falls as the penalty λ growsλ = 0: 1.786 (least squares)λ = 14: 0.8930246810121416182000.20.40.60.811.21.41.61.8The ridge slope 25/(14 + λ) falls as the penalty λ growsλsloperidge slopeGraph with x from 0 to 20 and y from 0 to 1.9. Shows the curve ridge slope; point λ = 0: 1.786 (least squares) at (0, 1.7857); point λ = 14: 0.893 at (14, 0.8929).

More penalty means more shrinkage towards zero.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: ordinary least squares

For , (no intercept) find the least squares slope.

Worked example
Answer
1.78571.7857
  1. Σxy and Σx²
  2. Slope

Example 2: ridge with λ = 1

For the same data find the ridge slope with .

Worked example
Answer
1.66671.6667
  1. Slope

Example 3: two predictors

For , and find .

Worked example
Answer
(1.5, 0.5)(1.5,\ 0.5)
  1. Matrix
  2. Solution

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

Σxy = 10, Σx² = 5, λ = 0

β̂ = Σxy/(Σx² + λ)

2.

λ = 5

β̂

1.

λ = 1

β̂

1.6667.

λ = 5

shrinkage factor Σx²/(Σx² + λ)

0.5.

Word problems

Ridge: β̂ = (XᵀX + λI)⁻¹Xᵀy. For one predictor β̂ = Σxy/(Σx² + λ), shrinking OLS by Σx²/(Σx² + λ).

OLS

λ = 0.

  1. 10/5
  2. Check: ✓2

λ = 5

Denominator 10.

  1. 10/10
  2. Check: ✓1

λ = 1 Higher

Denominator 6.

  1. 10/6
  2. Check: ✓1.6667

Shrinkage Higher

λ = 5.

  1. 5/10
  2. Check: ✓0.5

Common mistakes

λ = 0 shrinks

shrinks ✗

gives OLS ✓

No penalty.

Ridge sets coefficients to zero

exactly 0 ✗

shrinks towards 0 ✓

Lasso gives zeros.

Penalising the intercept

penalised ✗

not penalised ✓

Convention.

Not scaling predictors

raw units ✗

standardise ✓

Penalty depends on scale.

Exam tips

  1. β̂ = (XᵀX + λI)⁻¹Xᵀy.
  2. λ ↑ ⇒ shrinkage ↑.
  3. Standardise predictors.
  4. Choose λ by cross-validation.
  5. Reduces variance, adds bias.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. β̂: Σxy = 12, Σx² = 4, λ = 2 Foundation

2

2. β̂: Σxy = 9, Σx² = 3, λ = 0 Foundation

3

3. Shrinkage factor: Σx² = 8, λ = 8 Foundation

0.5

4. β̂: Σxy = 20, Σx² = 10, λ = 10 Higher

1

5. Shrinkage factor: Σx² = 6, λ = 2 Higher

0.75

6. β̂: Σxy = 15, Σx² = 5, λ = 5, 1 dp Higher

1.5

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Ridge regression
L2-penalised least squares.
Penalty (λ)
Strength of shrinkage.
Shrinkage
Coefficients pulled to 0.
Regularisation
Controls overfitting.
Cross-validation
Choose λ.
OLS
Ordinary least squares.

Frequently asked questions

What is ridge regression?

Least squares with an L2 penalty λ‖β‖².

What is the ridge solution?

(XᵀX + λI)⁻¹Xᵀy.

What does λ do?

Larger λ shrinks coefficients more.

Does ridge give exact zeros?

No; see soft thresholding for the lasso.

Which page covers bias and variance?

bias–variance decomposition.

Which page covers multiple regression?

multiple regression.

More in Advanced statistics and stochastic processes and the formula sheets.