Σxy = 10, Σx² = 5, λ = 0
β̂ = Σxy/(Σx² + λ)
2.
Ridge regression adds a penalty to least squares. The solution is , and it shrinks the coefficients towards zero as grows.
This page is easier if you are comfortable with:
Ordinary least squares gives . When predictors are nearly collinear, is almost singular (a large condition number) and the coefficients are unstable. Adding makes the matrix well conditioned. For one predictor without intercept, : for , , and , so at , at , and at .
With (two orthogonal predictors) and the solution is , which is for : every coefficient shrinks by the same factor. The bias grows with while the variance falls: see the bias–variance decomposition. The inverse uses matrix inversion.
From a Bayesian viewpoint ridge regression is the posterior mode with a normal prior on the coefficients (posterior ∝ likelihood × prior). The version, LASSO, sets coefficients exactly to zero, see soft-thresholding. Compare also multiple regression and the regression line.
Ridge regression shrinks coefficients to make them stable when predictors are correlated or data are scarce.
More penalty means more shrinkage towards zero.
These examples are worked by hand, step by step, so you can compare your own working.
For , (no intercept) find the least squares slope.
For the same data find the ridge slope with .
For , and find .
A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.
β̂ = Σxy/(Σx² + λ)
2.
β̂
1.
β̂
1.6667.
shrinkage factor Σx²/(Σx² + λ)
0.5.
Ridge: β̂ = (XᵀX + λI)⁻¹Xᵀy. For one predictor β̂ = Σxy/(Σx² + λ), shrinking OLS by Σx²/(Σx² + λ).
λ = 0.
Denominator 10.
Denominator 6.
λ = 5.
shrinks ✗
gives OLS ✓
No penalty.
exactly 0 ✗
shrinks towards 0 ✓
Lasso gives zeros.
penalised ✗
not penalised ✓
Convention.
raw units ✗
standardise ✓
Penalty depends on scale.
Try each one on paper, then open the answer. Use the solver above to see the full working.
2
3
0.5
1
0.75
1.5
Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.
Solve: …
Least squares with an L2 penalty λ‖β‖².
(XᵀX + λI)⁻¹Xᵀy.
Larger λ shrinks coefficients more.
No; see soft thresholding for the lasso.
bias–variance decomposition.
multiple regression.
More in Advanced statistics and stochastic processes and the formula sheets.