Data (1,1), (2,2), (3,4)
slope Sxy/Sxx = 3/2
1.5.
In multiple regression , the least-squares coefficients are , where the first column of the design matrix is all ones for the intercept.
This page is easier if you are comfortable with:
Start with simple regression: , . Then , , and (the inverse of a 2 × 2 matrix), so : the line , with prediction 6.667 at . The matrix solve is explained in solving a linear system with an inverse.
With two predictors, take the data points = , , , . They lie exactly on , and the normal equations return . Each coefficient is the change in for a unit change in that predictor with the other held fixed. The product entries are those of matrix multiplication and determinants of 2 × 2 matrices.
The fitted values are the projection of onto the column space of , a geometric view related to the rank–nullity theorem. Quality is measured by the coefficient of determination and related quantities and information criteria; correlated predictors make nearly singular, the reason for ridge regression.
Multiple regression fits a linear model with several predictors by solving the normal equations.
The line is the least-squares fit from (XᵀX)⁻¹Xᵀy.
These examples are worked by hand, step by step, so you can compare your own working.
Fit to , .
Predict at using that line.
Find for the points , , , .
A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.
slope Sxy/Sxx = 3/2
1.5.
ȳ − slope·x̄ = 7/3 − 3
−0.6667.
−0.6667 + 6
5.3333.
1 − SSE/SST
0.9643.
β̂ = (XᵀX)⁻¹Xᵀy gives least-squares coefficients; the design matrix X has a column of ones for the intercept. With one predictor this reduces to slope = Sxy/Sxx.
x = (1, 2, 3), y = (1, 2, 4). Find Sxy/Sxx.
Find ȳ − slope·x̄.
Predict y at x = 4.
SSE = 1/6, SST = 14/3. Find R².
no intercept ✗
add 1s ✓
Design matrix.
wrong order ✗
XᵀX ✓
p × p matrix.
invertible ✗
XᵀX singular ✓
Multicollinearity.
trust far outside ✗
stay within data ✓
Risky.
Try each one on paper, then open the answer. Use the solver above to see the full working.
1.5
1
11
0.5
-0.5
0.9
Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.
Solve: …
Regression with several predictors.
β̂ = (XᵀX)⁻¹Xᵀy.
Predictors plus a column of ones.
Highly correlated predictors.
coefficient of determination.
More in Advanced statistics and stochastic processes and the formula sheets.