Σ = (3 1; 1 3)
trace
6.
Principal component analysis finds the eigenvectors of the covariance matrix . The -th component explains the fraction of the total variance.
This page is easier if you are comfortable with:
The first principal component is the direction in which the data vary most, and the eigenvalue is the variance in that direction. The covariance matrix is symmetric, so its eigenvectors are perpendicular (symmetric matrices, eigenvalues of a 2 × 2 matrix). For the eigenvalues are 3 and 1, with directions and ; the first explains of the variance.
For a diagonal covariance the components explain 50%, 30% and 20%, so two components give 80%. For the first explains 80%. Keeping only the leading components reduces the dimension with the least loss, and is equivalent to the best low-rank approximation of the centred data (best low rank approximation, singular value decomposition).
The total variance equals the trace of , which equals the sum of the eigenvalues (trace and determinant from eigenvalues). Standardising the variables first uses the correlation matrix instead. The variances come from the variance of data, and the projection onto a component is an orthogonal projection. A different way of discovering hidden structure in data is the EM algorithm.
PCA finds the directions along which data vary most, so many variables can be summarised by a few.
The ellipse is longest along the first principal component, which explains 75% of the variance.
These examples are worked by hand, step by step, so you can compare your own working.
Find the eigenvalues of and the proportion of variance explained by the first component.
The covariance matrix is . Find the proportions explained and the cumulative proportion of the first two.
The covariance matrix is . What fraction does the first component explain?
A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.
trace
6.
eigenvalues
4 and 2.
PC1 explains
0.6667.
PC1 explains
0.8.
Principal components are eigenvectors of the covariance matrix; component i explains λᵢ/Σλⱼ of the variance.
Total variance.
Σ = (3 1; 1 3): 3 + 1.
4/(4 + 2).
λ = 4, 1.
raw ✗
centre first ✓
Covariance.
smallest ✗
largest first ✓
Most variance.
correlated ✗
orthogonal, uncorrelated ✓
Symmetric Σ.
raw units ✗
standardise ✓
Dominating variables.
Try each one on paper, then open the answer. Use the solver above to see the full working.
0.75
0.6
10
7
0.7
0.8
Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.
Solve: …
Finding orthogonal directions of maximal variance.
Eigenvectors of the covariance matrix.
λᵢ divided by the sum of eigenvalues.
Yes.
singular value decomposition.
symmetric matrices.
More in Advanced statistics and stochastic processes and the formula sheets.