Principal Component Analysis Explained with Worked Examples

Principal component analysis finds the eigenvectors of the covariance matrix . The -th component explains the fraction of the total variance.

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

What Principal component analysis means

The first principal component is the direction in which the data vary most, and the eigenvalue is the variance in that direction. The covariance matrix is symmetric, so its eigenvectors are perpendicular (symmetric matrices, eigenvalues of a 2 × 2 matrix). For the eigenvalues are 3 and 1, with directions and ; the first explains of the variance.

For a diagonal covariance the components explain 50%, 30% and 20%, so two components give 80%. For the first explains 80%. Keeping only the leading components reduces the dimension with the least loss, and is equivalent to the best low-rank approximation of the centred data (best low rank approximation, singular value decomposition).

The total variance equals the trace of , which equals the sum of the eigenvalues (trace and determinant from eigenvalues). Standardising the variables first uses the correlation matrix instead. The variances come from the variance of data, and the projection onto a component is an orthogonal projection. A different way of discovering hidden structure in data is the EM algorithm.

What is it used for?

PCA finds the directions along which data vary most, so many variables can be summarised by a few.

How to use principal component analysis

  1. Centre the data and compute the covariance matrix .
  2. Find its eigenvalues and eigenvectors.
  3. Order them by size; the leading eigenvector is the first component.
  4. Compute the proportion of variance .

Graph

The covariance matrix [[2, 1], [1, 2]]: the long axis is along (1, 1)first component (75%)-2-1.5-1-0.500.511.52-2-1.5-1-0.500.511.52The covariance matrix [[2, 1], [1, 2]]: the long axis is along (1, 1)xyGraph with x from -2 to 2 and y from -2 to 2. Shows one-standard-deviation ellipse (73); point first component (75%) at (1.2247, 1.2247).

The ellipse is longest along the first principal component, which explains 75% of the variance.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: a 2 × 2 covariance

Find the eigenvalues of and the proportion of variance explained by the first component.

Worked example
Answer
3, 1; 75%3,\ 1;\ 75\%
  1. Eigenvalues
  2. Proportion

Example 2: three variables

The covariance matrix is . Find the proportions explained and the cumulative proportion of the first two.

Worked example
Answer
50%, 30%, 20%; 80%50\%,\ 30\%,\ 20\%;\ 80\%
  1. Proportions
  2. First two

Example 3: a dominant direction

The covariance matrix is . What fraction does the first component explain?

Worked example
Answer
80%80\%
  1. Proportion

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

Σ = (3 1; 1 3)

trace

6.

same

eigenvalues

4 and 2.

same

PC1 explains

0.6667.

eigenvalues 4 and 1

PC1 explains

0.8.

Word problems

Principal components are eigenvectors of the covariance matrix; component i explains λᵢ/Σλⱼ of the variance.

Trace

Total variance.

  1. 3 + 3
  2. Check: ✓6

Eigenvalue

Σ = (3 1; 1 3): 3 + 1.

  1. 4
  2. Check: and 3 − 1 = 2 ✓4

Fraction Higher

4/(4 + 2).

  1. 4/6
  2. Check: ✓0.6667

Other case Higher

λ = 4, 1.

  1. 4/5
  2. Check: ✓0.8

Common mistakes

PCA on raw uncentred data

raw ✗

centre first ✓

Covariance.

PC with smallest eigenvalue first

smallest ✗

largest first ✓

Most variance.

Components correlated

correlated ✗

orthogonal, uncorrelated ✓

Symmetric Σ.

Scale ignored

raw units ✗

standardise ✓

Dominating variables.

Exam tips

  1. Centre the data.
  2. Eigendecompose Σ.
  3. Order by λ.
  4. Fraction = λᵢ/Σλ.
  5. Standardise variables if units differ.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. Fraction: eigenvalues 6 and 2 Foundation

0.75

2. Fraction: eigenvalues 3, 1, 1 Foundation

0.6

3. Total variance of Σ = (5 2; 2 5) Foundation

10

4. Eigenvalue of (5 2; 2 5): larger Higher

7

5. Fraction for (5 2; 2 5) PC1, 4 dp Higher

0.7

6. Cumulative fraction: λ = 5, 3, 2 (first two) Higher

0.8

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Principal component
Eigenvector of Σ.
Covariance matrix
Σ.
Eigenvalue
Variance along a component.
Explained variance
λᵢ/Σλ.
Orthogonal
Perpendicular.
Dimension reduction
Keep top components.

Frequently asked questions

What is PCA?

Finding orthogonal directions of maximal variance.

How do I find the components?

Eigenvectors of the covariance matrix.

How much does a component explain?

λᵢ divided by the sum of eigenvalues.

Do I need to centre the data?

Yes.

Which page covers the SVD?

singular value decomposition.

Which page covers symmetric matrices?

symmetric matrices.

More in Advanced statistics and stochastic processes and the formula sheets.