π = 0.5, densities 0.2 and 0.1
responsibility r₁
0.6667.
The EM algorithm fits models with hidden variables by repeating two steps: the E-step computes , and the M-step maximises over . The likelihood never decreases.
This page is easier if you are comfortable with:
A standard case is a mixture of normal distributions: each data point comes from one of two components, but we do not know which. The E-step computes responsibilities, the probability that each point belongs to each component given the current parameters, using Bayes' theorem with the normal density. The M-step recomputes each mean as a responsibility-weighted average, a weighted average.
Example: equal weights, unit variances, current means 1 and 3, data and . For : , , so the responsibility of component 1 is . By symmetry for it is . The M-step gives and : the means move to the data.
EM is a maximum likelihood method (maximum likelihood estimation) when direct maximisation is hard, and its convergence is a fixed-point iteration (fixed-point iteration). Applications include clustering, missing data and hidden Markov models. A cousin from linear algebra is principal component analysis.
EM fits models when some information is hidden, by alternately guessing the hidden labels and re-fitting the parameters.
At x = 0 the first density is far higher, so its responsibility is 0.982.
These examples are worked by hand, step by step, so you can compare your own working.
Two components and , equal weights. Find the responsibility of component 1 for .
With data and , responsibilities of component 1 of 0.9820 and 0.0180, find the new .
With responsibilities of component 2 of 0.0180 and 0.9820, find the new .
A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.
responsibility r₁
0.6667.
new π
0.6.
new μ
1.6667.
effective count
1.8.
E-step: compute responsibilities (posterior of the hidden variable). M-step: maximise the expected complete-data log-likelihood. The likelihood never decreases.
π₁f₁ / (π₁f₁ + π₂f₂).
Average responsibility.
Weighted average of x.
Σr.
0 or 1 ✗
soft responsibilities ✓
Posterior.
global ✗
local maximum ✓
Initialisation matters.
unnormalised ✗
divide by sum ✓
Responsibilities sum to 1.
decrease ✗
never decreases ✓
EM property.
Try each one on paper, then open the answer. Use the solver above to see the full working.
0.8
0.6667
3
0.3
1
5
Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.
Solve: …
An iterative method for maximum likelihood with hidden variables.
Computes the expected complete-data log-likelihood, i.e. responsibilities.
Maximises it over the parameters.
Not always.
maximum likelihood estimation.
Bayes’ theorem.
More in Advanced statistics and stochastic processes and the formula sheets.