H(X) = H(Y) = ln 2, H(X, Y) = ln 4
I
0 (independent).
The mutual information of two variables is . It measures how much knowing one reduces the uncertainty about the other, and it is 0 exactly when and are independent.
This page is easier if you are comfortable with:
It equals the Kullback–Leibler divergence between the joint distribution and the product of the marginals, so it is non-negative. If and are independent, and . If (a fair bit), , so nats, the full information of one bit. The entropies are defined in Shannon entropy.
Example: two bits agree with probability 0.8, with joint probabilities . Each marginal is uniform, , and . So nats: knowing one bit tells us a little about the other. Independence is the concept of conditional probability in which .
Mutual information detects any kind of dependence, not only a linear one, unlike Spearman's rank correlation and the product moment correlation. It is used for feature selection and for the capacity of communication channels. The sums involve logarithms and probabilities that add to 1; see logarithms.
Mutual information measures how much knowing one variable reduces uncertainty about another.
I equals twice H(X) minus H(X, Y); a small value means the bits are only weakly linked.
These examples are worked by hand, step by step, so you can compare your own working.
Two independent fair bits. Find .
A fair bit and . Find .
Joint probabilities 0.4, 0.1, 0.1, 0.4 for two bits. Find .
A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.
I
0 (independent).
I
0.6931.
I
0.3.
I
0.
I(X; Y) = H(X) + H(Y) − H(X, Y) ≥ 0 and is 0 exactly when X and Y are independent.
ln 2 + ln 2 − ln 4.
H(X, Y) = ln 2.
1.0 + 1.2 − 1.9.
2 + 2 − 4.
possible ✗
never ✓
I ≥ 0.
only independence ✗
I = 0 iff independent ✓
Strong claim.
product ✗
sum ✓
Formula.
larger ✗
at most the sum ✓
Subadditive.
Try each one on paper, then open the answer. Use the solver above to see the full working.
0.5
0
1
1
0.5
1
Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.
Solve: …
H(X) + H(Y) − H(X, Y).
When X and Y are independent.
No.
It is the KL divergence from the product of marginals.
Shannon entropy.
conditional probability.
More in Advanced statistics and stochastic processes and the formula sheets.