Mutual Information Explained with Worked Examples

The mutual information of two variables is . It measures how much knowing one reduces the uncertainty about the other, and it is 0 exactly when and are independent.

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

This page is easier if you are comfortable with:

What Mutual information means

It equals the Kullback–Leibler divergence between the joint distribution and the product of the marginals, so it is non-negative. If and are independent, and . If (a fair bit), , so nats, the full information of one bit. The entropies are defined in Shannon entropy.

Example: two bits agree with probability 0.8, with joint probabilities . Each marginal is uniform, , and . So nats: knowing one bit tells us a little about the other. Independence is the concept of conditional probability in which .

Mutual information detects any kind of dependence, not only a linear one, unlike Spearman's rank correlation and the product moment correlation. It is used for feature selection and for the capacity of communication channels. The sums involve logarithms and probabilities that add to 1; see logarithms.

What is it used for?

Mutual information measures how much knowing one variable reduces uncertainty about another.

How to use mutual information

  1. Find the joint distribution and the two marginals.
  2. Compute , and .
  3. Form .
  4. Check that and compare with 0 for independence.

Graph

For two correlated bits: H(X) = 0.693, H(X, Y) = 1.194 and I = 0.193H(X) = 0.693H(X,Y) = 1.194I = 0.19300.511.522.533.5400.20.40.60.811.21.4For two correlated bits: H(X) = 0.693, H(X, Y) = 1.194 and I = 0.193quantityyGraph with x from 0 to 4 and y from 0 to 1.4. Shows bars (3); point H(X) = 0.693 at (1, 0.6931); point H(X,Y) = 1.194 at (2, 1.1935); point I = 0.193 at (3, 0.1927).

I equals twice H(X) minus H(X, Y); a small value means the bits are only weakly linked.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: independent bits

Two independent fair bits. Find .

Worked example
Answer
00
  1. Entropies
  2. I

Example 2: identical bits

A fair bit and . Find .

Worked example
Answer
0.6931 nats0.6931\ \text{nats}
  1. Entropies
  2. I

Example 3: correlated bits

Joint probabilities 0.4, 0.1, 0.1, 0.4 for two bits. Find .

Worked example
Answer
0.1927 nats0.1927\ \text{nats}
  1. H(X) = H(Y)
  2. H(X, Y)
  3. I

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

H(X) = H(Y) = ln 2, H(X, Y) = ln 4

I

0 (independent).

H(X, Y) = ln 2

I

0.6931.

H(X) = 1.0, H(Y) = 1.2, H(X, Y) = 1.9

I

0.3.

H(X) = 2, H(Y) = 2, H(X, Y) = 4

I

0.

Word problems

I(X; Y) = H(X) + H(Y) − H(X, Y) ≥ 0 and is 0 exactly when X and Y are independent.

Independent coins

ln 2 + ln 2 − ln 4.

  1. 0
  2. Check: ✓0

Perfectly dependent

H(X, Y) = ln 2.

  1. ln 2
  2. Check: ✓0.6931

Numbers Higher

1.0 + 1.2 − 1.9.

  1. 0.3
  2. Check: ✓0.3

Independent Higher

2 + 2 − 4.

  1. 0
  2. Check: ✓0

Common mistakes

I negative

possible ✗

never ✓

I ≥ 0.

I = 0 means unrelated in every way

only independence ✗

I = 0 iff independent ✓

Strong claim.

Using H(X)H(Y)

product ✗

sum ✓

Formula.

H(X, Y) ≥ H(X) + H(Y)

larger ✗

at most the sum ✓

Subadditive.

Exam tips

  1. I = H(X) + H(Y) − H(X, Y).
  2. I ≥ 0.
  3. I = 0 iff independent.
  4. I ≤ min(H(X), H(Y)).
  5. Symmetric in X and Y.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. I: 1.5 + 1.5 − 2.5 Foundation

0.5

2. I: 0.7 + 0.7 − 1.4 Foundation

0

3. I: 2 + 3 − 4 Foundation

1

4. I: 1 + 1 − 1 Higher

1

5. I: 0.9 + 1.1 − 1.5 Higher

0.5

6. I: 2.2 + 1.8 − 3.0 Higher

1

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Mutual information
Shared information.
Joint entropy
H(X, Y).
Independence
I = 0.
Conditional entropy
H(X
Symmetric
I(X; Y) = I(Y; X).
KL divergence
I = D(joint ‖ product).

Frequently asked questions

What is mutual information?

H(X) + H(Y) − H(X, Y).

When is it zero?

When X and Y are independent.

Can it be negative?

No.

How does it relate to KL divergence?

It is the KL divergence from the product of marginals.

Which page covers entropy?

Shannon entropy.

Which page covers conditional probability?

conditional probability.

More in Advanced statistics and stochastic processes and the formula sheets.