Kullback–Leibler Divergence Explained with Worked Examples

The Kullback–Leibler divergence measures how different the distribution is from . It is never negative, and it is 0 only when .

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

This page is easier if you are comfortable with:

What Kullback–Leibler divergence means

The non-negativity follows from Jensen's inequality for the concave logarithm, or from (the derivative of ln x). It is not symmetric, so it is not a true distance. For and : , while . With every term is .

In statistics, minimising is equivalent to maximum likelihood; in machine learning the cross-entropy loss is , see softmax and cross-entropy. Information measures also include Shannon entropy and mutual information, which is a KL divergence from independence.

The logarithm base sets the unit: natural logs give nats, base 2 gives bits. The divergence appears in the choice of models with the AIC and BIC, which estimate it up to a constant. The sums use logarithms and probabilities that add to 1, see the sum of probabilities.

What is it used for?

KL divergence measures the information lost when one distribution is used to approximate another, and it is never negative.

How to use kullback–leibler divergence

  1. List the two distributions on the same outcomes (with wherever ).
  2. Compute for each outcome.
  3. Add the terms.
  4. Check that the result is non-negative.

Graph

D(P‖Q) = 0.511 and D(Q‖P) = 0.368 for P = (0.5, 0.5), Q = (0.9, 0.1)D(P‖Q) = 0.511D(Q‖P) = 0.368D(P‖P) = 000.511.522.533.5400.10.20.30.40.50.6D(P‖Q) = 0.511 and D(Q‖P) = 0.368 for P = (0.5, 0.5), Q = (0.9, 0.1)directionyGraph with x from 0 to 4 and y from 0 to 0.6. Shows bars (3); point D(P‖Q) = 0.511 at (1, 0.5108); point D(Q‖P) = 0.368 at (2, 0.3681); point D(P‖P) = 0 at (3, 0).

The two directions give different values, so the divergence is not symmetric.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: P to Q

For and find .

Worked example
Answer
0.5108 nats0.5108\ \text{nats}
  1. Terms
  2. Sum

Example 2: Q to P

For the same distributions find .

Worked example
Answer
0.3681 nats (not symmetric)0.3681\ \text{nats (not symmetric)}
  1. Terms
  2. Sum

Example 3: identical distributions

Find for .

Worked example
Answer
00
  1. Each term

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

P = (0.5, 0.5), Q = (0.9, 0.1)

D(P‖Q)

0.5108.

Q = (0.25, 0.75)

D(P‖Q)

0.1438.

P = Q

D

0.

P = (0.5, 0.5), Q = (0.5, 0.5)

D

0.

Word problems

D(P‖Q) = Σ pᵢ ln(pᵢ/qᵢ) ≥ 0, with equality only if P = Q. It is not symmetric.

Q = (0.9, 0.1)

P = (0.5, 0.5).

  1. 0.5 ln(0.556) + 0.5 ln 5
  2. Check: ✓0.5108

Q = (0.25, 0.75)

Same P.

  1. 0.5 ln 2 + 0.5 ln(0.667)
  2. Check: ✓0.1438

Equal Higher

P = Q.

  1. 0
  2. Check: ✓0

Same distribution Higher

ln 1 = 0.

  1. 0
  2. Check: ✓0

Common mistakes

D(P‖Q) = D(Q‖P)

symmetric ✗

not symmetric ✓

Not a distance.

Negative divergence

possible ✗

never ✓

Gibbs inequality.

Using log of difference

p − q ✗

ln(p/q) ✓

Ratio.

q = 0 where p > 0

finite ✗

infinite ✓

Support.

Exam tips

  1. D ≥ 0.
  2. D = 0 iff P = Q.
  3. Not symmetric.
  4. ln(p/q) weighted by p.
  5. Q = 0 with P > 0 gives ∞.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. D: P = (0.5, 0.5), Q = (0.8, 0.2), 4 dp Foundation

0.2231

2. D: P = (0.9, 0.1), Q = (0.5, 0.5), 4 dp Foundation

0.3681

3. D: P = (1, 0), Q = (0.5, 0.5), 4 dp Foundation

0.6931

4. D: P = Q = (0.3, 0.7) Higher

0

5. D: P = (0.25, 0.75), Q = (0.5, 0.5), 4 dp Higher

0.1308

6. D: P = (0.5, 0.5), Q = (0.6, 0.4), 4 dp Higher

0.0204

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

KL divergence
Difference between distributions.
Relative entropy
Same thing.
Gibbs inequality
D ≥ 0.
Support
Where p > 0.
Cross-entropy
H(P) + D.
Asymmetric
D(P‖Q) ≠ D(Q‖P).

Frequently asked questions

What is KL divergence?

Σ pᵢ ln(pᵢ/qᵢ).

Is it negative?

Never.

Is it a distance?

No, it is not symmetric.

When is it zero?

When P = Q.

Which page covers entropy?

Shannon entropy.

Which page covers mutual information?

mutual information.

More in Advanced statistics and stochastic processes and the formula sheets.