Kullback–Leibler Divergence Explained with Worked Examples
The Kullback–Leibler divergence DKL(P∥Q)=∑pilnqipi measures how different the distribution Q is from P. It is never negative, and it is 0 only when P=Q.
The non-negativity follows from Jensen's inequality for the concave logarithm, or from lnx≤x−1 (the derivative of ln x). It is not symmetric, so it is not a true distance. For P=(0.5,0.5) and Q=(0.9,0.1): D(P∥Q)=0.5ln0.90.5+0.5ln0.10.5=0.5108, while D(Q∥P)=0.9ln1.8+0.1ln0.2=0.3681. With P=Q every term is pln1=0.
In statistics, minimising DKL(Ptrue∥Qθ) is equivalent to maximum likelihood; in machine learning the cross-entropy loss is H(P)+DKL(P∥Q), see softmax and cross-entropy. Information measures also include Shannon entropy and mutual information, which is a KL divergence from independence.
The logarithm base sets the unit: natural logs give nats, base 2 gives bits. The divergence appears in the choice of models with the AIC and BIC, which estimate it up to a constant. The sums use logarithms and probabilities that add to 1, see the sum of probabilities.
What is it used for?
KL divergence measures the information lost when one distribution is used to approximate another, and it is never negative.