When you model data, you want to know how much is lost by using the wrong distribution. The KL divergence calculator takes a true distribution P and a model distribution Q over three categories, and gives the KL divergence in both directions, the Jensen-Shannon divergence, the cross-entropy and the entropy of P, all in bits.
How the calculation works
- Enter the weights of the true distribution P for three categories. They can be probabilities or counts, since the calculator rescales them to sum to 1.
- Enter the weights of the model distribution Q for the same three categories. Every weight must be positive.
- The KL divergence is D(P ‖ Q) = Σ P × log₂(P ÷ Q), in bits. It measures the extra bits needed per symbol if you encode data from P using a code made for Q.
- The calculator also gives the reverse divergence, the symmetric Jensen-Shannon divergence, the cross-entropy and the entropy of P. The cross-entropy equals the entropy plus the KL divergence.
Example
The true distribution is P = (0.5, 0.3, 0.2) and the model says Q = (0.4, 0.4, 0.2). The divergence is D(P ‖ Q) = 0.5 log₂(0.5 ÷ 0.4) + 0.3 log₂(0.3 ÷ 0.4) + 0.2 log₂(1) = 0.0365 bits. The reverse D(Q ‖ P) is 0.0372 bits, a little different, and the Jensen-Shannon divergence is 0.0092 bits. The entropy of P is 1.4855 bits and the cross-entropy is 1.5219 bits, which is 1.4855 + 0.0365.
Why it matters
KL divergence is at the heart of statistics and machine learning: it is the quantity that maximum likelihood fitting minimises, it shows how much a model loses against reality and it measures the information gained when you update a prior to a posterior. Seeing it for small examples builds intuition for larger ones.
Good to know
The calculator handles three categories only, and rescales your weights to probabilities. The values depend on the log base: bits for base 2, nats for the natural log. With very small weights in Q, the divergence can grow fast. Real data with many categories needs smoothing to avoid zero probabilities.
Related tools
The Shannon entropy calculator measures the uncertainty in one distribution, the Bayes theorem calculator updates beliefs, the chi-square distribution calculator tests how well counts fit a model, and the Matthews correlation coefficient calculator scores a classifier.