Skip to content
Omnikalc

KL divergence calculator: how far is your model from the truth?

Enter two distributions over three categories to get the KL divergence in both directions, the Jensen-Shannon divergence and the cross-entropy.

Updated on
Your inputs

Value out of range

Value out of range

Value out of range

Value out of range

Value out of range

Value out of range

Results

KL divergence D(P ‖ Q)

0.0365 bits

Nearly identical distributions
KL divergence the other way, D(Q ‖ P)
0.0372 bits
Jensen-Shannon divergence
0.0092 bits
Cross-entropy H(P, Q)
1.5219 bits
Entropy of P
1.4855 bits

When you model data, you want to know how much is lost by using the wrong distribution. The KL divergence calculator takes a true distribution P and a model distribution Q over three categories, and gives the KL divergence in both directions, the Jensen-Shannon divergence, the cross-entropy and the entropy of P, all in bits.

How the calculation works

  • Enter the weights of the true distribution P for three categories. They can be probabilities or counts, since the calculator rescales them to sum to 1.
  • Enter the weights of the model distribution Q for the same three categories. Every weight must be positive.
  • The KL divergence is D(P ‖ Q) = Σ P × log₂(P ÷ Q), in bits. It measures the extra bits needed per symbol if you encode data from P using a code made for Q.
  • The calculator also gives the reverse divergence, the symmetric Jensen-Shannon divergence, the cross-entropy and the entropy of P. The cross-entropy equals the entropy plus the KL divergence.

Example

The true distribution is P = (0.5, 0.3, 0.2) and the model says Q = (0.4, 0.4, 0.2). The divergence is D(P ‖ Q) = 0.5 log₂(0.5 ÷ 0.4) + 0.3 log₂(0.3 ÷ 0.4) + 0.2 log₂(1) = 0.0365 bits. The reverse D(Q ‖ P) is 0.0372 bits, a little different, and the Jensen-Shannon divergence is 0.0092 bits. The entropy of P is 1.4855 bits and the cross-entropy is 1.5219 bits, which is 1.4855 + 0.0365.

Why it matters

KL divergence is at the heart of statistics and machine learning: it is the quantity that maximum likelihood fitting minimises, it shows how much a model loses against reality and it measures the information gained when you update a prior to a posterior. Seeing it for small examples builds intuition for larger ones.

Good to know

The calculator handles three categories only, and rescales your weights to probabilities. The values depend on the log base: bits for base 2, nats for the natural log. With very small weights in Q, the divergence can grow fast. Real data with many categories needs smoothing to avoid zero probabilities.

The Shannon entropy calculator measures the uncertainty in one distribution, the Bayes theorem calculator updates beliefs, the chi-square distribution calculator tests how well counts fit a model, and the Matthews correlation coefficient calculator scores a classifier.

Frequently asked questions

What is the KL divergence?

The Kullback-Leibler divergence measures how one probability distribution differs from another. D(P ‖ Q) is the average extra information, in bits when logs are base 2, needed to describe data from P with a code optimised for Q. It is zero if and only if the two distributions are the same.

Is it a distance?

No. It is not symmetric, since D(P ‖ Q) is generally not equal to D(Q ‖ P), and it does not satisfy the triangle inequality. The Jensen-Shannon divergence fixes the symmetry by comparing both distributions with their average. Its square root is a true distance.

What is cross-entropy?

The cross-entropy H(P, Q) is the average number of bits needed to code data from P with a code for Q. It equals the entropy of P plus the KL divergence, so minimising the cross-entropy against a fixed P is the same as minimising the KL divergence. It is the usual loss function for classification in machine learning.

Why must all weights be positive?

If Q gives zero chance to something that P allows, the divergence is infinite, because the model would call an observed event impossible. In practice, small floors or smoothing are used. The calculator requires a small positive weight for each category to avoid this.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.