Shannon entropy measures how unpredictable a source is. A fair coin has one bit of entropy: each flip is a complete surprise. A coin that lands heads 99% of the time has almost none, because you can nearly always guess the result. Enter how often each outcome occurs, as counts or as weights, and the calculator returns the entropy in bits, along with the maximum possible value and how close the source comes to it.
How the calculation works
- Probability of each outcome = its count ÷ the total
- Entropy H = −Σ p × log₂(p), summed over the outcomes with p > 0
- Maximum entropy = log₂(k) for k outcomes that occur
- Efficiency = H ÷ maximum entropy
- Perplexity = 2^H, and nats = H × ln 2
Example
A source emits four symbols with counts of 4, 2, 1 and 1. Their probabilities are 50%, 25%, 12.5% and 12.5%. The entropy is 1.75 bits, against a maximum of 2 bits, so the efficiency is 87.5%. The perplexity is 3.364: the source is as unpredictable as a die with about 3.4 equally likely faces. An ideal code needs 1.75 bits per symbol on average, compared with 2 bits for a fixed-length code.
Where it is used
Data compression, cryptography, machine learning and linguistics all rely on entropy. Compression programs shrink files by giving short codes to frequent symbols. Security tools estimate how hard a password or key is to guess, and decision trees choose the questions that remove the most uncertainty. In ecology, a close relative measures species diversity.
Good to know
Entropy depends only on the probabilities, not on what the outcomes are. Outcomes with a zero count do not count. Estimating probabilities from a small sample tends to underestimate the true entropy, so collect enough observations. The result treats outcomes as independent: for text or sequences with patterns, the real entropy per symbol is lower than this single-symbol figure.
Related tools
For password guessing difficulty, try the password strength calculator. For ecological diversity, use the Shannon index calculator, and for counting arrangements, the combinations calculator.