Skip to content
Omnikalc

How unpredictable is your data source?

Enter the counts or probabilities of up to six outcomes: get the Shannon entropy in bits, the maximum entropy, the efficiency and the perplexity.

Updated on
Your inputs

Value out of range

Value out of range

Value out of range

Value out of range

Value out of range

Value out of range

Results

Shannon entropy

1.7500 bits

A moderately uneven distribution
Maximum entropy for these outcomes
2.0000 bits
Efficiency (entropy ÷ maximum)
87.5%
Perplexity (equivalent equally likely outcomes)
3.364
Entropy in nats
1.2130 nats

Shannon entropy measures how unpredictable a source is. A fair coin has one bit of entropy: each flip is a complete surprise. A coin that lands heads 99% of the time has almost none, because you can nearly always guess the result. Enter how often each outcome occurs, as counts or as weights, and the calculator returns the entropy in bits, along with the maximum possible value and how close the source comes to it.

How the calculation works

  • Probability of each outcome = its count ÷ the total
  • Entropy H = −Σ p × log₂(p), summed over the outcomes with p > 0
  • Maximum entropy = log₂(k) for k outcomes that occur
  • Efficiency = H ÷ maximum entropy
  • Perplexity = 2^H, and nats = H × ln 2

Example

A source emits four symbols with counts of 4, 2, 1 and 1. Their probabilities are 50%, 25%, 12.5% and 12.5%. The entropy is 1.75 bits, against a maximum of 2 bits, so the efficiency is 87.5%. The perplexity is 3.364: the source is as unpredictable as a die with about 3.4 equally likely faces. An ideal code needs 1.75 bits per symbol on average, compared with 2 bits for a fixed-length code.

Where it is used

Data compression, cryptography, machine learning and linguistics all rely on entropy. Compression programs shrink files by giving short codes to frequent symbols. Security tools estimate how hard a password or key is to guess, and decision trees choose the questions that remove the most uncertainty. In ecology, a close relative measures species diversity.

Good to know

Entropy depends only on the probabilities, not on what the outcomes are. Outcomes with a zero count do not count. Estimating probabilities from a small sample tends to underestimate the true entropy, so collect enough observations. The result treats outcomes as independent: for text or sequences with patterns, the real entropy per symbol is lower than this single-symbol figure.

For password guessing difficulty, try the password strength calculator. For ecological diversity, use the Shannon index calculator, and for counting arrangements, the combinations calculator.

Frequently asked questions

What is Shannon entropy?

A measure of the average uncertainty, or surprise, in a source of information. It is the average number of yes-or-no questions needed to identify an outcome, using the best possible questions.

Why is the unit bits?

The calculation uses logarithms in base 2. With base e you get nats, and with base 10 you get hartleys.

What is the maximum entropy?

The entropy when all outcomes are equally likely. For k outcomes it equals the base-2 logarithm of k, so four outcomes give 2 bits.

What does it have to do with compression?

Entropy is the lower limit of the average number of bits per symbol that any lossless code can reach. A source with an entropy of 1.75 bits cannot be compressed below that on average.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.