Two variables can be linked in ways that a correlation coefficient misses, and information theory has a measure for any kind of link. The mutual information calculator takes the four counts of a 2 × 2 table and gives the mutual information in bits, the entropy of X, the entropy of Y, the normalised mutual information and the share of the uncertainty about Y removed by knowing X.
How the calculation works
- Fill in the 2 × 2 table of counts for two yes-or-no variables X and Y: both present, only X, only Y and neither.
- The calculator turns the counts into joint and marginal probabilities.
- The mutual information is I(X;Y) = Σ p(x,y) × log₂( p(x,y) / (p(x) × p(y)) ), measured in bits. It is 0 when X and Y are independent.
- Read the entropies of X and Y, the normalised mutual information, which runs from 0 to 1, and the share of the uncertainty about Y that disappears once you know X.
Example
Out of 100 cases, X and Y are both present in 40, only X in 10, only Y in 15 and neither in 35. X is present half of the time, so its entropy is 1.000 bit, and Y is present 55 % of the time, so its entropy is 0.993 bit. The mutual information is 0.1912 bit. The normalised value is 0.192, and knowing X removes 19.3 % of the uncertainty about Y.
Why it matters
Mutual information is used to pick the most informative features in machine learning, to measure how much a symptom tells about a disease, how much a word predicts a topic, or how much a signal carries through a noisy channel. It gives a unit, the bit, that is easy to compare across problems.
Good to know
The calculator handles two binary variables from a table of counts, with the plug-in estimate, which is biased upward for small samples. It does not test for significance, and it treats the counts as exact. For continuous variables or more categories, you must bin the data first, and the result depends on the binning. Use it as a descriptive measure.
Related tools
The KL divergence calculator measures the gap between two distributions, the chi-square test calculator tests independence in a table, the correlation significance calculator tests a linear link, and the confidence interval calculator measures uncertainty.