Floating-point numbers store a fixed number of significant bits, so they cannot represent every value. The floating-point precision calculator takes a format (half, single or double precision) and a number and gives the decimal digits of precision, the rounding error, the range, the largest exact whole number, the gap between neighbouring numbers near yours and the storage size.
How the calculation works
- Decimal digits of precision = significand bits × log10(2). Single precision has a 24-bit significand (including the hidden bit), which gives about 7.2 digits.
- Rounding error of 1.0 = 2^(1 − significand bits), the gap between 1.0 and the next number. It is shown as a power of ten.
- Largest finite value is about 2^(2^(e − 1)), where e is the number of exponent bits, and the smallest normal value is 2^(2 − 2^(e − 1)). Both are shown as powers of ten.
- Gap between neighbouring numbers near your value = 2^(floor(log2 value) − (significand bits − 1)). It grows with the size of the number.
Example
In single precision (24 significand bits, 8 exponent bits) numbers have about 7.22 decimal digits of precision. The relative rounding error is about 10^−6.92, and the largest finite value is about 10^38.53. Whole numbers are stored exactly up to 16,777,216. Near 1,000,000 the gap between two neighbouring numbers is 2^(19 − 23) = 0.0625, so a value such as 1,000,000.01 cannot be stored.
Why it matters
Rounding errors cause real bugs in finance code, simulations, games and sensor processing. Knowing how many digits a format really holds, and how large the gaps become for big values, helps you pick the right type and decide how to compare numbers.
Good to know
The calculator follows the IEEE 754 binary formats, which are used by nearly all processors and languages. Subnormal numbers extend the range below the smallest normal value at the cost of precision, and special values such as infinity and NaN are not covered. Powers of ten are shown as exponents because the values would not fit on the screen.
Related tools
The error propagation calculator shows how uncertainties combine in a calculation and the integer range calculator gives the limits of whole-number types.