Skip to content
Omnikalc

Floating-point precision calculator: how accurate is a float?

Choose half, single or double precision and a number to get the decimal digits of precision, the range, the largest exact integer and the gap between neighbouring numbers.

Updated on
Your inputs

Value out of range

Results

Decimal digits of precision

7.22 digits

Rounding error of 1.0, as a power of ten
-6.92
Largest finite value, as a power of ten
38.53
Smallest normal value, as a power of ten
-37.93
Largest whole number stored exactly
16,777,216
Gap between neighbouring numbers near yours
0.062500
Storage size
4 bytes

Floating-point numbers store a fixed number of significant bits, so they cannot represent every value. The floating-point precision calculator takes a format (half, single or double precision) and a number and gives the decimal digits of precision, the rounding error, the range, the largest exact whole number, the gap between neighbouring numbers near yours and the storage size.

How the calculation works

  • Decimal digits of precision = significand bits × log10(2). Single precision has a 24-bit significand (including the hidden bit), which gives about 7.2 digits.
  • Rounding error of 1.0 = 2^(1 − significand bits), the gap between 1.0 and the next number. It is shown as a power of ten.
  • Largest finite value is about 2^(2^(e − 1)), where e is the number of exponent bits, and the smallest normal value is 2^(2 − 2^(e − 1)). Both are shown as powers of ten.
  • Gap between neighbouring numbers near your value = 2^(floor(log2 value) − (significand bits − 1)). It grows with the size of the number.

Example

In single precision (24 significand bits, 8 exponent bits) numbers have about 7.22 decimal digits of precision. The relative rounding error is about 10^−6.92, and the largest finite value is about 10^38.53. Whole numbers are stored exactly up to 16,777,216. Near 1,000,000 the gap between two neighbouring numbers is 2^(19 − 23) = 0.0625, so a value such as 1,000,000.01 cannot be stored.

Why it matters

Rounding errors cause real bugs in finance code, simulations, games and sensor processing. Knowing how many digits a format really holds, and how large the gaps become for big values, helps you pick the right type and decide how to compare numbers.

Good to know

The calculator follows the IEEE 754 binary formats, which are used by nearly all processors and languages. Subnormal numbers extend the range below the smallest normal value at the cost of precision, and special values such as infinity and NaN are not covered. Powers of ten are shown as exponents because the values would not fit on the screen.

The error propagation calculator shows how uncertainties combine in a calculation and the integer range calculator gives the limits of whole-number types.

Frequently asked questions

Why is 0.1 + 0.2 not exactly 0.3?

Computers store numbers in binary, and 0.1 and 0.2 have no exact finite binary form, just as 1/3 has no exact decimal form. Each is rounded to the nearest float, and the sum of the two rounded values lands on a neighbour of 0.3 and not on 0.3. Compare with a tolerance and not with equality.

What does the gap between neighbouring numbers mean?

Floats are spaced more widely as they grow, since the significand has a fixed number of bits. Between 2^k and 2^(k+1) the gap is constant, and it doubles at every power of two. For a large value, adding a small number can have no effect at all.

When should I use double precision?

Use double when you need more than 6 or 7 reliable digits, when you add many numbers of different sizes, or when errors build up over many steps. Single precision halves memory and suits graphics, machine learning and large arrays, as long as the accuracy is enough for the task.

Why is there a hidden bit?

In normal numbers the leading bit of the significand is always 1, so it is not stored. That gives 23 stored bits and 24 bits of precision in single precision, and 52 and 53 in double. The rule gives one free bit of precision.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.