Precision of the conversion calculations

As mentioned on the start page, there are two types of the conversion precisions: The double-precision floating-point and high precision calculation.

Double-precision floating-point

Technical details: Double-precision floating-point format is a computer number format that requires 64 bits in computer memory in binary format:

1 sign (+ or -)

11 bits for exponent

52 bits for fraction (52 digits would be the precision if we would display result in binary format). In decimal format: 252 = 4.503599627370496×1015. 15-digit precision is reduced to avoid rounding problems (from binary to decimal). Without this "reduction" some problems may appear like 0.1+0.2 = 0.30000000000000004 in JS.

Base info: Or maybe it is easier to understand over these examples:

decimal format exponential format
1234.5678901234 =0.12345678901234x104
123456789.01234 =0.12345678901234x109
0.00012345678901234 =0.12345678901234x10-3
1234567890123400 =0.12345678901234x1016

About floating-point, you can read more at mit.edu.

High precision

Base info: For conversions, where definition between source unit to base SI unit and base SI unit to destination unit is exactly defined, high precision, up to 800 decimal places can be chosen. This definition can be exact decimal number or fraction, for example metric foot is 1/3 m and not 0.33333333, this would be rounded number, no sense for high precision.

Technical details: High precision numbers are handled as textual value, converted into "big numbers" using bignumber.js, script by Michael Mclaughlin.

© 2017 conversion.org Terms of use