[Openstreetmap-dev] Server is truncating lat/lon values?

Immanuel Scholz immanuel.scholz at gmx.de
Mon Jan 23 08:55:22 GMT 2006


Hi again,

> Is it possible, that values are rounded somewhere in the import process?
> What is the precision of latitude and longitude values? 32-bit-float? Or
> 64-bit-double?

I want to add some math.. I hope I am wrong, so please correct me! :)

I want to verify whether float precision is good enough for
latitude/longitude values.


Lets look at longitude, which can go from -180 to 180.

- 32-Bit-Float has a mantissa of 23 Bit.
- 23 Bit is up to 8 Meg., which guarantees a precission of 6 decimal digits.
- Longitude has 3 digits before the dot -> 3 digits precision left after dot.
- So we get a maximum guaranteed precision of 0.001°
- 40000km perimeter of the earth divided by 360° are roughly 111km/°.
- So we have a precision of 111m (per 0.001°).

I think 111 Meter is far too inaccurate for GPS data. So we need double.
Correct?


Ciao, Imi.

PS: Even if we assume to have the seventh digit of the 23-Bit Mantissa as
precision, since the first digit of longitude never goes above 8 (I am not
sure, we can assume this), then we are left with 11 Meter precision, which
is still unsatisfying.






More information about the dev mailing list