[OSM-dev] Gosmore

[Robert (Jamie) Munro]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've found it impossible to get "gosmore --rebuild" to complete a full
planet with more recent versions of gosmore. It starts of taking lots of
processor time (which is a good sign) - a lot of memory is used, but is
paged out, and it continues happily, but once it starts swapping, cpu
usage falls to almost nothing and it just swaps for many hours until I
give up. Is there any way that this could be improved? I'm using a
powerPC mac with 512Mb of memory.

I'm not sure of the terminology of my suggestions below, but I hope they
make some sense. I may be way off with them.

AFAIK, it is doing some kind of bucket sort of the data. Perhaps it
could do 2 layers of buckets - first a small course bucket that will fit
into memory, even on a low memory machine, then once one of those
buckets is full, it could sort it to the final buckets, and then only
have to page in the small buckets that are part of that larger bucket.

This is just a guess, but it may be doing something like storing the
pointer to the end of the bucket at the start of the bucket, so it
always needs to page in the start of every bucket, even when the bucket
is being filled on the next page. If it kept the pointers to the end of
the buckets together in one page of memory at the beginning, it would
only need to page in that memory and the part of the bucket that is
being filled, which may reduce swapping considerably.

Robert (Jamie) Munro

Ps. I've also written a few comments on
http://wiki.openstreetmap.org/index.php/Talk:Gosmore
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGk7idz+aYVHdncI0RAlvLAKDCa5PxY0xNp9hzWg0cnwkZQswKyACgtTWP
RAACI26zCs4uylK9TkK++C8=
=ejuz
-----END PGP SIGNATURE-----