[Routing] Gosmore test results

[Lambertus]

Nic Roets wrote:
> The bottleneck with gosmore routing a.t.m. is that hashing business.
> Sometimes it allocates way too much memory, resulting in "thrashing" and
> cache misses. I always guessed that the only way to get rid of of the
> hashing, would be to have like 20GB of RAM.
> 
> But after this thread, I invented a new way of doing it that uses a more
> dynamic way of memory allocation.
> 
> So I implemented it, but could not see any difference with the 26MB South
> African gosmore.pak file. If someone (Lambertus) wants to try it out on a
> larger .pak file, here are some pointers :
> * Check out of SVN
> * make CFLAGS="-O2 -DROUTE_SRV"
> * On 32 bit machines it will only work with smaller .pak files, say smaller
> than 500MB. 64 bit machines should have no limit and may just be able to
> route from NY to LA in a reasonable time.
> * Potentially needing a lot of free disk space. See tmpfile()
> 
It took a while to rebuild the Gosmore databases with last Wednesdays 
planet file, but I've done some tests now. The Gosmore instance with new 
compiler option is available 'behind' the test layer on 
yournavigation.org. The older Gosmore is available behind the other layers.

Philadelphia to Boston using test layer returns no route. Looking closer 
reveals that Gosmore needs at least 10 minutes (after which I killed it) 
and all the available RAM (I've seen it using 2.6 GB).

The older Gosmore needs 35 seconds the first run and around 18 seconds 
for consecutive runs. This has probably something to do with the limited 
RAM (3 GB) and the big Gosmore database files (4.7 GB for America,
2.1 GB for the rest). Having a machine with at least 8 GB RAM would 
cause the first run to be as fast as the consecutive ones as well).

Unfortunately this test doesn't tell us anything about the speed 
difference between both engine versions, so another test.

New York to Boston uses 9.3 seconds using the test version (6.5 on 
consecutive runs) and 8.3 using the older one (7.2 on consecutive runs).

New York to Los Angeles can't be calculated using the older Gosmore 
version (Gosmore returns JUMP and a partial route). This is a strange 
behavior that I've seen before: Gosmore can't calculate a whole route 
but if you manually divide the route in shorter sections then there's no 
problem at all, so it's not a data error.

The test Gosmore version keeps calculating and slowly eating all the ram:

PID   USER      PR  NI VIRT  RES  SHR  S %CPU %MEM    TIME+  COMMAND
18568 lambertu  26  10 21.0g 2.6g 2.6g D  0.7 89.8   1:23.17 gosmore

The performance is almost completely disk-bound with this new version 
(confirmed by munin). After running for an hour or so I killed it.

Conclusion: In some cases it might help a bit, but for now the older 
Gosmore is a more stable performer. Then there's also the issue with 
extra long routes where Gosmore seems to give up.