[OSM-dev] Tiling the Planet and Missing Nodes

[Ben Supnik]

Hi Stefan,

Stefan de Konink wrote:
> How do you handle the LargeFile allocation? Because I tried mmaping on 
> some obscure 32 userspace valgrind 64 kerner lately, and that was prone 
> to failure.

It's a (very crude) "windowing" system - a smaller chunk of the file is 
read in, scanned, and scooted down.  If a single XML node were larger 
than the window, it would fail.  But...even in 32 bits you can make a 
very large window - I think we are unlikely to have a single tag be > 1 
GB, for example. :-)

In the long term I want to switch from my hacky crude lousy XML parser 
to expat...expat runs through XML at about half the speed mine does, but 
then it is a real correct robust XML parser whereas I am simply grepping 
blindly for strings (and wouldn't handle legal small changes in char 
spacing well).

Run time was about 18 hours last time I ran it...the biggest limit is 
the number of open file descriptors - only 1024 on my Mac.  Currently I 
do multiple passes over the input file to avoid open/closing output 
files...the locality of output was really poor when I examined the 
problem.

The program uses gzip on both input/output to let someone work entirely 
in terms of compressed XML.  It could easily be converted to bz2.

> Welcome in the fantastic world of OSM :D I thought had fixed all those 
> instances two days ago :) Cool you found another one.

At least - it only finds one and dies right now. :-)

> The idea with that situation is for ways:
> - Download the way; and just reupload it, then it will be solved in the 
> next run (because the way download hides invisible nodes)

Okay - so simply omitting the node would be a reasonable thing to do. 
Thanks!!

cheers
ben

-- 
Scenery Home Page: http://scenery.x-plane.com/
Scenery blog: http://xplanescenery.blogspot.com/
Plugin SDK: http://www.xsquawkbox.net/xpsdk/
X-Plane Wiki: http://wiki.x-plane.com/
Scenery mailing list: x-plane-scenery at yahoogroups.com
Developer mailing list: x-plane-dev at yahoogroups.com