[OSM-dev] planet.osm

[Nick Whitelegg]

Sent by:        dev-bounces at openstreetmap.org
To:     dev at openstreetmap.org
cc:      
Subject:        Re: [OSM-dev] planet.osm

>> I believe my UTF8sanitizer filters out bad UTF8:
>> <www.lysator.liu.se/~jonass/UTF8sanitizer.c>. May be hard to use
>> unless you are used to unix & C & and compiling.
>>
>> /Jonas

>Ok it passed the expat parsing!
>To import the entire file with no bounding-box reduction on a machine 
with
>low memory i used the db to lookup for existents node and segments.
>This slows the import process. On a dualcore 3.0 ghz the import was about 
18
>hours long on postgres. I think i can achieve better results with mysql, 
but
>the problem will be presented again while data continues to grow.
>We should begin to think at a binary planet.osm too.
>I'm just trying planetosm-to-db.sql too , top reports it's using 1.6Gb of 
ram!

Do you mean .pl? It stores every node in memory so for a whole planet that 
might be a problem.

However I have created a hacked version of planetosm-to-db.pl which allows 
you to extract data within a given bounding box. It still needs to keep 
nodes and segments within the bounding box in memory (the only way to tell 
whether to accept a segment is to test whether the 2 nodes are in the 
bounding box) but if the bounding box is, say 5 by 10 degrees it should be 
OK (I'm just running it on the UK now).

I'll commit this to subversion later today.

Nick