[OSM-dev] 120111 Planet Node Order?

[Lynn W. Deffenbaugh (Mr)]

On 1/15/2012 5:09 AM, Simon Poole wrote:
>
> current versions of osm2pgsql can import .pbf, so you are on the right 
> track there. Richards text is good, but it was written when planet 
> files were a third of the current size and is just a bit dated. If 
> osm2pgsql can't hold the nodes in it's cache it will have to retrieve 
> them from the database and that is in my and other peoples experience 
> at least  an order of magnitude slower.

Any pointers on the osm2pgsql command line differences for .pbf vs 
.bz2-compressed OSM data?  With the call for people to quit accessing 
the OSM tile servers, this really does need to be better, or more 
currently, documented.  I managed to get one running, but apparently 
there's better ways to do it now.

> To give you current numbers, I just did (10 days ago) an import on a 
> i7 2600, 16GB box that took 30 hours with the initial import phase 
> running at
>
> Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) 
> Relation(1243830 58.30/s)  parse time: 28598s
>
> Node stats: total(1322468982), max(1576326287) in 3887s
> Way stats: total(120291564), max(144049709) in 3375s
> Relation stats: total(1243833), max(1951174) in 21335s

THAT is the kind of data I was looking for and haven't noticed.  So your 
Ways are about 10% of the rate of the Nodes, and Relations are about 
twice as fast as the Ways.  and there are very few Relations by 
comparison.  There is definitely something different in my nodes going 
at 45K/s and my ways going at 0.12k/s.  I've since aborted that run and 
restarted with some VM changes, but it looks like I've got a ways to go yet.

>
> Because the box has limited memory I did an earlier attempt with -C 
> 8000 because I knew that it would swap a lot with -C 12000. However 
> that ran at roughly 4k/s ways and after a couple of hours I aborted 
> it. The good news is that you -can- import on a machine with less 
> memory (good luck keeping up with the updates on a VM though), for 
> example lonvia has imported on a machine with 6GB total (but with -C 
> 12000).
>
> The error message you got is weird and may point to an issue with the 
> export process, but it just states that the node cache is going to be 
> less efficient space wise, so IMHO you can simply ignore that.

I'm also downloading the previous week's planet file just to see if my 
configuration goes faster on the Ways without the error, assuming I 
don't get the Node order error on the previous week's planet file.

>
> So add some swap and try again :-)

I'll give that a go when the current run gets to the Ways and I see what 
the rate turns out to be.  I'm at 705000k Nodes right now, so not too 
much longer.  But I'm only getting 30.9k/s rate instead of my previous 
45 so I think some of my change in the VM had less-than-desirable effects.

Lynn (D)

>
> Simon
>
> Am 15.01.2012 02:57, schrieb Lynn W. Deffenbaugh (Mr):
>> I'm a rank amateur at this, can you provide a link on how to use (or 
>> what to use instead of) osm2pgsql to import from a pbf instead of 
>> planet-120111.osm.bz2?  .pbfs are not mentioned at 
>> http://weait.com/content/build-your-own-openstreetmap-server which is 
>> the best reference I've found on getting a tile server running for 
>> the planet.
>>
>> Lynn (D)
>>
>> On 1/14/2012 8:06 PM, Simon Poole wrote:
>>> Further tip: use the .pbf Files. It won't help a lot with your 
>>> current issue, but is quite a bit faster.
>>>
>>> Simon
>>> -- 
>>> Sent from my Galaxy Tab with Kaiten Mail. 
>>
>>
>> _______________________________________________
>> dev mailing list
>> dev at openstreetmap.org
>> http://lists.openstreetmap.org/listinfo/dev
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>