[OSM-dev] 120111 Planet Node Order?

Lynn W. Deffenbaugh (Mr) ldeffenb at homeside.to
Sun Jan 15 02:06:49 GMT 2012


On 1/14/2012 7:56 PM, Simon Poole wrote:
> Believe me, if you can't allocate enough memory (you should be running 
> a 64bit OS and have at least enough swap allocated if you don't have 
> sufficient memory) for the cache for the import in question, it is 
> going to be very very very very slow.

With System Monitor reporting only 53% memory in use, I don't think 
memory is the reason for the slow Way import.  The 1.329 billion nodes 
imported reasonably consistently and memory use hasn't changed with the 
transition from Nodes to Ways.  I really think it has more to do with 
the "out of order node" error from the planet file, but google hasn't 
helped me find any reference to what that really means.

>
> Don't be confused by the Florida or whatever import running fast, that 
> is peanuts in comparison to importing a full planet (1.4 billion nodes 
> or so).

Understood.  I did the Florida import just to make sure the tool-chain 
worked.  Didn't want to run the whole planet just to test it.

Lynn (D)

>
> Simon
>
>
>
> "Lynn W. Deffenbaugh (Mr)" <ldeffenb at homeside.to> schrieb:
>
>     Actually, the instructions I was following called for -C 2048
>     (http://weait.com/content/build-your-own-openstreetmap-server), but when
>     I did that I got the following error (running in a 4GB Ubuntu 10.04 VM):
>
>     >  Out of memory for dense node cache, reduce --cache size
>
>     I had to reduce to -C 1800 to make it even run.  Nodes imported nice and
>     fast (43.6k/s), but I received the following error:
>
>     >  Out of order node 244067335 (33792779,7) - this will impact the cache
>     >  efficiency
>
>     which to my novice reading implies that there was an issue in the Planet
>     file dealing with node ordering.  And the Way import is running MUCH
>     slower than I would have thought given what I've read.  0.12k/s is WAY
>     less than 43.6k/s.  I know ways are slower, but I
>     figured maybe 10% of
>     the speed worst case, not this much slower.
>
>     Lynn (D)
>
>     PS.  I haven't seen a -C 12000 recommended anywhere.  And interestingly
>     with -C 2048 complaining , -C 1800 is still only using 2GB of the 3.7GB
>     available according to Ubuntu's System Monitor.
>
>     On 1/14/2012 6:10 PM, Simon Poole wrote:
>     >
>     >  You do have -C 12000 set?
>     >
>     >  Haven't seen the error message before, but if you set the node cache
>     >  smaller than necessary to store -all- nodes importing will be very
>     >  slow (naturally if you don't have enough memory the machine will swap,
>     >  but it sill still be faster).
>     >
>     >  Simon
>     >
>     >  Am 14.01.2012 23:16, schrieb Lynn W. Deffenbaugh (Mr):
>     >>  Greetings,
>     >>
>     >>  I've been lurking in the group for a long time now and finally got
>     >>  motivated to try putting together my own Tile server.  I go!
>       t
>     >>  everything working, the whole way through mod-tile using a Florida
>     >>  extract as my data to keep things moving along.  It was good.
>     >>
>     >>  Then I downloaded planet-120111.osm.bz2 and started importing it
>     >>  yesterday.  It completed the nodes and is now working on the Ways,
>     >>  but I received on error message:
>     >>
>     >>  >  Out of order node 244067335 (33792779,7) - this will impact the
>     >>  cache efficiency
>     >>
>     >>  And the Ways are processing much more slowly than I would have
>     >>  expected, namely:
>     >>
>     >>  >  Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0
>     >>  0.00/s)
>     >>
>     >>  Has anyone else imported this particular planet file and is the slow
>     >>  Way loading to be expected after that error?
>     >>
>     >>  Would I be better off in scrapping this slow load a!
>       nd going
>     back
>     >>  another week to download the Planet and then apply the diffs to bring
>     >>  it up to date?
>     >>
>     >>  Lynn (D) -http://aprsisce.wikidot.com/ - A user of OSM data
>     >>
>     >>
>     >>
>     >>
>     >>
>     ------------------------------------------------------------------------
>
>     >>  dev mailing list
>     >>  dev at openstreetmap.org
>     >>  http://lists.openstreetmap.org/listinfo/dev
>     >
>     >
>     >
>     ------------------------------------------------------------------------
>
>     >  dev mailing list
>     >  dev at openstreetmap.org
>     >  http://lists.openstreetmap.org/listinfo/dev
>     >
>
>
>     ------------------------------------------------------------------------
>
>     dev mailing list
>     dev at openstreetmap.org
>     http://lists.openstreetmap.org/listinfo/dev
>
>
> -- 
> Sent from my Galaxy Tab with Kaiten Mail. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20120114/1290338f/attachment-0001.html>


More information about the dev mailing list