[OSM-dev] osm2pgsql -C vs --flat-nodes

[Lynn W. Deffenbaugh (Mr)]

BTW, I believe you can access my OSM machine's munin graphs at:

http://ldeffenb.dnsalias.net:6360/munin/localdomain/localhost.localdomain/index.html

If this doesn't work, please let me know and I'll see what I've not got 
configured correctly.

Lynn (D) - KJ4ERJ

On 9/19/2012 9:38 AM, Lynn W. Deffenbaugh (Mr) wrote:
> Greetings developers,
>
> I'm in the process of attempting to load the newly licensed planet and 
> have recently learned about --flat-nodes in osm2pgsql.  I'm trying to 
> make use of this feature to reduce the disk space consumption of the 
> --slim updatable database, but I'm having issues getting enough memory 
> allocated to my VMware virtual machine to complete the import.  It 
> runs out of memory querying the pending_ways.
>
> I've looked through the code and it appears that using a -C 14000 in 
> conjunction with --flat-nodes may be redundant as they're both 
> attempting to speed up access to a node's coordinates, the -C by 
> keeping it completely in RAM and --flat-nodes doing a RAM-based cache 
> of 10,000 disk-based blocks of 1024 nodes each.  Granted the -C 14000 
> manages to hold all 1,569,263k nodes in RAM (at 98.9% full) while the 
> --flat-nodes will only hold 10,240k nodes in RAM, so I can expect a 
> (significant?) slowdown, but...
>
> Can I dramatically reduce, or nearly eliminate the -C node cache and 
> let the --flat-nodes pick up the slack for the planet import? Will 
> this work?  And will it be nearly fast enough to be reasonable?
>
> Lynn (D) - KJ4ERJ
>
> PS.  I've got a 6 core VM with 24, 28, and then 32GB of RAM hosted on 
> an 8 core i7 with only 28G of physical RAM.  I know I'm paging the 
> VM.  Disk configuration is one virtual drive for the root and 3 
> virtual drives (each on a different physical spindle) lashed as a 
> RAID0 array for the gis DB.  I'm using the following import command:
>
> osm2pgsql --slim -d gis -C 14000 --number-processes 6 --flat-nodes 
> /mnt/SSD/flatnodes/flatnodes.osm /mnt/raid0/planet/planet-120912.osm.pbf
>
> PPS.  I started with bunzip2 -c 
> /mnt/raid0/planet/planet-120912.osm.bz2 |  osm2pgsql ... /dev/stdin, 
> but WOW is the PBF faster, especially on the node portion with a 
> 107k/sec node rate for the bz2 and 807k/sec for the pbf.
>
> PPPS.  I know I need SSD, but that's not in the $$$ picture at the 
> moment.  After the planet import is complete, I move the rendering 
> tables onto an SSD, but I'm not sure how to tell osm2pgsql that I'd 
> like those tables created in the alternate dataspace (called, 
> appropriately, SSD in postgresql).
>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>