[OSM-dev] osm2pgsql planet import time issues?

Jon Burgess jburgess777 at googlemail.com
Wed Jan 21 00:08:18 GMT 2009


On Wed, 2009-01-21 at 00:13 +0100, Ivo Brodien wrote:
> Hi Dave,
> 
> 
> 
> > You can check http://www.openstreetmap.org/stats/data_stats.html to
> > see the current object counts. It's about 24m ways.
> 
> 
> ahh. Thanks. Did not know this page.
> 
> 
> 
> 
> > If you don't need the whole planet then use an extract -- it goes a
> > lot quicker. Otherwise invest in some more RAM -- it helps! :-)
> 
> 
> I need the whole planet. So I guess I need to go for more RAM. My
> motherboard only likes max 4GB of RAM.
> 
> 
> > Yes, you can't now run it on a 32 bit system in non-slim mode
> > without
> > getting a segfault as you run out of address space.
> > In general give it as much cache as you can. I do it with -C 2000
> > and
> > it works (slowly, about 8-9 hours for the whole planet). But I have
> > 8GB of RAM to play with. With only 2GB you're going to have to go to
> > swap to complete the process anyway, so I'd just leave it on the
> > default.
> 
> 
> I did not get that 100%, sorry. So you mean I should leave the cache
> to the default of 800 MB or not use the "--slim" option? If I get you
> right, I should still use the --slim option (being on 32bit I have to)
> and then give it more RAM. If I have installed 4 GB how much could I
> use, 3GB? If you have 8GB why do you just use 2GB?


The main tile server imports the data with:
$ osm2pgsql --slim -C 2100

This results in a 91% cache hit rate so to cache all the nodes. 

node cache: stored: 269092844(90.87%), storage efficiency: 97.76%, hit
rate: 91.12%

An older version of the osm2pgsql code had a bug which meant the maximum
size was limited to 2048MB but that was fixed a while ago.

When I set the 2100 number that was enough to cache all the nodes giving
a 100% hit rate. Increasing the cache limit will make no difference once
you can hold all the nodes in the cache. 

If you have a 32 bit process then the maximum data it can hold in RAM is
somewhere around 3GB. You can achieve more by using special techniques
but 64bit CPUs and processes are the best solution.

	Jon






More information about the dev mailing list