[OSM-dev] Memory error while converting osm to gml

Brett Henderson brett at bretth.com
Thu Apr 2 23:25:45 BST 2009


Stefan de Konink wrote:
> Does anyone have any timings on importing the current planet.osm in 
> PostgreSQL with any tool available? Using lets say 8GB of ram and 
> 'typical' disk?
>   
I don't have full numbers but I do have some figures which may be useful.

Early in January Jochen Topf did some timings with Osmosis, the osmosis 
pgsql schema and a Europe osm extract (between 1 - 1.5 GB compressed).  
I believe his machine had 8GB RAM, I don't know other details.

We were testing the timings for creating PostgreSQL "copy" files which 
can then be bulk loaded into the database.

The osmosis schema has some optional components, an "action" table, a 
linestring column on the way table, and a bbox column on the way table.  
The linestring and bbox columns are more expensive to populate due to 
the need to look up node locations when processing ways.

The baseline measurement was the basic schema (ie. no geometry columns 
on the way table) which took 16 minutes to process.
Creating geometry columns requires approx 5GB of temporary storage 
during processing to hold node locations.
Using a simple node location file using random access lookups (can't 
mmap a 5GB file on 32-bit Java) took 29 minutes.
Using a 64-bit JVM with an in-memory temporary node store took 18 minutes.

So producing geometry columns on the way table only adds a small 
overhead if you have a 64-bit JVM and a minimum of 6GB RAM.  If you have 
less RAM and/or a 32-bit JVM, you need to use the temp file 
implementation which is much slower due to disk seeking.

The numbers above don't include the bulk import to the database itself, 
and will also need to be multiplied by 5'ish times to reach the full 
planet volume.

Brett





More information about the dev mailing list