[OSM-dev] Memory error while converting osm to gml
brett at bretth.com
Thu Apr 2 23:25:45 BST 2009
Stefan de Konink wrote:
> Does anyone have any timings on importing the current planet.osm in
> PostgreSQL with any tool available? Using lets say 8GB of ram and
> 'typical' disk?
I don't have full numbers but I do have some figures which may be useful.
Early in January Jochen Topf did some timings with Osmosis, the osmosis
pgsql schema and a Europe osm extract (between 1 - 1.5 GB compressed).
I believe his machine had 8GB RAM, I don't know other details.
We were testing the timings for creating PostgreSQL "copy" files which
can then be bulk loaded into the database.
The osmosis schema has some optional components, an "action" table, a
linestring column on the way table, and a bbox column on the way table.
The linestring and bbox columns are more expensive to populate due to
the need to look up node locations when processing ways.
The baseline measurement was the basic schema (ie. no geometry columns
on the way table) which took 16 minutes to process.
Creating geometry columns requires approx 5GB of temporary storage
during processing to hold node locations.
Using a simple node location file using random access lookups (can't
mmap a 5GB file on 32-bit Java) took 29 minutes.
Using a 64-bit JVM with an in-memory temporary node store took 18 minutes.
So producing geometry columns on the way table only adds a small
overhead if you have a 64-bit JVM and a minimum of 6GB RAM. If you have
less RAM and/or a 32-bit JVM, you need to use the temp file
implementation which is much slower due to disk seeking.
The numbers above don't include the bulk import to the database itself,
and will also need to be multiplied by 5'ish times to reach the full
More information about the dev