[OSM-dev] Osm2pgsql failure with low-end server

[Andrew M. Bishop]

Martijn van Oosterhout <kleptog at gmail.com> writes:

> On 8 December 2011 11:13, Nick Whitelegg <Nick.Whitelegg at solent.ac.uk> wrote:
>>
>>>700 MB is a tiny machine, but then the Finland data set isn't that large
>>>either... it must be possible somehow ;)
>>
>> Incidentally why would it run out of memory in slim mode? I had the same
>> problem too some time ago when trying to import the whole of England, back
>> in the days before the county extracts.
>
> I did some research into this a while back and it has to do with the
> code that goes over all pending ways after the import, to deal with
> polygons. It looks like the number of pending ways is a lot more than
> it used to be. Osm2pgsql requests all the pending ways in a single
> query which fails on small machines.
>
> Can someone check if there is really the case. i.e. show the number of
> pending ways after a simple import.

With this command line:

osm2pgsql --create --database GIS --slim --cache 128 great_britain.osm

I get this output:

-------------------- osm2pgsql great_britain.osm --------------------
Reading in file: great_britain.osm
Processing: Node(32899k) Way(4022k) Relation(80855)  parse time: 3481s

Node stats: total(32899023), max(1541436207)
Way stats: total(4022307), max(140763013)
Relation stats: total(80855), max(1905258)

Going over pending ways
processing way (1639k)

Going over pending relations

node cache: stored: 8047817(24.46%), storage efficiency: 47.97%, hit rate: 25.52%
...

osm2pgsql SVN version 0.70.5
-------------------- osm2pgsql great_britain.osm --------------------

This runs on a VPS with ~512MB RAM (I think that this is the
guaranteed level but I have certainly used ~50% more at times).

-- 
Andrew.
----------------------------------------------------------------------
Andrew M. Bishop                             amb at gedanken.demon.co.uk
                                   http://www.gedanken.org.uk/mapping/