[OSM-dev] osm2pgsql killed when pending ways

[Frederik Ramm]

Hi,

On 06/06/2012 07:46 AM, Lauri Kajan wrote:
> I have some trouble to import osm data to my postgis database.
> I am trying to import europe dump from gefabrik but the process was killed.
> Is this a memory issue or something else?

Likely a memory issue; type "dmesg" and you will probably see a message 
about an out-of-memory condition that made Linux kill your process.

> Is there any possibilities to continue the process from the point of
> the crash? Or should I start all over again?

I'm afraid you will have to start over.

> I used latest revision 28427 of the osm2pgsql.
> I have 8GB of RAM on my server and I used these configurations on my postgresql:
> shared_buffers = 1024MB
> work_mem = 8MB
> maintenance_work_mem = 1024MB
> checkpoint_segments = 20
> autovacuum = off
> Are these settings a way off?

They are ok but if you have Postgres 9.0 or higher, my experience is 
that smaller shared_buffers values work just as well. Also add "fsync=off".

> And this was the command:
> osm2pgsql --create --database osm --username osm --slim --cache 4096
> --hstore --number-processes 2 -l --keep-coastlines europe.osm.pbf

You will probably have to reduce the cache parameter to avoid the out of 
memory situation. (Having a little swap space configured on your system 
will also help - if you have swap space then exhausting physical memory 
will make your system slow down but not kill a process.)

You could propbably speed up the process by

* not keeping coastlines
* modifying your default.style file to delete unwanted tags (with 
--hstore, a lot of rubbish like "xyz-import-uuid=some-long-hex-string" 
is imported into your database)
* cut out a smaller area from the Europe file (using Osmosis) and import 
that, instead of all Europe
* storing the database on an SSD, or if you don't have SSD but you have 
more than one disk, then on a Linux software RAID0 built from partitions 
on two or more disks

Bye
Frederik

PS: A list of frequent but useless tags for adding to default.style, as 
generated by Sven Geggus. Works only with newer osm2pgsql versions that 
have wildcard support, and only makes sense when you use --hstore:

node,way        3dshapes:*      text    delete
node,way        AND_nosr_r      text    delete
node,way        b5m:*   text    delete
node,way        bak:*   text    delete
node,way        canvec:*        text    delete
node,way        created_by      text    delete
node,way        dcgis:* text    delete
node,way        FIXME   text    delete
node,way        geobase:*       text    delete
node,way        gnis:*  text    delete
node,way        kms:*   text    delete
node,way        KSJ2:*  text    delete
node,way        massgis:*       text    delete
node,way        mvdgis:*        text    delete
node,way        naptan:*        text    delete
node,way        ngbe:*  text    delete
node,way        nhd:*   text    delete
node,way        NHD:*   text    delete
node,way        note    text    delete
node,way        note:*  text    delete
node,way        notes   text    delete
node,way        osak:*  text    delete
node,way        source  text    delete
node,way        source:*        text    delete
node,way        sourcedb:*      text    delete
node,way        tiger:* text    delete
node,way        WroclawGIS:*    text    delete
node,way        yh:*    text    delete
node,way        rednap:* text   delete
node,way        ideewfs:* text  delete
node,way        ign:* text      delete

-- 
Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09" E008°23'33"