[OSM-dev] osm2pgsql killed when pending ways

[Lauri Kajan]

Thank you Frederik.

It was about "out of memory". I though have a 4GB swap partition.
I'll try again with smaller memory reservations. And actually without
coastline and hstore.
I also added the fsync=off.

How could I monitor the import process and memory usage so that I
could optimize the configurations?

Thanks. I'll report if this works or not.


On Wed, Jun 6, 2012 at 11:34 AM, Frederik Ramm <frederik at remote.org> wrote:
> Hi,
>
>
> On 06/06/2012 07:46 AM, Lauri Kajan wrote:
>>
>> I have some trouble to import osm data to my postgis database.
>> I am trying to import europe dump from gefabrik but the process was
>> killed.
>> Is this a memory issue or something else?
>
>
> Likely a memory issue; type "dmesg" and you will probably see a message
> about an out-of-memory condition that made Linux kill your process.
>
>
>> Is there any possibilities to continue the process from the point of
>> the crash? Or should I start all over again?
>
>
> I'm afraid you will have to start over.
>
>
>> I used latest revision 28427 of the osm2pgsql.
>> I have 8GB of RAM on my server and I used these configurations on my
>> postgresql:
>> shared_buffers = 1024MB
>> work_mem = 8MB
>> maintenance_work_mem = 1024MB
>> checkpoint_segments = 20
>> autovacuum = off
>> Are these settings a way off?
>
>
> They are ok but if you have Postgres 9.0 or higher, my experience is that
> smaller shared_buffers values work just as well. Also add "fsync=off".
>
>
>> And this was the command:
>> osm2pgsql --create --database osm --username osm --slim --cache 4096
>> --hstore --number-processes 2 -l --keep-coastlines europe.osm.pbf
>
>
> You will probably have to reduce the cache parameter to avoid the out of
> memory situation. (Having a little swap space configured on your system will
> also help - if you have swap space then exhausting physical memory will make
> your system slow down but not kill a process.)
>
> You could propbably speed up the process by
>
> * not keeping coastlines
> * modifying your default.style file to delete unwanted tags (with --hstore,
> a lot of rubbish like "xyz-import-uuid=some-long-hex-string" is imported
> into your database)
> * cut out a smaller area from the Europe file (using Osmosis) and import
> that, instead of all Europe
> * storing the database on an SSD, or if you don't have SSD but you have more
> than one disk, then on a Linux software RAID0 built from partitions on two
> or more disks
>
> Bye
> Frederik
>
> PS: A list of frequent but useless tags for adding to default.style, as
> generated by Sven Geggus. Works only with newer osm2pgsql versions that have
> wildcard support, and only makes sense when you use --hstore:
>
> node,way        3dshapes:*      text    delete
> node,way        AND_nosr_r      text    delete
> node,way        b5m:*   text    delete
> node,way        bak:*   text    delete
> node,way        canvec:*        text    delete
> node,way        created_by      text    delete
> node,way        dcgis:* text    delete
> node,way        FIXME   text    delete
> node,way        geobase:*       text    delete
> node,way        gnis:*  text    delete
> node,way        kms:*   text    delete
> node,way        KSJ2:*  text    delete
> node,way        massgis:*       text    delete
> node,way        mvdgis:*        text    delete
> node,way        naptan:*        text    delete
> node,way        ngbe:*  text    delete
> node,way        nhd:*   text    delete
> node,way        NHD:*   text    delete
> node,way        note    text    delete
> node,way        note:*  text    delete
> node,way        notes   text    delete
> node,way        osak:*  text    delete
> node,way        source  text    delete
> node,way        source:*        text    delete
> node,way        sourcedb:*      text    delete
> node,way        tiger:* text    delete
> node,way        WroclawGIS:*    text    delete
> node,way        yh:*    text    delete
> node,way        rednap:* text   delete
> node,way        ideewfs:* text  delete
> node,way        ign:* text      delete
>
> --
> Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09" E008°23'33"
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev