[OSM-dev] osm2pgsql failing with low cache size

[Jukka Rahkonen]

Hi,

I am running osm2pgsql on my home computer and then I use dump and restore
for updating the tables on the little remote server.

-Jukka Rahkonen-

Michael Corey wrote:
> Hello all:
>
> Maybe this just isn't going to work, but I'm trying to use osm2pgsql to
> process a small set of XML that I've downloaded through the Overpass API.
>
> I'm running this on an EC2 micro instance, which supposedly has up to
> 613 MB of RAM. That's not a lot, but the XML I'm trying to import is
> only 23 ways made up of 191 nodes. This will grow, but not by orders of
> magnitude.
>
> So I'm running:
>
> osm2pgsql ./data/latest.xml -d border_processing -S
> /home/border/osm_scraper/mystyle.style --slim --cache 128
> --extra-attributes
>
> Which fails, returning at the end:
>
> Allocating memory for dense node cache
> Allocating dense node cache in one big chunk
> Allocating memory for sparse node cache
> Sharing dense sparse
> Node-cache: cache=128MB, maxblocks=16385*8192, allocation method=11
> Mid: pgsql, scale=100 cache=128
> Setting up table: planet_osm_nodes
> NOTICE:  table "planet_osm_nodes" does not exist, skipping
> NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index
> "planet_osm_nodes_pkey" for table "planet_osm_nodes"
> Setting up table: planet_osm_ways
> NOTICE:  table "planet_osm_ways" does not exist, skipping
> NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index
> "planet_osm_ways_pkey" for table "planet_osm_ways"
> Setting up table: planet_osm_rels
> NOTICE:  table "planet_osm_rels" does not exist, skipping
> NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index
> "planet_osm_rels_pkey" for table "planet_osm_rels"
>
> Reading in file: ./data/latest.xml
> StartElement: Unknown element name: note
> Unknown node type 3
> EndElement: Unknown element name: note
> StartElement: Unknown element name: meta
> EndElement: Unknown element name: meta
> Segmentation fault (core dumped)
>
> If I up the cache to anything over 500 MB then I run out of memory on
> the EC2 instance.
>
> How much memory does osm2pgsql need for something seemingly small like
> this? Can I reduce the load significantly by cutting out other
> attributes in my config file? Any other suggestions?
>
> Thanks much,
>
> --
> Michael Corey
> News Applications Developer
> Center for Investigative Reporting
> 510.809.3178
>
> /2012 Recipient of the MacArthur Award for Creative and Effective
> Institutions/
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>