[OSM-dev] Osmosis duplicate nodes / osm2pgsql primary keys

[Frederik Ramm]

Hi,

On 03/15/11 09:51, Nick Whitelegg wrote:
> If I don't use slim mode I get out of memory errors when importing the
> tracks file (the others are small enough so go in OK) - have to admit
> I'm surprised I get out of memory errors as it's only a 176MB file.

osm2pgqsl allocates memory in blocks of 8k nodes, each node takes 8 
bytes of memory. The whole block is allocated if you have even one node 
in that block. This means that if your nodes are distributed thinly but 
evenly across the ID range, each single node will trigger the allocation 
of 64k of memory, up to a maximum of (currently) around 9 GB.

> There doesn't seem to be a facility in osmosis for removing duplicates.

You might be able to work around this by converting your files to diffs 
(replace <osm> by <osmChange><create> and similar for the closing tag), 
and then having Osmosis read, merge, and simplify them (--rxc --rxc 
--rxc --mc --mc --simc --wxc), and manually change the result back to an 
OSM file.

Bye
Frederik