[OSM-dev] [Talk-GB] Motorway and Railway progress (simplifying OSM data)

[Nick Burch]

On Sun, 5 Nov 2006, Jon Burgess wrote:
> This new version is also much more memory efficient by filtering the 
> node/segments & ways as they are being read. Processing the whole of 
> planet.osm in one go is still beyond my machine. It looks like it needs 
> around 3GB of memory for this. I'll add an --area option which should be 
> trivial using Geo::Filter::Area.

You should take a look at on of the scripts in 
http://svn.openstreetmap.org/utils/planet.osm/perl/, eg
http://svn.openstreetmap.org/utils/planet.osm/perl/planetosm-excerpt-area.pl

You'll want to use Bit::Vector instead of a regular perl hash wherever you 
can. You may also find you need to do two passes, one to figure out what 
nodes/segments/ways you want (populating a nice and efficient 
Bit::Vector), then a second to actually do your filtering (so you only 
store lots of data on things of interest)

I managed to get the memory use on my scripts to drop from 3gb to under a 
hundred meg, just by switching to Bit::Vector where I could.

Nick