[OSM-dev] [Talk-GB] Motorway and Railway progress (simplifying OSM data)

[Joerg Ostertag (OSM Munich/Germany)]

On Monday 06 November 2006 11:13, Nick Burch wrote:
> On Sun, 5 Nov 2006, Jon Burgess wrote:
> > This new version is also much more memory efficient by filtering the
> > node/segments & ways as they are being read. Processing the whole of
> > planet.osm in one go is still beyond my machine. It looks like it needs
> > around 3GB of memory for this. I'll add an --area option which should be
> > trivial using Geo::Filter::Area.
>
> You should take a look at on of the scripts in
> http://svn.openstreetmap.org/utils/planet.osm/perl/, eg
> http://svn.openstreetmap.org/utils/planet.osm/perl/planetosm-excerpt-area.p
>l
>
> You'll want to use Bit::Vector instead of a regular perl hash wherever you
> can. You may also find you need to do two passes, one to figure out what
> nodes/segments/ways you want (populating a nice and efficient
> Bit::Vector), then a second to actually do your filtering (so you only
> store lots of data on things of interest)
>
> I managed to get the memory use on my scripts to drop from 3gb to under a
> hundred meg, just by switching to Bit::Vector where I could.

In my tests using a bitvector didn't save too much memory compared to using an 
array. But I'll post more as I do more tests.

-
Joerg