[OSM-dev] [Talk-GB] Motorway and Railway progress (simplifying OSM data)
Joerg Ostertag (OSM Munich/Germany)
openstreetmap at ostertag.name
Mon Nov 6 11:04:55 GMT 2006
On Monday 06 November 2006 11:13, Nick Burch wrote:
> On Sun, 5 Nov 2006, Jon Burgess wrote:
> > This new version is also much more memory efficient by filtering the
> > node/segments & ways as they are being read. Processing the whole of
> > planet.osm in one go is still beyond my machine. It looks like it needs
> > around 3GB of memory for this. I'll add an --area option which should be
> > trivial using Geo::Filter::Area.
> You should take a look at on of the scripts in
> http://svn.openstreetmap.org/utils/planet.osm/perl/, eg
> You'll want to use Bit::Vector instead of a regular perl hash wherever you
> can. You may also find you need to do two passes, one to figure out what
> nodes/segments/ways you want (populating a nice and efficient
> Bit::Vector), then a second to actually do your filtering (so you only
> store lots of data on things of interest)
> I managed to get the memory use on my scripts to drop from 3gb to under a
> hundred meg, just by switching to Bit::Vector where I could.
In my tests using a bitvector didn't save too much memory compared to using an
array. But I'll post more as I do more tests.
More information about the dev