[OSM-dev] [Talk-GB] Motorway and Railway progress (simplifying OSM data)
jburgess at uklinux.net
Mon Nov 6 23:19:30 GMT 2006
On Mon, 2006-11-06 at 10:13 +0000, Nick Burch wrote:
> You should take a look at on of the scripts in
> http://svn.openstreetmap.org/utils/planet.osm/perl/, eg
> You'll want to use Bit::Vector instead of a regular perl hash wherever you
> can. You may also find you need to do two passes, one to figure out what
> nodes/segments/ways you want (populating a nice and efficient
> Bit::Vector), then a second to actually do your filtering (so you only
> store lots of data on things of interest)
> I managed to get the memory use on my scripts to drop from 3gb to under a
> hundred meg, just by switching to Bit::Vector where I could.
I can probably make use of bit vectors in one or two places to save some
memory but I don't think I can save much with my current algorithm.
The current code relies on more than just knowing whether a node is
present or not. It has to represent a set of closely spaced nodes from
the input as being replaced by a single node in the output. This is
tricky to do without storing at least one ID per node of the input. The
only alternative I can think of is to iterate the OSM data a huge number
of times (perhaps once per node) which isn't practical either.
More information about the dev