[OSM-talk-nl] [OSM-dev] conversion AND data

[Marc Kessels]

On Mon, 2007-07-30 at 18:53 +0200, Stefan de Konink wrote:
> On Mon, 30 Jul 2007, Marc Kessels wrote:
> 
> > the reason behind this huge memory consumption is that it needs to know
> > all nodes and segments, in order to prevent nodes with the same
> > coordinates, or multiple segments between the same two nodes.
> 
> On the main list I already mentioned why this not the way to go. You don't
> need to know *all* nodes in memory. You need to set a bounding box (based
> on next and previous segments), this reduces the the requirement memory
> and is able to 'stream' the through the AND file, with (indexed-)lookups
> in the output OSM file.
> 
> 
> Stefan

first: I know dev, talk and talk-nl, but could not find a post where you
mention this. could be I don't have all the messages, or is there
another list?

Anyhow: my program actually does this: you can specify a bounding box
(in the source, could be a command line parameter, but "make 2AND" is
just as simple for me....) and it will look at ways which have at least
one point in this bounding box. The program will use much less memory in
that case, of course. 
In case you make the bounding box as large as the Netherlands (or make
the bouding box always return "-1") you do need to remember all nodes,
and it will consume a lot of memory. 
I suppose you are suggesting to divide the complete country into several
bounding boxes, and export per box. Am I right? 
This would require that you go through the AND data a multiple number of
times, which would take quite long. Also you will have an issue with
ways that extend through a multiple number of boxes. Of course you could
write some logic to catch this (e.g. by using only the most top left
coordinate of a way) and output each way only one time, but you still
have an issue that you are most likely creating duplicates of nodes
outside of the current box. (the way will be attached to another way in
another box, for wich also a new nodeID will be created). This could be
solved by looking through the already created OSM file, but than you
again have a file-access, and I think it will then be as slow as
swapping data from and to memory...

Obviously, I did not need to go through this hassle, since my machine
did have enough memory.... :)