[osmosis-dev] Improving completeWays/completeRelations performance

[Frederik Ramm]

Scott,

On 02/18/11 14:15, Scott Crosby wrote:
> I used two bitsets for every output file. One indicating which nodes
> were already output and another (built when processing ways)
> indicating what node ID's were missed and will need to be grabbed on
> the next pass. I have another two pair of bitsets for ways and
> relations. Thats around 400mb of RAM per output.

I see. Means I overestimated a little ;)

> Really, almost everything you want is already done from the mkgmap
> splitter crosby_integration branch. Just make it support
> non-rectangular regions, and track output/missed entities with
> bitsets. I'd say about 4 hours and 2gb+400mb/region  to generate as
> many outputs as you want.

I'll have a look at that.

>> * all nodes in the bounding box
>> * all ways using any of these nodes
>> * all nodes used by any of these ways even if outside
>> * all relations using any of these nodes or ways
>> o all nodes and ways used by any of these relations even if outside
>> o but NOT all nodes used by a way drawn in through a relation.
>>
>> (The points marked "*" are what the API does; the API does not do the "o"
>> marked points even though users could be interested in them.)
>
> I did not know that it was allowed to miss the things in "o". That
> makes the job easier.

The API doesn't do them on purpose even though, having access to an 
indexed database, it could easily add the "o" points. Reason being that 
other relation members may not even be geographically nearby, and if you 
load a small part of European countryside you might not be interested in 
receiving the full detail about the Madrid-Peking long distance cycle 
route just because that happens to pass through your area of interest ;)

> There's a second approach that could split the entire planet into
> extracts, simultaneously, in 2-3 minutes; a new planet format that I'm
> working on that is geographically sorted. Progress is ongoing, but
> slow.

Sounds interesting - and difficult, in case of the Madrid-Peking long 
distance cycle route ;)

Bye
Frederik