[osmosis-dev] Relations with > 65k members

[Brett Henderson]

On Fri, Nov 13, 2009 at 11:44 AM, Frederik Ramm <frederik at remote.org> wrote:

> Hi,
>
>
> Brett Henderson wrote:
>
>> I've updated the Relation class to use a 32-bit int instead of 16-bit char
>> when persisting temp data to disk (which I assume is where it broke).
>>
>
> The problem occured in the Relation's "store" routine where it attempted to
> make a Char value of the relation member count. I am, however, surprised
> that you say this has anything to do with persisting temp data to disk, as
> the problem occured when doing a --rri --wxc combination which only handled
> a rather small amount of data.


Yep, that explains it.  --rri internally downloads all the files, merges
them together, then sorts the results before sending to output.  This
guarantees that all data is sent downstream in node, way relation order with
ascending ids and versions.  Sorting uses a class called FileBasedSort which
is where all the temp file based merge sort implementation occurs.

FileBasedSort could probably be optimised so that small sorts don't persist
to disk but it's well tested and relatively efficient so I've never touched
it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20091113/3fc4e59a/attachment.html>