[osmosis-dev] Relations with > 65k members

[Brett Henderson]

On Thu, Nov 12, 2009 at 12:40 PM, Brett Henderson <brett at bretth.com> wrote:

>
>
> On Thu, Nov 12, 2009 at 9:36 AM, Frederik Ramm <frederik at remote.org>wrote:
>
>> Hi,
>>
>>    we just had a problem with diff updates because someone created
>> relations with more than 65k members and Osmosis tries to store them as
>> a char which raised an exception.
>>
>> I did a quick hack that would simply drop the excessive members with a
>> warning:
>>
>> http://www.remote.org/frederik/tmp/remove_excessive_members.diff
>>
>> I'm not checking that in.
>>
>> I thought about simply patching Osmosis to use an integer but was
>> informed by Jon Burgess that osm2pgsql (which is next in my toolchain)
>> would carp on > 65k members as well so I didn't bother - especially as
>> it turned out that the relations in question were an error in somebody's
>> automated import and have been meanwhile fixed by Lennard (they
>> contained the same element over and over).
>>
>
> If > 65k members is legitimate then feel free to change Osmosis.  I only
> picked smaller data types to save some memory when doing things like large
> sorts.  If the API allows it then perhaps Osmosis should too.
>
> Another one that will have to be changed eventually is entity ids.  They'll
> only allow up to 2^31 currently (signed integer) which will probably break
> eventually given the rapid growth of the database.
>
> I've updated the Relation class to use a 32-bit int instead of 16-bit char
when persisting temp data to disk (which I assume is where it broke).  This
occurs in tasks like --sort which performs a disk based merge sort.  The
only reason for using a smaller data type was to reduce the amount of data
being persisted which is largely unimportant for Relations because there
aren't a lot of them anyway.

It's only checked into svn for now and will show up in nightly builds.  I'm
overdue for a proper release so will have to do that at some stage.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20091113/9466a4dc/attachment.html>