[osmosis-dev] Relations with > 65k members
brett at bretth.com
Fri Nov 13 00:35:40 GMT 2009
On Thu, Nov 12, 2009 at 12:40 PM, Brett Henderson <brett at bretth.com> wrote:
> On Thu, Nov 12, 2009 at 9:36 AM, Frederik Ramm <frederik at remote.org>wrote:
>> we just had a problem with diff updates because someone created
>> relations with more than 65k members and Osmosis tries to store them as
>> a char which raised an exception.
>> I did a quick hack that would simply drop the excessive members with a
>> I'm not checking that in.
>> I thought about simply patching Osmosis to use an integer but was
>> informed by Jon Burgess that osm2pgsql (which is next in my toolchain)
>> would carp on > 65k members as well so I didn't bother - especially as
>> it turned out that the relations in question were an error in somebody's
>> automated import and have been meanwhile fixed by Lennard (they
>> contained the same element over and over).
> If > 65k members is legitimate then feel free to change Osmosis. I only
> picked smaller data types to save some memory when doing things like large
> sorts. If the API allows it then perhaps Osmosis should too.
> Another one that will have to be changed eventually is entity ids. They'll
> only allow up to 2^31 currently (signed integer) which will probably break
> eventually given the rapid growth of the database.
> I've updated the Relation class to use a 32-bit int instead of 16-bit char
when persisting temp data to disk (which I assume is where it broke). This
occurs in tasks like --sort which performs a disk based merge sort. The
only reason for using a smaller data type was to reduce the amount of data
being persisted which is largely unimportant for Relations because there
aren't a lot of them anyway.
It's only checked into svn for now and will show up in nightly builds. I'm
overdue for a proper release so will have to do that at some stage.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the osmosis-dev