[OSM-dev] Bin-test failed (different test)

Brett Henderson brett at bretth.com
Mon Oct 4 08:01:36 BST 2010


On Mon, Oct 4, 2010 at 1:56 AM, Scott Crosby <scrosby at cs.rice.edu> wrote:

> On Thu, Sep 30, 2010 at 7:41 PM, Brett Henderson <brett at bretth.com> wrote:
>
>> On Fri, Oct 1, 2010 at 12:23 AM, Curt Nowak <nowak at bwl.uni-hildesheim.de>wrote:
>>
>>> Hello dev,
>>>
>>> I also tested osmosis 0.37-SNAPSHOT with the bin-support for various
>>> countries downloaded at Geofabrik.
>>> In contrast to the -xml tasks I *sometimes* ran into an exception using
>>> the -bin tasks in combination with the --used-node task.
>>>
>>>
>>>
>>> What's up with that? Why is there suddenly a negative user id involved
>>> that is not present in the .osm file of Slovenia? Btw: This does *not*
>>> happen for all tested input files. E.g. Andorra worked fine.
>>>
>>
>> For users that aren't public, a user called NONE is used within the
>> Osmosis pipeline.  It has an id of -1.  But that is typically not written to
>> data files.  The bin tasks may need to update their handling of private
>> users ...
>>
>>
> I think I know the cause:
>
> When metadata is stripped during writing. No UID is stored. If that file is
> later read the binary reader has to plug in some UID for downstream osmosis
> and OsmUser.NONE is used. If the output of this processing is written to a
> binary file, without metadata stripping, the code will preserve the UID=-1
> to the file. Now, when that file is read again, the reader passes the UID=-1
> to downstream osmosis, which doesn't like it.
>
> The root cause is osmosis assumes and requires that all entities have
> metadata, and I offer a feature that allows it to be stripped for a15% gain
> in filesize and processing speed. Brett, how do you suggest I handle the
> case of stripped metadata to downstream osmosis?
>
>
> FIXES:
>    1. I suspect that somewhere in Frederik's pipeline is an
> omitmetadata=true, but downstream processing reads a file written with that
> flag set, and writes a file without that flag.
>    2. Error out when writing a binary file that contains UID<0, to catch
> mistakes of type 1.Those situations are sorta silly as space is wasted
> storing useless metadata.
>    3. Transparently map UID<0 in the file to OsmUser.NONE when reading.
>    4. Make osmosis not care if UID's are negative.
>
> My preference is fix #4, but warn if negative UID's are being written, to
> detect use errors with of the discussed in fixes #2 and #1.
>

Are you aware that a user is not always available?  Some entities in the
planet will not have a user id available, it is optional.  This is because
some data has been edited by users without their public flag set.  So we
have to deal with missing user data in all cases, not just in the case where
you've dropped metadata.  If you are writing -1 to a file when OsmUser.NONE
is received, then all planet derived data files will include this type of
data.

OSM in general doesn't allow negative UIDs so in general they should never
exist.  I set the Osmosis.NONE user to -1 which indicates the special case
where no user is available.  I added the negative check to Osmosis as a
check to verify this.  In your case it sounds like you should be checking
for the -1 in your data files (if that's how you wish to represent
OsmUser.NONE) and using OsmUser.NONE instead of trying to instantiate a new
OsmUser (ie. Option 3).

Cheers,
Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20101004/f1e804dd/attachment.html>


More information about the dev mailing list