[OSM-dev] Osmosis Output Error

Karl Newman siliconfiend at gmail.com
Wed Dec 10 00:41:08 GMT 2008


On Tue, Dec 9, 2008 at 4:16 PM, Brett Henderson <brett at bretth.com> wrote:

> Andrew Ayre wrote:
>
>> Karl Newman wrote:
>>
>>> On Tue, Dec 9, 2008 at 2:40 PM, Andrew Ayre <andy at britishideas.com<mailto:
>>> andy at britishideas.com>> wrote:
>>>
>>>    Brett Henderson wrote:
>>>
>>>        Karl Newman wrote:
>>>
>>>            So, sounds like the most recent changes are not the issue.
>>>            Do you know if SRTM2OSM creates entities with negative ids?
>>>            If not, you may run into collisions with the other data file
>>>            that you're trying to merge.
>>>
>>>            Karl
>>>
>>>        Andrew, it would be great if you could put your files online
>>>        somewhere.  I'll take a look this evening if Karl doesn't beat
>>>        me to it.
>>>
>>>        My biggest concern at this point is that you don't appear to be
>>>        getting a decent error message.  Perhaps I'm losing an exception
>>>        stack trace somewhere.  It makes it hard to diagnose the problem.
>>>
>>>        Brett
>>>
>>>
>>>    Karl - I'm not sure if SRTM2OSM produces entities with negative IDs.
>>>    I'll try to check into that. But I think other people are doing the
>>>    same thing I'm trying to do, making me wonder if I am doing
>>>    something wrong here.
>>>
>>>    Brett - I've uploaded the files to:
>>>
>>>    http://files.britishideas.com/index.php?dir=osmosis/
>>>
>>>    Note that I am running this on a set of 26 pairs of tiles. I get
>>>    this errors on six of the pairs. The others seem to be OK, although
>>>    I haven't rendered them yet to check.
>>>
>>>    Thanks for the help! Andy
>>>
>>>
>>> Looking at your source data (generated by srtm2osm), there are a couple
>>> potential issues. The first is that the ids are not negative, although they
>>> start at 1E9, so you probably won't run into a collision (I don't think the
>>> new entity ids are anywhere near that yet). The bigger problem is that it
>>> intermingles nodes and ways, and generally Osmosis works better if all the
>>> nodes come first, then all the ways, then all the relations. For a merge,
>>> they have to be sorted. That's easy enough to do, though--just add a --sort
>>> task to your command line (for both input files). i.e., osmosis --rx
>>> osmfile.osm --sort --rx srtmfile.osm --sort --merge --wx outfile.osm
>>>
>>> In reality, maybe the problem is entirely with your osm data and not at
>>> all with the SRTM. It's possible some of your osm data files have items out
>>> of order, which is causing a problem with the merge.
>>>
>>
>> Thanks for looking into this. I added --sort after each input file name
>> and it still gives the same error. The OSM map file was generated by
>> osmosis, which I used to split planet.osm.
>>
>> java -Xmx1560m -jar /home/nav/scripts/osmosis.jar --read-xml
>> file="/home/nav/temp/tile.63255095.osm" --sort --read-xml
>> file="/home/nav/temp/eletile.63255095.osm" --sort --merge --write-xml
>> file="/home/nav/temp/mergedtile.63255095.osm"
>>
>>  I had a quick play and think I've found the problem.
>
> Firstly, when you receive an error in osmosis, look further down the stack
> trace.  The real reason is almost always in there somewhere.  Osmosis is
> multi-threaded so it is difficult to have the real reason show up at the
> start of the error message.  In your case the first bit of error was from
> the input xml readers detecting that the merge task had aborted.  The merge
> task output was further down the error messages.
>
> In this case there were two problems:
> 1. The data was unsorted.
> 2. Some data doesn't appear to have dates attached (I haven't found the
> offending entry yet).
>
> This command line works:
> osmosis --rx tile.63255095.osm.gz enableDateParsing=false --sort --rx
> eletile.63255095.osm.gz enableDateParsing=false --sort --merge --wx
> out.osm.gz
>
> It sorts all data before the merge, and ignores all dates in the input data
> (replacing them with current system time).  This is a bit of a kludge,
> ideally the data itself should be fixed to include the correct dates but
> hopefully it points you in the right direction.
>
> Hope that helps.
> Brett
>
>
The SRTM data has no timestamps, which is causing the problem. So you only
need to set enableDateParsing=false for that file. It might be good to show
a warning if an entity has no timestamp attribute in the XML. I propose
adding the following lines in BaseElementProcessor.java:

+    private static final Logger log =
Logger.getLogger(BaseElementProcessor.class.getName());

...

    protected TimestampContainer createTimestampContainer(String data) {
        if (enableDateParsing) {
+            if (data == null || data == "") {
+                log.warning("Element missing timestamp attribute.");
+            }
            return new UnparsedTimestampContainer(timestampFormat, data);
        } else {
            return dummyTimestampContainer;
        }
    }

That encapsulates it in the lowest possible level, but the downside is that
it doesn't report anything about the problematic element--Ideally it would
show the id and whether it's a node, way or relation. That could conceivably
be solved by passing that information to the createTimestampContainer
method, but that seems sloppy.

Karl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20081209/5984ca39/attachment.html>


More information about the dev mailing list