[OSM-dev] strange Osmosis/XML/...? problem yesterday

[Brett Henderson]

Andy Allan wrote:
> On Fri, Aug 14, 2009 at 3:16 PM, Andy Allan<gravitystorm at gmail.com> wrote:
>   
>> On Fri, Aug 14, 2009 at 12:54 PM, Frederik Ramm<frederik at remote.org> wrote:
>>     
>>> Hi,
>>>
>>> Frederik Ramm wrote:
>>>       
>>>> The result file should have been something like 400 bytes. This sounds
>>>> trivial but in the original case where the .osc contained a large number
>>>> of these characters, I suddenly had 2 MB of data in one tag.
>>>>         
>>> I forgot to mention: I'm posting this here on dev and not on the osmosis
>>> list because it seems that other (at least Java) programs are also
>>> affected; someone fixed then node later with a commit comment of "JOSM
>>> says string too long" or so...
>>>       
>> The code points for these gothic characters are fine. See the
>> following (awesome) site:
>>
>> http://decodeunicode.org/en/gothic
>>
>> A rough transliteration is HEJSPANOA. However, they lie outside the
>> Basic Multilingual Plane (BMP) and can't be represented by a 16bit
>> integer. Java stores characters internally as 16-bit UCS-2 characters
>> and so everything is going horribly wrong.
>>     
>
> Installing an SMP-aware font shows what JOSM is doing more easily than
> reading Unicode code-points.
>
> http://code2000.net/code2001.htm
>
> I'll keep my (horrid) transliterations going here for the sake of everyone else.
>
> v31 - HEJSPANOA
> v32 - HHEHEJHEJSHEJSPHEJSPAHEJSPANHEJSPANOHEJSPANOA
>
> i.e. the first letter, the first two letters, the first three letters etc.
>
> I can see how you can quickly end up with a 2MB tag using this encoding scheme!
>
> Cheers,
> Andy
>   
Thanks for all this.  These unicode problems are the bane of my 
existence :-)  Any help is much appreciated.

I've run some experiments.  I've been using the unicode character 
0x10330 and experimenting with creating a test file then copying it via 
osmosis.  It seems that if I create a tag containing a single instance 
of that character I can copy it okay.  But when I create a tag with 
multiple 0x10330 characters it starts to get duplicated.

If I create a tag with a single 0x10330 character it gets copied correctly.
If I surround the character with normal latin characters it copies 
correctly.
If I put 2 0x10330 characters in the tag, 3 get written to the output.
If I put 3 0x10330 characters in the tag, 6 get written to the output.
If I make each 0x10330 character non-consecutive by surrounding them 
with latin characters they still get duplicated.

I've run this under a debugger and it seems that the data gets 
duplicated during input, not output.  My ElementWriter class may have 
some issues with surrogate pairs, but it appears that it isn't the 
source of this problem.

I've opened the file directly using a UTF-8 input stream under a 
debugger and the characters are read in correctly there as well.

I've tried using the osmosis --fast-read-xml-0.6 task and the problem 
goes away.  This alternative XML reading task uses the Woodstox StAX XML 
parser.

So to summarise it seems like the standard Java XML parser (based on 
Apache Xerces I believe) is somehow introducing surrogate pair 
duplication when multiple surrogate pairs are involved.  I don't know if 
this is a bug or a problem in how we're using it.  I'm always hesitant 
to assume bugs in the Java runtime but it seems like there might be one 
here.

Options:
1. Try to find the source of the problem in the Java XML parser or our 
use of it.
2. Switch over to the Woodstox StAX XML parser which isn't exhibiting 
the problem.

Given that Woodstox StAX parsing gives an approx 20% performance 
improvement, it might be a good time to implement option 2.

Brett

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20090816/8acb1108/attachment.html>