[OSM-dev] way 27483626 UTF-8 truncation

[Florian Lohoff]

On Sat, Oct 04, 2008 at 06:34:12PM +1000, Brett Henderson wrote:
> Subject: Re: [OSM-dev] way 27483626 UTF-8 truncation
> 
> Florian Lohoff wrote:
> >On Sat, Oct 04, 2008 at 03:24:12PM +1000, Brett Henderson wrote:
> >  
> >>>Another 2 change files contain utf-8 bugs and osmosis refuses to process
> >>>them:
> >>>
> >>>200810031022-200810031023.osc
> >>>200810031023-200810031024.osc
> >>> 
> >>>      
> >>I've tested both of these files and they seem okay.  The only problem I 
> >>can find is way 27483626 which has a broken "note" tag in file 
> >>2008100310-2008100311.osc.  Are you sure these files are broken?
> >>    
> >
> >wget -O - 
> >http://planet.openstreetmap.org/minute/200810031022-200810031023.osc.gz | 
> >gzip -d | iconv -f utf8 -t utf8
> >[...]
> >    <way id="14783001" timestamp="2008-10-03T10:22:11Z" user="logictheo">
> >      <nd ref="145957773"/>
> >      <nd ref="163161140"/>
> >      <nd ref="146004252"/>
> >      <nd ref="301736490"/>
> >      <tag k="name" v="Οδός Ιουστινιανού"/>
> >      <tag k="created_by" v="Potlatch 0.6a"/>
> >      <tag k="highway" v="residential"/>
> >      <tag k="name:en" v="Ioustinianou Street"/>
> >      <tag k="note" v="Ρώτησα ένα φίλο που μένει 
> >      καιρό εδώ εάν αυτός ήταν κάποτε 
> >      δρόμος. Το κοίταξα και από κοντά. 
> >      Βλέπω μπάρες και στις 2 άκρες που 
> >      είναι για να εμπ
> >iconv: illegal input sequence at position 16342
> >
> >
> >wget -O - 
> >http://planet.openstreetmap.org/minute/200810031023-200810031024.osc.gz | 
> >gzip -d | iconv -f utf8 -t utf8
> >[...]
> >    <way id="27483626" timestamp="2008-10-03T10:23:02Z" user="logictheo">
> >      <nd ref="301736490"/>
> >      <nd ref="145958259"/>
> >      <nd ref="301736491"/>
> >      <tag k="name" v="Οδός Ιουστινιανού"/>
> >      <tag k="created_by" v="Potlatch 0.6a"/>
> >      <tag k="highway" v="pedestrian"/>
> >      <tag k="name:en" v="Ioustinianou Street"/>
> >      <tag k="note" v="Ρώτησα ένα φίλο που μένει 
> >      καιρό εδώ εάν αυτός ήταν κάποτε 
> >      δρόμος. Το κοίταξα και από κοντά. 
> >      Βλέπω μπάρες και στις 2 άκρες που 
> >      είναι για να εμπ
> >iconv: illegal input sequence at position 58891
> >
> >Flo
> >  
> Ah, sorry. I misread your first email. I didn't realise you were 
> referring to minute changesets. I didn't realise there were two errors 
> in that hourly file. I have to leave now, I'll try to take another look 
> tomorrow morning (approx 15 hours from now).

To get the ROMA database in sync again i replaced the notes by
"broken-utf8" - As notes typically get not rendered thats not a problem
for me though. ROMA was down for a half a day before i discovered the
broken files and fixed them ...

Flo
-- 
Florian Lohoff                  flo at rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20081004/e5c45b65/attachment.pgp>