[OSM-dev] api0.6 - only one value per key?

Shaun McDonald shaun at shaunmcdonald.me.uk
Sun Feb 1 01:30:36 GMT 2009


On 1 Feb 2009, at 00:39, Andreas Kalsch wrote:

> Shaun McDonald schrieb:
>>
>> On 30 Jan 2009, at 15:11, Andreas Kalsch wrote:
>>
>>
>>>
>>> There are some imported values which are longer than 255 characters.
>>>
>>
>> Strictly speaking at the moment it is 255 latin characters, which  
>> can is different to 255 UTF-8 multi-byte characters, hence the  
>> problem at the moment of large UTF-8 multibyte strings being  
>> truncated and causing invalid UTF-8 errors.
> MySQL 5 will let you use up to 255 wide chars, if your database is  
> Unicode.


1. Rails talks UTF8 to the web
2. Rails thinks the database is Latin1
3. Rails converts the data from UTF8 to Latin1 to talk to the database.
4. The database then converts the latin1 to UTF8
5. The database tables are UTF8 so there is a silent truncation when  
the data is stored to varchar 255 fields based on the conversion of  
the data in steps 3 and 4. With multibyte chars this is a problem as  
it can be mid way through the character that the truncation happens so  
you get the UTF8 errors.

Shaun




More information about the dev mailing list