[Openstreetmap-dev] CSV transport encoding scheme

Tom Carden tom at tom-carden.co.uk
Thu Jan 26 10:25:49 GMT 2006


>> Immanuel Scholz wrote:
>
> I was able to reduce the time XML take to about 30 seconds by changing the
> implementation but Steve found out something went wrong and postponed my
> implementation until I have time to trace the bug and fix ist (maybe this
> weekend, but no promise). However, 30 seconds only for encoding is too
> much!
>

Ah, it looks to me like you stopped using a library to write XML.  That's
a bad idea, since it's really easy to introduce bugs and invalid code.  A
good XML library won't let you emit invalid XML - that's a really good
thing!

http://www.openstreetmap.org/trac/browser/ruby/api/map.rb?rev=812 line 58
seems to me to be missing a '>'.  Also, if you're worried about
performance, why put in '\n' after every node?  Pretty printing XML isn't
hard, all good XML editors will format it for you if you want to edit/read
it yourself.

>
> Someone (sorry, don't remember) suggested CSV as a very performant
> replacement for XML some month before, so my first idea was using this
> encoding scheme as replacement for XML.
>

Imi, I think that suggestions to use CSV may have been sarcastic, or at
least someone playing devil's advocate.

> Other solutions to this may be replace the ruby REXML library for
> something more performant, replace ruby with a language that fits,
> invalidate my profiling by showing that the problem is elsewhere or
> redefine the XML data structure that it can be processed more performant.
>

Your profiling efforts are good ones, but I would be more interested in
seeing profiles for real regions of OSM (Birmingham, Oslo, etc), and for
sizes of regions we'll actually be requesting - take a look at the logs
and use a typical bounding box, for example.

I don't think we will see significant performance improvements by using
CSV, and in the short term it will be a huge waste of developer time to
rewrite 4 clients to a new schema.

Tom.







More information about the dev mailing list