[Openstreetmap-dev] CSV transport encoding scheme
immanuel.scholz at gmx.de
Thu Jan 26 09:05:07 GMT 2006
> Immanuel Scholz wrote:
>> (*sigh* Another "own invented text based format". Maybe fixing XML would
>> be an attractive alternative to this.)
> Are you here to create maps or to create new formats? People
> invented XML to get rid of this issue. Why don't you use it?
I am very unsatisfied with the current server performance, especally for
map.rb - requests.
Currently the server takes several minutes (without transfer) to process a
request for 30000 nodes and 30000 lines out of about 100000 lines / nodes
about 5 seconds to read the nodes.
about 30 seconds to read the line segments.
Almost all of the rest is taken to encode the xml data structure.
This was measured on my private maching, Athlon 64 3000+ with 1 GB RAM,
Linux, DB with (hopefully) representative but auto-generated data.
I was able to reduce the time XML take to about 30 seconds by changing the
implementation but Steve found out something went wrong and postponed my
implementation until I have time to trace the bug and fix ist (maybe this
weekend, but no promise). However, 30 seconds only for encoding is too
This is not a RAM-swap-cache issue, the percentage are the same when only
processing 500 data entries (except the time to retrieve the line segments
which is not linear to the number of nodes).
This lead me to the conclusion (which I posted around new year) that the
XML encoding currently used is a significant bottleneck and should be
addressed as problem.
Someone (sorry, don't remember) suggested CSV as a very performant
replacement for XML some month before, so my first idea was using this
encoding scheme as replacement for XML.
Other solutions to this may be replace the ruby REXML library for
something more performant, replace ruby with a language that fits,
invalidate my profiling by showing that the problem is elsewhere or
redefine the XML data structure that it can be processed more performant.
PS: I am in the very unfortunate position that I am the first who suggest
changing the transport encoding. I would like to keep XML too (surprise!
surprise! I was the one suggesting the API0.2 scheme ;), but I did not
found an alternative XML implementation and I am too scared to ask Steve
whether he would consider switching to something other than ruby ;-)
More information about the dev