[OSM-talk] Why isn't any XAPI server available ?
osm at petschge.de
Sat Feb 19 10:12:57 GMT 2011
> Am I missing something here...? People are complaining about how bogged
> down and slow the current service is, so its being re-written in java?
> Is there any language slower or more resource intensive than java? If
> the service isnt designed to be portable (it only runs on one system
> currently, in the world), then who cares about java, why isnt it written
> in optimized C or some other similarly lowish level language, rather
> than java?
You miss a whole lot in that picture. First of all openstreetmap has a
really nice growth rate when it comes to mappers but we still lack
developers, especially ones that create "infrastructure" and not nice
webapps for the general public. So if one of the not-so-many developers
actually finds the time and starts to improve a service that is in dire
need of developer love who can dictate which tools he is to use?
The current (or some would say old) XAPI is written in mumps with a GT.M
key-value database. This language has quite a bit of history and it (as
well as the XAPI implementation in it) have been able to keep up with
openstreetmaps growth for quite a long time and have provided an
invaluable data source.
A while back the tiles at home clients switched away from XAPI to the newly
created TRAPI read only mirrors to reduce load from simple "map" calls
on the XAPI in the hope that this would free enough resources for the
more complicated queries to the XAPI. Never the less it is no longer
able to keep up with the increasing demand of "I need all organic=yes
nodes on london and I need it now" requests. Part of the problem is a
whole bunch of stupid data scraper which just use huge map calls (that
the main API disallows) to XAPI instead of using TRAPI or going straight
to the planet dump or extracts there of.
Now given that XAPI is written in a rather exotic language other
developers can not just go in and try to improve the current state of
affairs and 80n is busy doing other things and I don't blame him. I know
very well how much time can be gobbled up by developing for openstreetmap.
This is about the situation we face end of last year / start of this
year. Several developers including myself started to say "if we can't
improve the current XAPI we might have to rewrite it". At first we said
that as a joke because we very well know how large such a task is. It
needs tons of manpower to get to the point where the existing XAPI is,
you need a fairly large box to handle all of the planet and you might
have moments where you have to reload the database from scratch and
can't develop for a whole week. At this point iandees stepped up to the
task and I'm immensely glad that he did.
iandees now chose java as the language of choice, a language that I
personally don't like and that I questioned when he first proposed it.
However he gave a couple of reasons that really convinced me:
* "It has to be a language that I know." Now how could I argue with
that? I had been thinking about a rewrite quite some time but didn't
find the time and motivation. I know how badly I would have reacted if
somebody told be "you can't use C" so who am I to tell iandees "you
can't use java". You want a C version? Go write it.
* "It has to be a language that is spoken by a large number of
developers." Now mumps was the favorite tool of 80n and it worked very
well from him judging from the nice service he wrote in it. Be there
aren't all that many mumps developers around, at least not in the
openstreetmap ecosphere. I personally don't like java too well but at
least I can read (and write if need be) it and I know a whole bunch of
developers who use java. So if iandees doesn't have time for
openstreetmap in 5 years or gets eaten by a grue then jXAPI wont die but
others can take over and keep it running.
* The third point, and this is the point that really sold be, was: "If I
don't do it completely wrong the real bottleneck wont be parsing the
result but the speed at which I can retrieve the data from hard disk.
Consequently I wont write that part myself but I will use postgis, a
well-known, well-optimized piece of software that is written in C by a
whole lot of people who can throw way more manpower at it then I ever
could." I have to applaud him for that idea and I complete agree with
his analysis that java performs sufficiently well to act as the thin
glue layer between HTTP queries and the postgis backend.
So far we have no real performance numbers for the new jXAPI much less a
direct comparison to XAPI-mumps on the same hardware. If you want to
benchmark that: Go ahead! If you ask politely we can even hand you a
(anonymised!) query log to the XAPI to use as a basis for your
benchmark. If you think you can write a faster alternative then postgis
in C or another low-level language: Go ahead! We wont hold our breath
Patrick "Petschge" Kilian
More information about the talk