[OSM-talk] [OSM-dev] donating read-only api-mirrors
tom at compton.nu
Fri Feb 6 16:36:43 GMT 2009
Stefan de Konink wrote:
> Tom Hughes wrote:
>> This is only partly about rails. Even if you take rails out of the
>> equation you still need to keep substantial amount of data in memory
>> in order to know which objects to fetch.
> The substantial amount of memory you are talking about is the resultset?
> Or the partially generated XML per query? In implementation detail this
> could be offloaded to the webserver when generated.
It's several things. It is holding all the XML in memory, it is holding
all the nodes in memory while we find the ways that use them, etc. Some
are avoidable by not using rails and some are not.
>> The alternative is do much more complicated database queries which are
>> likely to be slower (impossibly so on mysql, but possibly manageable
>> with a different database).
> I still do not see how your database queries can be your main pain. That
> is just a one time bench and test action. Again MySQL is not /that/ bad.
> I really don't see a reason why first so much fuzz need to be generated
> to make things better for everyone.
MySQL cannot cope with joining the node and way tables so you have to
hold all the nodes in memory and then do a fetch of the ways which use
those nodes. I have tried various ways to construct a query which
fetches all the ways in a given bbox directly and have failed to find
anything that the optimiser can cope with.
> [Sarcasm]Since the T at H folks that I think are /wasting/ resources on
> anything, /waste/ their human resources also on building a balanced
> infrastructure. I really don't mind to see them /waste/ their time on a
> quick main API.[/Sarcasm]
I think the T at H folks are completely bonkers, as do quite a few other
people that I've spoken to.
>>> Don't forget our last IRC conversation where you were unable to tell
>>> me why a specific FCGI client was borking on any request (get/put).
>>> While a new request just worked. This indicates there are more
>>> problems with Ruby-on-Rails read/write actions than you are now
>>> talking about now.
>> I'm not aware of having spoken to you on IRC to be honest. If I'd
>> known it was you I probably wouldn't have entered into the conversation.
> I hope you are kidding.
Why would be kidding. You've been trolling here for quite long enough
for me to know what you like and (until today) I've been pretty much
ignoring any thread that you're posting in for at least a month now.
>> If it is the conversation I think it is then I didn't say I didn't
>> know why it was breaking - there are a number of reasons why that
>> happens some of which I understand and some of which I don't. What I
>> said was that I had no way to tell which daemon was causing the problem.
> (17:36:04) Skinkie: TomH: could you check the api server for a hanging
> fcgi process?
> (17:43:01) TomH: Skinkie: I don't think there's any way to tell if an
> fcgi is hung
> (17:44:21) Skinkie: i am currently hammering request to the api and i
> see some request hanging, therefore set a timeout
> (17:44:28) Skinkie: this solves the issue
> (17:44:35) Skinkie: but it is clear to see something is broken
> (17:45:13) TomH: Skinkie: what sort of requests are they?
> (17:46:37) Skinkie: first i tried GET now only DELETE
> (17:48:15) TomH: I don't care what the HTTP method is, just which part
> of the API you are calling (it affects where I need to look)
> (17:48:30) TomH: presumably this is indivdual node/way calls, not map
> (17:53:10) Skinkie: only on nodes
> [that was the last thing about this conversation I left before 21:00]
Thanks for proving my point. To quote your log "I don't think there's
any way to tell if an fcgi is hung" which is exactly what I said I had
said, that although one may be hung there is no way to tell which one.
Which is why I restarted all the daemons on that occasion.
Tom Hughes (tom at compton.nu)
More information about the talk