[OSM-dev] Too many slow queries in db
Stefan de Konink
skinkie at xs4all.nl
Tue Sep 4 09:01:55 BST 2007
On Tue, 4 Sep 2007, Tom Hughes wrote:
> In message <20070904025956.B39700-100000 at xs2.xs4all.nl>
> Stefan de Konink <skinkie at xs4all.nl> wrote:
> > On Tue, 4 Sep 2007, Jon Burgess wrote:
> >> As you mentioned, gaining the experience of handling this volume of data
> >> is critical to the import being successful.
> > In that case: stick with MySQL for the time, import your own data into
> > this new server. And implement this 'proxy'. Now, I can understand that
> > you would like to host this 'new' server close to the proxy, in respect
> > to bandwidth.
> Do you know how the proxy handle allocation of new values to
> auto increment columns? That seems like a major problem with
> splitting up the data to me?
Now you are describing INSERT queries. I hope you are not saying that this
type of query is the reason of the current slowness. I'm not aware of the
internals of *any* MySQL product, but implementing a loadbalancing system
for at least fail over is pretty common these days.
If this system could smear requests over two servers, maybe even with some
code to do relative simple requests (classification?) on a smaller
machine, many people could find their speed back I guess.
> To be honest, somebody pointed out this proxy thing before and I did
> have a look at it then and found the web page very hand-wavy and
> lacking in any serious detail or documentation. It looks like it
> is a bit new and maybe not very mature? or is that just me?
I have some trust in code claimed to be 'Enterprise', but you should
benchmark it, anything that you install anyway :)
> If the only reason you think it is useful is as a way to split the
> data over multiple physical machines, then I have to say that I'm
> not sure we've reached that point yet - we may do with all the Tiger
> data but it sounds like Jon is looking at that and in the meantime
> there are other things I think we can do.
Keeping everything on one machine 'because you can' is not the way to look
to the future. I'm a very pro for geo splitting data (distribution wise)
but still even still solution doesn't make the system *scale*. It is only
divide and conquor and not plug and play.
More information about the dev