[OSM-dev] Too many slow queries in db
jburgess777 at googlemail.com
Tue Sep 4 01:25:53 BST 2007
On Tue, 2007-09-04 at 00:46 +0100, Tom Hughes wrote:
> In message <1188862129.4515.91.camel at localhost.localdomain>
> Jon Burgess <jburgess777 at googlemail.com> wrote:
> > On Tue, 2007-09-04 at 00:08 +0100, Tom Hughes wrote:
> > > In message <200709040850140140.0A99AA0F at smtp.nsw.exemail.com.au>
> > > "Brent Easton" <b.easton at exemail.com.au> wrote:
> > >
> > > > >Yeah sure. My magic query accelerator will sort it all out.
> > > > >
> > > > >Seriously, if you've got a magic way to make a database query
> > > > >a 100 million row table in the blink of an eyelid I'll be only
> > > > >to happy to deploy it.
> > > >
> > > > So, what where the plans again to handle queries on a 2 billion row table
> > > > post TIGER?
> > >
> > > Um, we don't have one?
> > >
> > > Seriously, I think it is (a) get a proper database and (b) find somebody
> > > with sufficient experience of designing hardware and software solutions
> > > for massive database to make it work.
> > I'm still making progress with investigating a high speed tiger import
> > process. It it too early to give any hard numbers but the signs are
> > promising that we'll be able to achieve something significantly faster.
> Getting it into the database isn't really the issue - in fact I would
> probably prefer it to come in slowly so we have a chance to scale things
> properly as it goes in.
> The issue is having a usable database once the data is in, not getting
> it in there.
The way I look at it, getting the data into a DB in a couple of days
enables me to get a handle on how much disk space is required and what
the read / update performance is like. This data doesn't have to go
anywhere near the production DB to still provide useful feedback.
I'm afraid that if we just let it grow too slowly we:
- have little way to plan for the problems we hit until it is too late
- impact all the other API users for the duration of the import
- put off people that are interested in updating the Tiger data
As you mentioned, gaining the experience of handling this volume of data
is critical to the import being successful.
More information about the dev