[OSM-dev] Planet dump today delayed (ie not today) :-(

[Martijn van Oosterhout]

On 8/23/07, Tom Hughes <tom at compton.nu> wrote:
> Well the only thing I can think of is that I don't know how efficient
> the page based fetching it is doing is, especially when there is a
> condition involved.

My guess, probably very inefficient. My experience is that it's very
difficult problem to optimise, especially once you consider concurrent
updates. On other databases they recommend the use of cursors, or
streaming the output in one go.

> Unless it is very clever it will probably have to effectively start
> reading at the beginning of the table each time and just not return
> records until it reaches the starting point.

One quick way to speed it up is instead of saying start at record X is
to remember the last record id you received and make your next query:

SELECT ... FROM blah WHERE <condition> AND id > $lastid LIMIT 500000;

That way MySQL can use the index to jump straight to the right spot.

But in the end Osmosis is likely to be a better bet.

Have a nice day,
-- 
Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/