[Rebuild] Strategy for running the bot in "regions"

[Andy Allan]

On 25 June 2012 11:26, Dermot McNally <dermotm at gmail.com> wrote:

> For tests against subsets of the data set we can do a test of a "whole
> planet" that just happens to consist only of a bbox extract. If this
> works well, could we consider it safe to run the real redaction
> against the whole planet first time?

There's a few practicalities that this approach throws up:

* We don't have a reasonbly-fast .osh loader. The ruby one I've
written takes about 3 mins for a 1.7k-node extract (cut from an
extract of Andorra with 69k nodes), so significant sized extracts need
a faster .osh loader to be written.
* It's much easier for the community to see what's going on, and spot
any issues, on the main API
* I'd love for the regional tests to give an idea of timescale to
complete the whole world, and running them on test APIs on other
hardware denies us this. I don't know, for example, if there's
anything scaling non-linearly with the number of available nodes in
the db, so there might be different performance with processing 2k
nodes in a db with 2 billion nodes loaded.

... but it's still a valid option to do this, if we find that it's best.

Cheers,
Andy