[Rebuild] Strategy for running the bot in "regions"
frederik at remote.org
Mon Jun 25 11:28:35 BST 2012
On 06/25/12 12:13, Andy Allan wrote:
> There's been previous agreement on the list that we want to run the
> bot "per region", rather than e.g. by id range (first thousand nodes,
> second thousand etc). Has anyone had any thoughts as to how to select
> all the data in a region? And how to process region-by-region while
> making sure we (eventually) process everything? Map calls are one
> approach, but throw up complications around the area of nested
> relations and general resource usage (it takes a lot of calls to cover
> Ireland, for example). Alternatively, the bot could read from the
> database directly, and build its own idea of what data is in a region.
I'd just crawl over the world in a grid, e.g. 1° by 1° or smaller (small
enough to have reason to assume that, on average, one changeset will be
enough to make all modifications required in the area). I'd be loading
the data from the database with code that mimicks the map call.
I think it would be ok, if performance considerations make that
attractive, to choose a method that would possibly omit some cases (e.g.
nested relations), and then collect them in a couple broad strokes after
the main job is done, even if that would lead to a few world-spanning
changesets in the end.
Frederik Ramm ## eMail frederik at remote.org ## N49°00'09" E008°23'33"
More information about the Rebuild