[Rebuild] Notes on rebuild conference call, 19th Feb 2012

Dermot McNally dermotm at gmail.com
Sun Feb 19 20:58:36 GMT 2012


Hi all,

Thanks to all who participated in the conference call. Here are the
notes of what was discussed:

* Rebuild strategy.

Consensus exists that an in-place schema update of the existing
database is the best approach. There is also a generally positive
attitude to the idea of "continuous changeover", though with two
separate viewpoints on the meaning of this phrase. Most participants
anticipate an approach the necessary redaction of map data is applied
progressively to defined regions, once per region affecting the
production data set immediately, each at a decided time prior to the
published cutoff date. The alternative approach (Dermot's) is to
process selected objects (or regions) potentially repeatedly prior to
the cutoff date, but to record the redacted version to a separate data
store where it would be used only to validate migration rules and
provide reports or other tools for remapping.

* Clarifying final migration rules through test-driven approach

Matt has been experimenting with a test-driven approach for ensuring
that the licence migration software respect a visible and accountable
set of rules. His work so far is here:

https://github.com/zerebubuth/openstreetmap-license-change

We agreed to collaborate on extending and refining these rules.
Frederik and Dermot have volunteered to assist. *** Important: we are
now calling for participation in compiling a good set of test cases
(in human-readable form, need not be code)***

* WTFE rules

Frederik cautioned that, although WTFE represents the commonly
accepted (including by LWG) reference for migration logic, certain
edge cases still exist. It was agreed that the tests mentioned above
will allow us to confront any remaining gaps. We also considered the
"V0 proposals" for handling the migration rules for relations. We
reached consensus that the V0 proposals are acceptable not just for
relations, but also for ways and nodes. The matter will be discussed
at LWG with a view to an official LWG statement on the matter.

* Consumers of diffs and how they will be affected by the change

It was noted that consumers of diffs will need to know about the
changes and should be contacted [note: I do not recall any individual
undertaking to do so]. We need to decide whether to engineer the
change in order to wind down the existing diffs and require explicit
migration to a new set. This would ensure that consumers acknowledge
important issues related to the change such as the very large initial
diff set and the need to change attribution. It was noted that an
early heads-up is needed due to the engineering difficulty involved.

* Test server for migration code

It was decided to test all migration code on the normal dev server on
a subset of data. Once the production database has been migrated to
the new server, the existing production server _may_ be available to
us, but it is known to be defective, so we must not be reliant on
this. In any event, it is not known how soon it could be available. We
also ruled out renting a server, opting instead to look for early
stabilisation of the code  on dev so it can be put into production.

* Visualisation and remapping tools

Frederik mentioned the large effort involved in deploying any
additional slippy map to better visualise the effect of the licence
change. We agreed that the existing tools, while not without
limitations, are sufficient for our needs.

* The plan to produce one last CC Planet file

A long-standing intention has been to produce an official
last-CC-planet file immediately prior to the changeover. The
usefulness of this is challenged by both the reality of remapping,
which has been happening for some time and the possibility of early
data migration in some areas. It may be opportune to not release an
official planet, but leave it to users to decide, based on their own
needs, what point in time offers them the best CC planet extract.
Alternatively, we could create the last CC planet immediately prior to
the start of any continuous migration plan.

This issue will be discussed by LWG with a view to issuing a statement.


Thank you all, I hope I have given an accurate account.
Dermot

-- 
--------------------------------------
Igaühel on siin oma laul
ja ma oma ei leiagi üles



More information about the Rebuild mailing list