[Rebuild] Tests of rebuild process - we need your input and help

[Dermot McNally]

Hi Frederik,

Thanks for the swift reply - I think you may have covered sufficiently
all that we really need for the tests, with any missing pieces either
in progress or not needed. Comments inline...

On 23 March 2012 07:39, Frederik Ramm <frederik at remote.org> wrote:

> I have such a list but it is not perfect. It might (a) still list some
> objects that are meanwhile clean because people have recently agreed or came
> out of anon-user status, and it might (b) miss some objects that have been
> deleted while one of my scripts was not looking. But the majority of objects
> in this list are "suspect".

This means that your list should be very close to the "deadly
accurate" numbers and, as such, be valid for benchmarking purposes.
The inclusion of objects since cleaned keeps us on the safe side -
these will incur unnecessary processing time but ultimately be passed
over with no alterations, not a big deal.

Case (b) is more interesting for the production run. Although the
likelihood of a mapper after the rebuild choosing to revive an unclean
deleted version is low, you'd have to consider that we resolved to
make it impossible (without using devious means like referring to an
old CC planet) to do this. For this reason it may prove desirable for
us to build a fresh suspect list before then. A full history planet
file should provide us with sufficient data to do so, right?


>> * A "Changesets to be exceptionally considered clean" list
>
>
> My list is basically just copied from the Wiki page:
>
> http://wtfe.gryph.de/changesets.txt

Perfect for testing. For production the only additional piece will be
the declaration of a freeze on the list, something which is in plan.


>> * A "Changesets to be exceptionally considered _non_ clean" list
>
>
> I don't have such a list.

I'm guessing that nobody has. This is fine for testing and _may_ be OK
for production too. The main case that I am aware of here is Polish
mappers having agreed (presumably in good faith) even though they may
have performed a UMP data import using their main account. We may
still be able to resolve this case without such a list depending on
the detail of how the next point gets solved, see below...


>> * Lists of _Objects_ deemed exceptionally clean (or unclean, depending
>> what is being compiled)
>>
>> Again, this may not yet be available, but it refers to the list
>> supposedly being compiled for data originally imported from UMP
>> (Poland).
>
>
> I'm in touch with Andrzej who said he would provide such a list within the
> next days.


It's great that this is in motion too. What would be useful is early
clarity on whether this will list clean or dirty objects. I'm drawn
towards a dirty list for two reasons:

a) It will (I hope) be smaller
b) It will render the dirty changeset list named above unnecessary (at
least for the suspected UMP problem cases)

The need for object level support for exceptions got onto the
development agenda fairly late in the day but Matt has already created
tests for the case. These tests could probably do with a few more eyes
on them so anybody who hasn't seen them yet, please do look and
comment.

Thanks,
Dermot