[Potlatch-dev] Testing new versions

[Andy Allan]

On 7 March 2012 20:12, SomeoneElse <lists at mail.atownsend.org.uk> wrote:
> This may be a silly question - asking for something that exists already, but
> just in case not:
>
> How hard would it be have a version of Potlatch2 available after merging for
> "final testing" before becoming the default P2 version on the site?  To be
> useful it'd need to be able to be invoked with a GPX file (not necessarily
> as another "edit" dropdown, though that'd be nice).  With the best will in
> the world, sometimes when fixing stuff other stuff breaks, and it'd be nice
> to be able to catch any issues before they go live for everyone.

There's two approaches here.

First, for the "when fixing stuff other stuff breaks" - the correct
solution there is automated tests. We use them extensively on other
projects, such as the rails port. For most of the projects I work on I
have a simple rule of thumb - writing exhaustive tests is exhausting,
but any time there's a real-world breakage, write a test for that.
Breakages are inevitable, but breaking the same thing twice is
embarrassing!

But there's plenty of worthwhile acceptance testing beyond just
automated tests. The issue at the moment is that we run a very tight
loop between Richard committing to potlatch2 master and it being
deployed on osm.org. For "final testing" to work we would need to
deliberately add a delay, and we need to work out whether the delay is
fundamentally worthwhile.

If it is, then one easy way to control the delay is using numbered
releases. We could commit to master (which means it appears on e.g.
http://random.dev.openstreetmap.org/potlatch2/ ), and then give it a
day or so before tagging that with a release number. OSM.org then
would only deploy versions with an explicit release tagged.

But at the moment, I'm not sure whether it's worth adding in the
artificial delay between fixing a problem and deploying the new
version. I suspect it's more worthwhile to work on the automated
tests, so that when we fix something, we're confident that we haven't
broken anything else.

Cheers,
Andy