[OSM-dev] Automated regression testing of osm2pgsql (was Re: [Tile-serving] Parallelizing more of osm2pgsql)

[Kai Krueger]

On 07/15/2013 02:03 AM, Andy Allan wrote:
> I realise I'm in danger of sounding like a broken record here, but it
> would be extremely valuable to work on a unit test framework for
> osm2pgsql. The lack of confidence in whether refactored code gives the
> same output as battle-hardened existing code discourages me (and
> presumably other people) from testing the new code. The lack of
> certainty presumably also discourages other developers from attempting
> refactoring, given that there's no sensible manner to find out if the
> output is correct in all the (many) gnarly edge cases.
>
> Trying to verify code correctness by importing different planet files
> and applying different sequences of diff updates and then hoping that
> whoever is brave enough to do this on a production server with enough
> users so that one of them happens to spot a broken multipolygon update
> - well, lets say I can think of a better approach.
>
> Unit tests.
Although they aren't unit tests, osm2pgsql now has an improved
integration test framework to help check that the produced output of
osm2pgsql is correct. Indeed while writing the initial set of tests, it
has already revealed a couple of issues with the existing "battle
hardened code". So hopefully they will in future continue to prove useful.

The way the tests work are that the testing framework imports a well
defined osm or osc file with varying parameters and then runs a set of
sql queries on the resultant database to check against known good results.


So far the tests include a reasonably comprehensive coverage of
multi-polygon behaviour and a more "summery level" coverage of
everything else. It would be great if other people could help contribute
more test cases to get to a thorough coverage of all aspects of osm2pgsql.

It is pretty easy to contribute new test cases. A test case consists of
an SQL statement which produces a single numeric output, the known
correct numeric output and a set of import parameters to which this test
case applies to.

SQL queries can be arbitrarily complex, but queries like the following
seem probably most appropriate: " SELECT ST_NumInteriorRing(way) FROM
planet_osm_polygon WHERE osm_id = -3 and landuse = \'residential\'and
name = \'Name_rel11\''", which check that the specially crafted
multi-polygon relation 3 is correctly imported with two holes for
geometry and the correct tagging is taken from the member ways

To run the tests. Just type "python tests/regression-test.py" in the
osm2pgsql directory. You will either need to have setup a separate
postgresql tablespace called " tablespacetest" or it will ask for the
root password to set it up for you.

Hopefully this, together with the release process of osm2pgsql ensures
that osm2pgsql remains reliable and "trust worthy" and does not scare of
new developers to contributing to the code.

Kai
>
> Cheers,
> Andy