[Talk-GB] OSM UK's first tile layer

[Adrian]

The test will be, if Rob is able to produce example areas processed with the OS look-up table transformation, do the misalignments go away?

Sorry for a rather long and technical message.

I've done some more investigation and testing. QGIS reckons EPSG:27700 is the OS look-up table transformation, while JOSM thinks it is the Helmert transformation. The ultimate authority is the EPSG registry https://epsg.org/crs_27700/OSGB-1936-British-National-Grid.html . Unfortunately that page is not entirely clear. But there is a reference to the OSTN15 grid file in a footnote, so I think EPSG:27700 is intended to be the look-up table transformation. So, apologies for saying previously that EPSG:27700 is the Helmert transformation.

The work to incorporate a large number of projections into JOSM was done nearly five years ago. It is based on proj4. Both proj4 and proj5 have EPSG:27700 as the Helmert transformation (based on looking at the strings, and on the behaviour of JOSM). proj6 and proj7 have EPSG:27700 as the most basic transformation, which gives misalignments of over 100m (based on looking at the strings). By 'string', I mean the line of text that begins +proj=. Some file formats for geographic information (GIS files), can accommodate a range of projections. Most of these declare the projection near the beginning of the file. The Land Registry open data are such files and they declare EPSG:27700. If you process them with proj, or an app that uses proj, you are not going to get the right results unless you can override the declaration.

The strange thing is that QGIS also uses proj. The developers of QGIS must have altered the definition of EPSG:27700 from the one built-in to proj. But I haven't discovered exactly what has been done.

I set about loading some of the Land Registry open data into JOSM, with the look-up table transformation. With the opendata plugin, JOSM can read a range of GIS file formats. Unfortunately that does not include .gml, the format of the Land Registry files. The Land Registry suggest using QGIS, so I did. JOSM and QGIS have four formats in common, KML, geoJSON, Esri shapefile .shp, and MapInfo file .mif. I am a complete newbie to QGIS, but it is a nightmare! The option to disable projection handling (No CRS) does not work properly. (This would give a means of overriding the declaration in the file.) With three of the four formats for the output file, KML, .shp and .mif, QGIS simplified the polygons, weeding out nearly half of the nodes. QGIS gave no warning of this, and I could not find an option to turn off this behaviour. (There is an option to turn off this behaviour for rendering. Perhaps it would have turned it off for output too. But why were geoJSON files unaffected?) When writing a .mif file in EPSG:27700 projection, QGIS wrote the most basic transformation as the projection, without giving a warning. QGIS did this because the .mif format has limitations on the projection definitions that it can handle. Perhaps the latest version of QGIS is a bit buggy and I should have used the LTS version.

I tried doing the transformation in QGIS, then loaded the output file into JOSM. All four file formats worked, and gave the same results (apart from the loss of nodes with three of the formats). So if you're using QGIS, I'd recommend doing it this way and using geoJSON. It doesn't even need the opendata plugin. If you want to do just the file format conversion in QGIS, and do the transformation in JOSM, it's more tricky. The KML and geoJSON formats are ruled out because they must by definition contain WGS 84 lats and longs. So you are stuck with the loss of nodes. The .shp format gives up to 5m misalignment because QGIS declares EPSG:27700 in the file and the opendata plugin provides no means for overriding the EPSG:27700 and using the custom projection I described previously. The .mif format works (in terms of getting no misalignment) if you do a simple hack. Open the .mif file in a text editor and change the fourth line from CoordSys Earth Projection 8, 79, "m", -2, 49, 0.9996012717, 400000, -100000 to CoordSys Nonearth Units "m" Bounds (0.0, 0.0) (1300000.0, 1300000.0) ensuring that none of the spaces are omitted. This means that no projection is defined in the file and the opendata plugin will then ask you which projection you want to use. You simply choose the custom projection, provided that you have previously set it up. (You may need to scroll down to see the Okay button.) The transformation in JOSM gives essentially the same results as the transformation in QGIS.

For the newbie, here's how to do the transformation in QGIS. Launch QGIS. Create a new project. Then Layer > Add Layer > Add Vector Layer and open the .gml file you have downloaded from the Land Registry open data. Accept the suggested CRS of EPSG:27700 and close the Data Source Manager dialog. Then Layer > Save As > choose Format GeoJSON, filename as desired, and CRS EPSG:4326 WGS 84. To do just the file format conversion in QGIS, Save As > choose Format Mapinfo MIF, and CRS EPSG:27700. (If that CRS is not an available option, then Layer > Set CRS of Layer and choose EPSG:27700.)

I haven't tried this, but I think it might be less problematic to use the ogr2ogr tool of GDAL to read, convert and, if desired, transform the Land Registry .gml files. The tool has an option for overriding the projection declared in the file. Note that GDAL uses proj. ogr2ogr is a command line tool, which makes it straightforward to script it. You may want to script the tool if you have a number of files to convert.

By sticking to the British National Grid, the Ordnance Survey has caused a great deal of confusion. I think, at the least, they ought to provide their products in two formats, BNG and ETRS. Just look at the list of 102 projections added by Highways England to the gamut of projection numbers. You have to think it probably would not have been necessary if they had been using ETRS. The EPSG registry hasn't helped either, with a less-than-clear definition of EPSG:27700.