[Imports] Road data imports from CC0 government database for Sweden

[Anders Torger]

Hello Frederik,

The skepticism is as follows. In all OSM communities there are some that 
prefer if all mapping is made manually and no imports are made at all, 
to conserve the originality of OSM data. In this view, quality in terms 
of positional accuracy, richness of tags and completeness of the map is 
secondary. I have much respect for this view and understand that getting 
data from an external database can be seen as a bit "boring" and taking 
the joy out of mapping.

There's also been concerns that the import would be made in a careless 
way, such as deleting and replacing with fully automatically translated 
data without review. There's also been concerns that NVDB which has a 
built in classification of roads in 9 levels does not match how OSM has 
classified roads (trunk->tertiary), and that these levels would replace 
the classification already done.

I would not say that these concerns are "irrelevant", I have much 
respect for these views and concerns. However, we have designed our 
import process in a way that in that context they are indeed irrelevant. 
NVDB is also in terms of data complementary, it doesn't have all the 
information OSM has, paths, and even some small roads are lacking, 
detail inside cities is lacking etc, it doesn't have any area polygons 
etc. The information it has is good, but there's still a large need to 
make local manual mapping as well, so it's certainly not a replacement 
for the traditional mapping. Just a help to keep the overall map in a 
high quality state, and relevant for Sweden, better positional accuracy 
which today is rather poor in many places.

As always, if a careless person start applying the automatic translated 
layer without following the appropriate manual merging process, these 
concerns are surely relevant. Therefore it's important to us in the 
community to follow up and see what people are actually doing so it's 
not misused. We present a toolbox and a process, we cannot stop anyone 
from misusing it until we detect it has happened. Fortunately there are 
quite many of us in the Swedish community that have an eye on various 
supervision tools to detect suspicious activity, just like what happened 
past year.

The process is detailed here (in Swedish)
https://wiki.openstreetmap.org/wiki/Import/Catalogue/Sweden_highway_import/Import_guidelines

A few details from it:

1) Only modify operations are used, never delete/insert, to keep history 
and make sure existing tags and relations are properly transferred. The 
key tool used here is JOSMs "replace geometry", which is used in a 
special workflow with matching segment splitting to make sure tag 
transitions happen in the right places and everything gets transferred 
and possible tag conflicts are manually resolved.

2) There are extensive documentation on how the NVDB translated layer 
should be pre-processed manually, adjusting highway types, making sure 
bridges are properly placed (the bridge data has unfortunately not good 
positional accuracy in NVDB), describing limitations in multi-lane roads 
and how these need manual additions etc.

3) A process of how to use the JOSM validator and interpret its warnings 
is documented, to go through issues in translated NVDB layer, and in the 
manual merging phase to make sure all roads are connected and no 
duplicates are accidentally left in.

4) A process of how to adjust positional errors of other features, like 
buildings and natural features, to match the improved accuracy of roads. 
In many places in Sweden the positional accuracy is so poor that if you 
just care about roads they will end up going right through buildings. In 
fact, in some municipalities there will be more work with correcting 
positional errors than actual upgrading the road geometry, and this too 
is an important goal for this project.

5) Suggestions of how to manually enrich the data with things like 
turning circles (prominent features in our forestry road network for 
example)

6) Discussions about different styles in NVDB geometry and pre-existing 
OSM geometry, especially in cities and how to relate to these, for 
example if cycleways are mapped separately or not, how noexit roads are 
mapped, turning circles etc.

7) NVDB is never used as the sole source in the merging process. You 
always use aerial photos, also make use of the Lantmäteriet raster map 
which is a great positional guide and provides some additional details, 
and of course carefully observe the previously existing data. Highway 
types are nearly always kept the same as previously mapped, unless those 
are obviously wrong.

Indeed there is zero relationship with this and the attack we had on 
Sweden last year, I followed that personally and was actually one of the 
"policing" guys which analyzed a lot of those accounts and the change 
sets and reported the activity. That import used some sort of old 
foreign database (not NVDB or any other publically available Swedish 
database) and was obviously highly automated and left roads unconnected, 
and did actually not provide much tags like maxspeed, road names and 
road surfaces which would be expected by a quality import from NVDB. We 
in the Swedish OSM community never figured out what database they used 
or what their motives were. It was surely not a helpful or careful 
import, even using hijacked accounts to some extent. There were no 
Swedish local mappers involved in that activity, we didn't have the 
ability to track it down but it seemed like it came from Russia and/or 
other eastern countries.

(For the regular mapping activity there are mostly Swedish mappers 
together with some very active and experienced mappers from the German 
community.)

Using our manual merging process, the mapping speed compared to 100% 
manual mapping is say 5-10 times higher, so it's fast, but not 
super-fast, no way near an automated import. Going through Sweden is 
expected to take a number of years. Going through a municipality of 
medium size outside the rural areas is about 80 hours of manual work. If 
there are detailed mapping with large positional errors in a mountainous 
region, it will take more time than that. Updating cities with several 
multi-lane roads and bridges and tunnels will also takes lots and lots 
of time, but we will not focus on dense cities to start with as these 
are already mapped with pretty decent quality.

I've made an announcement to talk-se as the process requries, no reply 
yet. But it was recent and there is very low activity in that group. 
Still waiting. I actually do not know about the visibility, there's 
relatively low activity in also in the Facebook group, lots of passive 
listeners which I do not know the names of. That I ended up there is by 
going through all the various forums etc to find where "all the action 
were", and while relatively low activity it was more than in the other 
places. (Facebook groups for all sorts of things is very popular in 
Sweden).

I do know about a few long-term guys active in the facebook group which 
are also active in some of the international mailing lists (not sure 
about talk-se specifically though), and also several prominent mappers 
far up in the Swedish top list, including those that are known to use 
NVDB data in their personal mapping and has published previous 
processes, like this: 
https://wiki.openstreetmap.org/wiki/%C3%96ppnadata_erfarenheter, and we 
have got endorsement from them.

We are not expecting a huge number of mappers to join in on this effort. 
Most mappers active in Sweden are casual iD users, and to do this type 
of work you need to be an experienced JOSM user preferably with plenty 
of time on your hands. I would personally expect 1 - 5 active people at 
any given time doing this work, although I hope for more.

/Anders

On 2021-03-23 23:50, Frederik Ramm wrote:
> Hi,
> 
> On 3/23/21 21:51, Anders Torger wrote:
>> This work has been in the open during the whole process and so far 
>> been discussed in the most popular informational communication channel 
>> in the Swedish OSM Community, which is a facebook group. It has been 
>> very well received, with a dash of the usual amount of healthy 
>> skepticism that any import effort generates.
> 
> Since the Facebook group does not have a publicly accessible archive,
> could you be so kind as to summarize the discussion, outlining what
> the "usual amount of healthy skepticism" was about and whether the
> different points of critique have been (a) addressed in the import
> plan or (b) considered irrelevant and if so, why?
> 
> There was a very large (90k changesets) undiscussed and quite harmful
> import of road data in Sweden end of last year where significant
> effort had been spent on making the import hard to detect and
> difficult to revert. If I remember correctly, that import suffered
> from having lots of small bits of track that connected to nothing. I
> hope there is zero relationship between that effort and what you are
> proposing here? Have any of the names on the list
> (https://lists.openstreetmap.org/pipermail/talk-se/2020-December/003928.html)
> popped up anywhere in the Facebook discussion?
> 
> Bye
> Frederik