[Talk-dk] open source days & værktøjer til udveksling af geodata

[Emil Tin]

Hej,
Jeg synes det er fantastisk at der snart bliver åbnet op for adgangen til en masse grunddata! :)

Sidste år blev der afholdt en spændende workshop omkring værktøjer til at gøre det lettere at sammenligne og udveksle geodata mellem OSM og offentlige myndigheder. I forlængelse af dette snakkede jeg på sidste Cph Data Drinks med to personer fra Open Source Days i Danmark, der arbejder med "geodata" som tema på deres event i foråret.

De mente at temaet omkring open source værktøjer til udveksling af data mellem OSM og offentlige myndigheder kunne være spændende at arbejde med i forbindelse med deres event - og ekstra relevant når der nu bliver åbnet op for adgangen til FOT data, osv.

Jeg har tidligere skrevet nedenstående oplæg, som jeg derfor sendte til Open Source Days.

Hvad tænker i som OSM brugere? Hvad er behovet? Hvordan kan man bedst involvere, engagere og fokusere open source udviklere, OSM brugere og kommuner/myndigheder omkring dette tema i forbindelse med Open Source Days i foråret 2013?

Venlig hilsen
Emil Tin

...........


Open source tools for exchanging data between OpenStreetMap and public authorities

Background
OpenStreetMap is gaining increased popularity, and is already being used for many purposes, including route planners and mobile apps.
Both public authorities and OpenStreetMap could potentially gain a lot from more effective ways of exchanging and comparing data. The different datasets have different strengths. Data exchange could help improve the map quality on both sides.
OSM is often very detailed and updated where many people live, but is still often a bit lacking in more remote areas. Data from authorities a often has a more even quality, but can be outdated, or might lack details like bicycle infrastructure, which is often well mapped in OSM.
20+ municipalities in the Copenhagen area are currently developing a bicycle route planner based on OSM data. Copenhagen has gone through a manual process of comparing internal GIS map data with OSM data, as well as updating both datasets. The process has been very useful, and has resulted in higher quality data on both sides.
However, the process has taken quite a bit of time and energy, and might be impractical for other municipalities. More automated tools would be very useful for many municipalites and OSM volunteers. Better map data would also benefit a large number of projects using OMS data.
A workshop was held in Copenhagen in the winter of 2011, with representatives from municipalites, road authoroties, OSM volunteers, and other, exploring ideas for how to exchange map data. Ideas from the workshop are listed later in this document.

Purpose
The purpose of the project is to make it easier and cheaper to exchange map data between public authorities and OpenStreetMap, by developing one or more suitable open source tools.

Goals
The goal is to have a functional tool by summer 2013.

The initial target area is 20+ municipalities in the Copehagen area in Denmark, as well as danish OSM volunteers, but the tools should have international appeal. More widespread use will benefit everyone.

The tool must be develed as open source and released under an open source software license approved by Open Source Initiate.

An important part of the task will be to understand the involved use-cases, and decide what is possible within the allocated ressources and time.

Should we settle for a tool for easier visual comparison? Or should we build a issue tracking system that integrates with the GIS systems of municipalities?

Could we use or extend the merging tool for UK Ordinance Survey data created by CycleStreets.net? (See Inspiratation below.)

Many municipalties do not yet have experience with OSM. They will only be interesting in using the tool if they can see immediate benefits.
Future operating costs needs to be considered carefully. Setting up a central server is not entirely impossible, but tools that integrating into existing GIS systems, bug trackers, route planners or OSM infrastructure will probably be more attractive.

Tasks
1.      Understand the needs and use-cases of authorities and OSM.
2.      Develop a concept for one or more open source tools.
3.      Impement the tool(s).
4.      Test, evaluate and document the tool(s)

Challenges
Copy-paste or other mass-imports is generally not possible nor attractive, because data formats are different, or it would result in inconsistencies, poor map quality or valid data being overwritten. Instead each detail usally needs to be evaluated by a human.

One of the attractions of OSM is that it covers the world in a single consistent map format. In contrast to this, municipalities use a plurality of systems, formats, projections, etc. The tools should be useful by a wide range of authorities and organizations.

GIS is usually based on SQL databases with a fixed set of attributes, while OSM can can have arbirtrary tags on each object. GIS usually works with multiple layers, while OSM have just a single layer, which you can then filter.

Objects are often modellen a bit differently  in OSM and other systems. A big street could be modelled as either a single street, or as two separate one-way streets. This needs to be taken into account when comparing maps.

Inspiration
GitHub's issue tracking system, wich integretes with repository commits:
https://github.com/blog/411-github-issue-tracker/

Import of UK Ordinance Survey data:
http://wiki.openstreetmap.org/wiki/Ordnance_Survey_Opendata#New_ideas_and_automated_imports<http://wiki.openstreetmap.org/wiki/Ordnance_Survey_Opendata>

Merging tool for Ordinance Survey data in Potlatch 2:
http://www.cyclestreets.net/blog/2011/10/24/osm-merging-tool/
http://wiki.openstreetmap.org/wiki/Snapshot_Server


Ideas
Issue Tracking
Inspired by issue trackers used for software development, the tool could be based around tracking individual map "issues".

Automated comparison and analysis could create such issues, which can then be reviewed and processed by humans.

Suppose a street is added in OSM, which the corresponding municipality does not yet have in their GIS system. The tool creates an issue, with details about the street, location, attributes, etc, and notifies the municipality. Someone from the municiapality can the easily review the issue, and choose what to do. Perhaps the tool could provide an easy way to insert the street into the GIS system.

In case the municipality adds a street, OSM volunteers would be notified. Perhaps an OSM changeset could be constructed automatically, which OSM volunteers can choose accept. This enables easy tracking and rollback of changes in OSM.

Municpilaties and OSM volunteers should be able to specify what kind of changes they're interested in. Tey could also choose just to run a comparison manually.

Issue could have various states: open, rejected, need to verify in the field, closed, invalid, etc. Many issue tracking systems also provide the ability to assign priority, person, etc to issue.

Perhaps an existing open source issue tracker could be used as the starting point.

Mutual notification

*       Two-way notifications. Authorities will be notified when there are relevant changes in OSM, and vice versa.
*       Authorities could write to OpenStreetBugs.
*       Changesets which the counterparty can accept. If a municipality builds a new bike path, an OSM changeset could be automatically generated. OSM volunteers can review and accept if it looks good. If they accept it, the changes are automatically integrated into OSM via the normal editing OSM editing systems. It could also work the other way around.
*       Mutual advice about roadworks.
*       Sometimes things incorrect edits are performed. Usually they're correctly shortly after. Perhaps the other party would only be notified when an edit has persisted for a certain period of time.

Comparison

*       Mapping of roads between OSM, GIS systems, and road databases. Perhaps by geographic location, street names, road type, possibly another?
*       Translating GIS attributes to OSM tags. Perhaps through a profile that each municipality can set up.
*       Tool that compares datasets, and spits out the differences. In cases where public data cannot be releases into the public, perhaps the authority could run such a comparison and post the results.
*       'Invisible' things, such as turn restrictions, are often lacking in OSM. Comparing with road registers and sign databases could be useful.
*       Use map comparison to identify topological errors, such as unconnected ways, improper alignment, etc. Errors that could otherwise be difficult to find..
*       Use route calculation to find errors in the map and present the results visually. Run the same route calculation on OSM and data from public authorities, and comparing routes.

Visual inspection

*       You should be able to compare both points lines and surfaces.
*       Viewing the differences between OSM and authority data, both in list form, and on a map.
*       Using data from public authorities as background bitmap (tile) maps in OSM. Since it's only bitmap, this could be a way around public data that cannot yet be releases for legal reasons.
*       Individual street networks are colored according to how recently they were modified. Once you've reviewed a certain street, you could mark it as 'reviewed' and it would fade visually. This would make it easy to review changes in the other dataset. Especially if you could filter according to attributes, road type, etc.


Credibility

*       Establish a "reliability score" in OSM, on the basis the source of data, when data were last checked, etc. Is it from aerial, GPS tracks, or just from the address points? The score may decrease with time.
Use GPS tracks to determine reliability. If there are many tracks in the same place, the credibility larger, and can help to ensure quality control.