[OSM-talk] Administrative boundaries export

Thu Oct 3 12:00:00 UTC 2013

On Thu, Oct 03, 2013 at 12:32:36PM +0200, César Martínez Izquierdo wrote:
> Hi Frederik, regarding software, I am already familiar with Mapit scripts
> code, which are able to extract admin boundary polygons for each level (it
> is not creating relationships though). How do you see Nominatim or
> Osmium/osmjs better for the purpose? 

Nominatim already does a lot of the stuff you are planning to do. It 
creates geometries for admin boundaries from OSM data and puts them 
in a proper hierarchy. It is able to process updates and in the 
process makes sure that boundaries do not just disappear if somebody
breaks the relation. If you only process the data that interests you (boundary=administrative and place nodes) it is not even that 
resource-hungry.

It does have support for listing broken boundaries [1] and during the
last hack day Brian has written a proof-of-concept for browsing admin
hierarchies[2]. There is even a script to dump objects within a certain 
boundary[3] which could be easily extended to dump entire hierarchies.
All these functions should currently be used with care though. There are
known bugs and the output needs to be improved to make them really
usable.

Basically, most of the work to do would be on the output side, the 
heavy lifting of processing the data is already done. Well, apart 
from checking the OSM boundaries against reality. But I think wiki data 
is a good starting point for that.

> I will probably try the first option, but I expect that any of them would
> be costly to maintain if there are frequent changes of the tagging scheme
> for each country. (But nobody said it would be an easy task :-)

Making boundary tagging more visible will hopefully help to stablize
and unify the tagging schemas. The less country-specific exceptions
the better.

Sarah

[1] http://nominatim.osm.org/polygons
[2] http://nominatim.openstreetmap.org/hierarchy.php?place_id=97944985
    (This is for demonstration only, please do not scrape. It will not
		always give you the expected results anyways because there is a
		known bug with parenting which still lingers in the DB.)
[3] https://github.com/lonvia/Nominatim/blob/export-script/utils/export.php

> 2013/10/2 Frederik Ramm <frederik at remote.org>
> 
> > Hi,
> >
> > On 02.10.2013 18:23, César Martínez Izquierdo wrote:
> > > I plan to create and make easily available a world-wide administrative
> > > layer based on OSM data, ideally including existing administrative codes
> > > (ISO, NUTS in Europe, etc) for each level and producing regular updates
> > > (for instance once a year).
> >
> > This is something I have been thinking about for a long time (but never
> > written any usable code).
> >
> > Nominatim is probably a good starting point - a better one than MapIt I
> > should think.
> >
> > If you're only after extracting certain relation polygons then you could
> > as well use osmjs (part of Osmium) and have it generate shape files for
> > you, or adapt the shapefile/ogr export samples in Osmium; this will not
> > yet give you a hierarchy, only individual boundaries, and you have to
> > find out the hierarchy yourself.
> >
> > Finding out the hierarchy is going to be tricky. Nominatim does go to
> > some lengths to do that already. It sounds easy ("find all polygons with
> > an admin level smaller than X where this polygon I'm looking at lies
> > in"). But in reality you will encounter at least:
> >
> > * missing polygons on all levels - sometimes simply not mapped,
> > sometimes missing by design, e.g. Germany has some areas where admin
> > levels 8, 6, and 4 coincide, these are mapped as admin_level 4, so draw
> > a map of all admin_level 8 areas in Germany and you have lots of holes
> > in them
> >
> > * broken polygons on all levels; brokenness changes by the day, i.e.
> > what is working today may be broken tomorrow and vice versa
> >
> > * occasionally (e.g. Japan) linear regional boundaries that simply go
> > from coast to coast without including the coastline
> >
> > * occasionally (e.g. Chile) a regional boundary that is not a
> > multipolygon relation but instead a grouping of smaller regional entities
> >
> > * sometimes small geometric inaccuracies mean that e.g. a state boundary
> > fails the "is-in" test for the country boundary because there's just a
> > little square metre somewhere that is mapped as belonging to the state
> > but not the country
> >
> > * overlapping admin polygons of the same admin level
> >
> > I think that ate the very least you need to run the evaluation regularly
> > and compare: Do I have new polygons this week - have others vanished,
> > and if so, is that because they were explicitly deleted/replaced, or
> > were they just accidentally broken and I should continue to use last
> > week's?
> >
> > What we would really need though, is something much bigger: A separate
> > database of admin hierarchies, where people could - in a crowdsourced
> > manner - record things like:
> >
> > "There is an adminlevel 2 entities called Germany"
> > "It is divided into 16 exclusive adminlevel 4 entities with the
> > following names: ..."
> > "These 16 entities cover the area of Germany completely (no holes or sea
> > areas that would be outside of one of the entities)"
> > "The adminlevel 4 entity named 'Brandenburg' is divided in X adminlevel
> > 6 entities..."
> >
> > and so on. A tree of arbitrary size where people can add and edit at will.
> >
> > Now you will say "but this tree could be generated from OpenStreetMap",
> > and I grant that one could attempt to build such a tree but it will
> > always be faulty and reflect the current brokenness of geometries in
> > OSM. One could *start* with an OSM-generated tree, but after that, the
> > tree must be kept separate. People should be able to add stuff to the
> > tree even when it is not in OpenStreetMap - "there should be an
> > adminlevel 8 boundary called so-and-so". A regularly-running process
> > would then compare the tree to OpenStreetMap, and generate error reports
> > that can be presented visually:
> >
> > "The tree says that there should be a region called X in Germany but OSM
> > doesn't have one."
> >
> > "There is an area here that is not covered by any adminlevel 4 area but
> > the tree says that taken together the adminlevel 4 areas must cover all
> > of the country."
> >
> > "The tree claims there should be a region called X but in OSM there's
> > only a region with the similar name Y, which one is correct?"
> >
> > and so on. - I would expect the tree to be much more stable than the
> > data in OSM. Most of all, the tree could be worked on independently,
> > even by people unfamiliar with OSM. Of course the tree could link to OSM
> > objects but these links would regularly be checked and perhaps even
> > changed by the automated comparison system.
> >
> > As I said, a simple export is easy but it will have too many weaknesses
> > - you couldn't even say to what level a country is "complete". Some
> > people have started to link regional boundaries in OSM together with
> > concepts like "subarea" but I don't think this can replace an external
> > country structure tree because the tree could describe what is expected
> > to be there whereas in OSM you never know if some thing is missing
> > because it doesn't exist (cf. my example about the "missing" admin6/8
> > boundaries in Germany) or if it just hasn't been mapped yet, or is broken.
> >
> > It would be great to see someone drive this important topic but it
> > certainly isn't something you can set up in a week or two ;)
> >
> > Bye
> > Frederik
> >
> > --
> > Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09" E008°23'33"
> >
> > _______________________________________________
> > talk mailing list
> > talk at openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/talk
> >
> 
> 
> 
> -- 
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>    César Martínez Izquierdo
>    GIS developer
>    -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
>    ETC-SIA: http://sia.eionet.europa.eu/
>    Universitat Autònoma de Barcelona (SPAIN)
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

> _______________________________________________
> talk mailing list
> talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk