[Imports] import of data from LINZ

Paul Norman penorman at mac.com
Tue Aug 21 03:06:08 BST 2012

Trimming the replies down to keep the size of email manageable, but it's
good to get some review and discussion going on.

> From: Robin Paulson [mailto:robin.paulson at gmail.com]
> Sent: Monday, August 20, 2012 4:45 PM
> To: imports at openstreetmap.org
> Subject: Re: [Imports] import of data from LINZ
> Note those (the area previously cited) are from the old dataset, the new
> ones are much cleaned up.

How can we see these new areas to review them?

> > But did you look at just the LINZ documentation or did you verify that
> > the mappings are correct with imagery or local knowledge?
> 1. There is no 100% local coverage for decent aerial maps in NZ. We use
> them when available to check the data we import. For examples some of
> the outlying islands don't even show on the satellite imagery it is that
> low quality/cloud cover, etc.
> 2. New Zealand is a nation on 4 million people in a country the size of
> the Great Britain. We cannot rely on local knowledge for all of the data
> especially those items in the middle of wilderness areas with little to
> no human access.
> LINZ however did generate a lot of their data from other aerial
> surveys/site visits and review content on a rotating basis. Although not
> 100% accurate their data provides the base for the NZ community to move
> forward with a decent base map that will be enhanced and cleaned over
> time.

The point is to make sure that each mapping from a LINZ tag to an OSM tag
has been verified, not each individual object. For example, with the US NHD
the feature code 46006 was imported as waterway=river. This was reasonable
based on the written description but in practice >95% of these "rivers" were
really waterway=stream. 

> We know there is a general leaning towards hand edits are best and this
> works well where there is a dense population base. It will be far easier
> for us to engage the wider mapping community once we have a base layer
> of decent data that requires smaller amounts of hand editing rather than
> an empty match. We believe the trade off in this instance is worth the
> small amount of incorrect data.

Just to clarify, my comments were not about the value of imports for use in
mapping remote regions, it was about the process of writing the conversions
from data source attributes to OSM tags which experience has shown cannot be
done accurately purely based on the descriptions from the data source.

The question of if the import in general is worth it is an issue better left
to the local community. 

> On a related noted our road data will be coming from the NZ Open GPS
> project which has taken the LINZ road data and added a lot of meta data,
> removed paper roads and generally made it good enough for GPS units.

Is this part of this import, or a new import that will be proposed at a
later date?

> We will look into how we tag attribution and source. A source tag is
> important for us for the update process but we can move the attribution
> somewhere else if we can automate its insertion so it doesn't
> accidentally get left off.

Although it is possible for tags to be accidentally left off of a changeset,
attribution=* can be edited and the data source would have to be okay with
attribution from the history in that case.

I have also seen the attribution tag inspire reluctance in editing by newer
users who are not sure if they should edit the tag when combining multiple
sources of information.

> It is important to emphasize that the LINZ:layer and LINZ:dataset tags
> are what will make later data releases from the gov't able to be
> incorporated (pre-per-feature ID tags), and bulk corrections to already
> uploaded data possible.

Are there features with the same OSM tags that are found in multiple LINZ
layers? If so then I can see the need, but if not then the LINZ layer can be
inferred by the tagging of the object.

> > It looks like some display information from their database made it in.
> Yes, that was done on purpose in case someone wanted to use it when
> cleaning up/merging the tags to see what/where offsets vs the nearby map
> features and how important the thing was (size). Also if anyone wanted
> to use it for cartography.
> Those fields are gone now in the new release.

Good to hear. Can you post a new .osm file of an area with the new tagging?

> > http://www.openstreetmap.org/browse/node/767114113 has tags which
> > belong on the nearby waterway=river
> This is part of a post import cleanup process that will be under taken.
> There is a description layer which labels some things like rivers (some
> rivers also have tags directly). Post import we plan to have a few check
> scripts that look for hotspots to review and hand edit. We expect the
> whole import process to take months once we get it up and running but we
> will get there.
> > http://maps.paulnorman.ca/imports/review/streamconverge.png is a point
> > where three waterways converge on one spot. It appears that some
> > waterways are reversed.
> Yes - We can't gaurentee the direction of rivers right now. LINZ are
> working on fix this in the base data. One of the notes on importing
> further rivers is to check direction and fix if we can work out which
> way is downhill...

Ah, I didn't realize it was an issue inherited from the source. There's not
really a great way to handle directionality of waterways in OSM. The default
assumption is that they're directional but there's no widely accepted way to
indicate the directionality is unknown or that it has no directionality.
There's the directional=* proposal but that hasn't seen much usage.

I'm going to have to tackle this issue with the US NHD data where some
waterways are indicated as having no direction so I'd be interested in what
you work out.

> We have a post import task to review these. Interestingly the point
> description layer has a tag called beach which will allow us to zoom on
> quite a few of these areas.
> Also: The sat imagery is misleading, we believe they are tall (10m+)
> dunes and not a classic beach front.

I guess then you could tag them either way then, OSM tagging being what it
is :)

> > The display information on some nodes is the most serious issue, but
> > it should be easy to fix.
> We will look at the tagging of the multi-ploygons and review the
> attribution issues.
> > Overall it looks not bad, but I look forward to seeing an updated .osm
> > file with these issues fixed.
> We will be doing some more targeted imports to check tags on the layers
> over the next week. We will keep the group posted/

You should not be doing any imports while you are still developing the
tagging. If you need to test scripts you can do it on the dev APIs, but the
live API is not for testing imports. 

More information about the Imports mailing list