[Talk-ca] [nzopengis] LINZ -> OSM data import scripts, Garmin codes
Sam Vekemans
acrosscanadatrails at gmail.com
Thu Jun 18 22:00:23 BST 2009
Hi all,
cc: talk-ca
You all seem to be having the same (IDENTICAL) issues that im (talk-ca
list) are having with the CanVec data, &geobase please join in on the
talk-ca list as your ideas are valuable, and of mutual benifit :-)
the talk-ca list has less than 100 subscribers.
With only a few of us working on geobase, and even fewer working on CanVec.
We too want to make the map Garmin happy, as well as conform to OSM standards.
And ALSO want to make the best of the available data.
Are there LINZ people who are also mappers? That helps.
Is there the same issue about how the data is sometimes old?
Hopefully your following along with the RoadMatcher progress, as well
as the latest for Vancouver Island CanVec sample area.
So far, the only solution (for roads) is manual attaching to OSM
data.This will make it routable. The downside is that its anoying, and
takes almost as long to just trace it.
Cheers,
Sam Vekemans
Across Canada Trails
On 6/18/09, Joe Richards <smoovjoe at gmail.com> wrote:
>
> I've been doing some work on the LINZ -> OSM data import, partly on
> the licensing/attribution area but mostly on the actual import.
> Graeme did an import sometime in 2008 which used the Python script in
> OSM's subversion area, but many of the types aren't converted and he
> was also told to stop.
>
> So I'm setting up a dev/test area for imports, and pushing ahead with
> the script. Here is a continuation of our emails, so that anyone can
> join in...
>
> > [Joe Richards wrote:]
> > I've improved the script that Graeme used a while ago which
> seems to be a
> > basic MP/Polish Map import, but it still needs a lot of work.
> Most of the
> > ways, nodes etc were coming through more-or-less untagged (apart
> from
> > import-specific tags) and so there's some matching up of the MP/
> Garmin
> > features to their OSM tag equivalents...
> >
> > So this will mean support for a lot more features (more stuff
> showing up
> > correctly), and ratification with NZ & talk on many of them (ie
> what tag
> > combinations are appropriate for specific feature types). I
> have a table of
> > extra ones to add, but I am going through and combing the maps
> for these
> > feature types one-by-one to see what sorts of features carry
> that type,
> > whether they fit with the proposed tag description and how they
> would be
> > rendered.
> Robin Paulson wrote:
> you might be better off using the api to download the nz data,
> then
> parsing it for various items - i'm sure that would be easier than
> doing it by hand, and more reliable too
>
> JR:
> Yup sorry, I wasn't clear, I mean I'm modifying my script to support
> more types, and then letting it rip on the data, and looking
> afterwards to see if it makes sense.
>
> JR:
> > Also levels ("EndLevel") I think corresponds to layers in OSM,
> but I haven't
> > explored that yet, ie what to do about bridges that go over
> other features
> > etc.
> >
> > Obviously there is also the question of how to merge this stuff
> with the
> > existing data. The Java-based RoadMatcher library might help
> with this, but
> > I haven't looked at it closely yet. Some places (e.g. Auckland,
> Wellington
> > and Christchurch) are fairly detailed already, and need tagging
> tweaks
> > (names etc) and a few missing features added. Other places
> (most of the
> > countryside and small towns) are pretty poor coverage.
> RP:
> i think a lot of the detailed areas would be better served by
> manually
> filling in what's missing, if it's only a small amount - such as
> owairaka and onehunga in auckland.
>
> JR:
> After we've checked that the import script is doing a good job with
> the source data, we should be able to start tackling this stuff. We
> might be able to import chunks of LINZ at a time, and let people merge/
> delete/combine things, until it looks sane, and then move onto the
> next chunk.
>
> RP:
> the problem with doing an automatic/scripted import in areas with
> data
> existing already, is what to do with the interface between the
> linz
> data and the stuff already there. for instance, what tolerance do
> we
> use for joining roads that aren't aligned, but close together?
> lots of
> questions around this
>
> JR:
> Yup, any automated approach will have quite a few limitations. I'm
> inclined to let people do it in whatever way makes sense to them. The
> good thing is big areas (e.g. countryside) will be mostly LINZ, but
> the rest might be a case of merging tags (to give one feature with a
> rich tagset) and essentially overlaying one over the other to see if
> any features need tweaking.
>
> RP:
> is there a list somewhere of what categories the nzogps data has?
> is
> it just roads, or are there parks, schools, churches, petrol
> stations,
> footpaths, etc, etc.? is all the nzogps data available for osm, or
> just the stuff that came from linz?
>
> Joe Richards:
> Attached (at the bottom of this posting) is a list of the Garmin types
> that the script has encountered processing the stuff from Graeme
> (which I assume is the latest nzogps). This is across the parts of NZ
> I found, which I think covers everything but e.g. Stewart Island etc.
> They are nasty hex codes like 0x6353 or whatever. Attached also is a
> list of the ones I found in Canterbury, with my notes about what they
> are, and how I approached tagging them (still in progress). You'll
> get more of an idea when I send you the dictionary in the conversion
> script.
>
> Robin Paulson:
> maybe you could put your script in svn now, with plenty of
> warnings
> that it's a work in progress, so we could all take a look at it?
> if
> that's ok? many eyes, etc.
>
> JR:
> Ok give me til tonight (morning your time) and I'll send you a link.
> I will heavily change the structure of the script over the next few
> days though, so if you want to contribute, it might be best to focus
> on the garmin type -> osm tags, and send me your comments until I put
> a proper version of the script up (I think this weekend).
>
>
> JR:
> > In other news:
> > * I got a development instance: linz.dev.openstreetmap.org but
> > unfortunately the dev server maintainer seems to be absent, so
> it's
> > basically a blank database and 0.6 API... Importing from JOSM
> seems to fail
>
> RP:
> cool. so we have a test area under dev? something we can do test
> imports to, before hitting the actual data servers?
>
> JR:
> In theory, although I haven't actually got it to import anything yet,
> it just freezes and uses up lots of memory on the server. I suspect
> the server is misconfigured, but feel free to test, point JOSM to
> http://linz.dev.openstreetmap.org/api
>
> My ideal setup is to have an API server running that works, and Mapnik
> generating tiles from it so we can see what an import looks like,
> without any other data there. Given that I don't have admin access on
> the dev server and apparently the server is short a dev admin, this
> might be best done elsewhere, we'll see. I might be able to get the
> API server going, in which case we can stay there, and just install
> mapnik, osmosis and osm2pgsql there. Then later we can put the
> existing data in, and talk about merge approaches.
>
> JR:
> > Tried an import (with massive memory haemorrage on the way)...
> Tiles seem to be coming from
> > the main server. We might have to setup our own servers to do
> this import
> > and get people looking at the results, but that's not impossible
> since I
> > have access to a few
> > * Have SVN write access, so I can commit the code when it's
> worth
> > commenting on
> >
> > Do you know anyone who has access to the actual LINZ data
> itself? Graeme
>
> RP:
> the guys at koordinates.com do. i've talked to ed through the
> auckland
> map meetup group, he was very keen to help osm out (i think sam
> you
> were there too?)
>
> JR:
> Can you put us in touch?
>
> JR:
> > can you comment on how the LINZ data dictionary/types have been
> translated
> > to Garmin types? What sort of 'native' format does NZ OpenGPS
> keep its data
> > in? It would be good to get access to the LINZ data and at
> least one
> > update, to see if/how we can approach future updates and to do
> comparisons.
> > For instance, there are SUFI identifiers in Graeme's extracts,
> but they
> > correspond sometimes to several ways (not just one), so I'm not
> sure what to
> > do with them.
>
> RP:
> could we move this to the nzopengis mailing list (where most osm
> discussion has taken place)? that way we can get others involved,
> and
> i won't miss someone out when i forget to do reply all
>
> http://groups.google.com/group/nzopengis
>
>
> Absolutely - good idea - continued on Google groups - hence this
> posting :)
>
> What follows is all the garmin/polish map codes encountered in the
> files found from Graeme, then followed by Canterbury with my notes (in
> progress). The actual script has better notes, so I'll get that in
> SVN soon...
>
> --
> Joe
>
> === canterbury codes ===
> Type=0x1 large urban area
> Type=0x13 steep track
> Type=0x14 railroad
> Type=0x16 path footway
> Type=0x160f ?
> Type=0x17 city park leisure=park area=yes
> Type=0x18 golf course leisure=golf_course
> Type=0x19 leisure=sports_centre
> Type=0x1a landuse=cemetery
> Type=0x1c boundary (political boundary) boundary=administrative
> admin_level=4
> Type=0x1d boundary (political boundary) boundary=administrative
> admin_level=4
> Type=0x1e00 TODO, Poi=MID SOUTH ISLAND REGION
> Type=0x1f #river
> Type=0x2 #principal hwy thick
> Type=0x28 sea/ocean [unmapped]
> Type=0x2b03 # tourism=caravan_site
> Type=0x2c06 + poi leisure=park [also reserve]
> Type=0x2c08 + poi leisure=track [for racing, either motor or horse],
> should have ideally sport=*
> Type=0x2d05 + poi leisure=golf_course
> Type=0x2d06 + poi sport=skiing
> Type=0x2f04 + poi aeroway=aerodrome [airport]
> Type=0x2f0b @@parking? TODO - shows up only on Queenstown Airport
> Type=0x3002 + poi amenity=hospital
> Type=0x3c #large lake natural=water
> Type=0x3e + natural=water
> Type=0x4 # highway=secondary
> Type=0x40 TODO polygon ???? water??
> Type=0x47 + polygon waterway=riverbank
> Type=0x48 # polygon waterway=riverbank
> Type=0x5 # way highway=tertiary
> Type=0x50 + polygon nature=wood [ also reserve]
> Type=0x5500 + poi waterway=dam
> Type=0x5800 @@ locked gate in reserve? TODO - in middle of forest
> Type=0x6 # highway=residential
> Type=0x600 @@ label Poi CHRISTCHURCH - TODO maybe large area label?
> Type=0x6402 + poi amenity=shelter, building=yes, tourism=alpine_hut
> for huts,bivouacs,cabins,shelters
> Type=0x6403 + poi amenity=grave_yard
> Type=0x6406 + poi mountain_pass=yes,place=locality TODO check
> (saddle=low, col=??, pass=high)
> Type=0x640a + poi place=locality
> Type=0x640c # poi man_made=mineshaft
> Type=0x6412 + poi highway=track, place=locality this is basically the
> start of a trekking walk/trek
> Type=0x6413 @@ cave?? TODO - current setup as tunnel=yes
> Type=0x6501 + poi natural=glacier
> Type=0x6508 # poi waterway=waterfall
> Type=0x650c # poi place=island
> Type=0x650d # poi place=locality TODO this is for lakes, check they
> have water too
> Type=0x6511 + poi spring
> Type=0x6512 + poi TODO waterway=* (canal, river diversion, gorge
> drain, drain, creek
> Type=0x6513 + poi natural=wetland (swamp etc)
> Type=0x6603 + poi landuse=basic (TODO check)
> Type=0x6607 + poi natural=cliff
> Type=0x660a + poi natural=wood (bush)
> Type=0x660d @@ TODO Goose's neck - only one there, let's check more of
> these
> Type=0x6611 @@ todo poi mountain range
> Type=0x6613 @@ todo poi mountain ridge
> Type=0x6616 # poi natural=peak [summit, top of hill or mountain]
> Type=0x6617 + poi place=locality [TODO: gully,gorge,trench missing]
> Type=0x7 + polyline highway=service [seemed to be residential but was
> used for carpark roads etc], polygon aeroway=aerodrome (areas)
> Type=0x8 + polyline highway=tertiary, turning=true
> Type=0x800 + poi place=town [ashburnton, timaru, oamaru, crowmwell,
> queenstown but not chch]
> Type=0x9 @@ ??? link? overpass?
> Type=0x900 + poi place=village
> Type=0xa # highway=track,surface=unpaved
> Type=0xb
> Type=0xb00 + poi place=locality (or hamlet, maybe even town?)
> Type=0xc
> Type=0xc00
> Type=0xe
> Type=0xe00
> ==== end canterbury codes ====
>
> ==== start all codes combined, no notes ====
> Type=0x1
> Type=0x1000
> Type=0x1100
> Type=0x1200
> Type=0x13
> Type=0x14
> Type=0x16
> Type=0x160f
> Type=0x17
> Type=0x18
> Type=0x19
> Type=0x1a
> Type=0x1b
> Type=0x1c
> Type=0x1c01
> Type=0x1c09
> Type=0x1d
> Type=0x1e
> Type=0x1e00
> Type=0x1f
> Type=0x2
> Type=0x20
> Type=0x2500
> Type=0x28
> Type=0x2800
> Type=0x2C0A
> Type=0x2D04
> Type=0x2D06
> Type=0x2F02
> Type=0x2F07
> Type=0x2F08
> Type=0x2F09
> Type=0x2a
> Type=0x2a02
> Type=0x2a03
> Type=0x2a04
> Type=0x2a05
> Type=0x2a07
> Type=0x2a08
> Type=0x2a0a
> Type=0x2a0e
> Type=0x2a0f
> Type=0x2b00
> Type=0x2b01
> Type=0x2b02
> Type=0x2b03
> Type=0x2c01
> Type=0x2c02
> Type=0x2c03
> Type=0x2c04
> Type=0x2c05
> Type=0x2c06
> Type=0x2c07
> Type=0x2c08
> Type=0x2c09
> Type=0x2c0b
> Type=0x2d
> Type=0x2d01
> Type=0x2d02
> Type=0x2d03
> Type=0x2d05
> Type=0x2d06
> Type=0x2d08
> Type=0x2d09
> Type=0x2e
> Type=0x2e01
> Type=0x2e02
> Type=0x2e03
> Type=0x2e04
> Type=0x2e05
> Type=0x2e07
> Type=0x2e08
> Type=0x2e09
> Type=0x2e0b
> Type=0x2f
> Type=0x2f00
> Type=0x2f01
> Type=0x2f03
> Type=0x2f04
> Type=0x2f05
> Type=0x2f06
> Type=0x2f08
> Type=0x2f0b
> Type=0x2f0c
> Type=0x2f0d
> Type=0x2f0f
> Type=0x2f11
> Type=0x2f17
> Type=0x3
> Type=0x3001
> Type=0x3002
> Type=0x3003
> Type=0x3004
> Type=0x3005
> Type=0x3008
> Type=0x3c
> Type=0x3d
> Type=0x3e
> Type=0x4
> Type=0x40
> Type=0x400
> Type=0x41
> Type=0x45
> Type=0x47
> Type=0x4700
> Type=0x48
> Type=0x4800
> Type=0x4a
> Type=0x4a00
> Type=0x4c
> Type=0x4d00
> Type=0x4e
> Type=0x4e00
> Type=0x5
> Type=0x50
> Type=0x500
> Type=0x5100
> Type=0x5200
> Type=0x5300
> Type=0x5400
> Type=0x5500
> Type=0x5800
> Type=0x5904
> Type=0x6
> Type=0x600
> Type=0x6300
> Type=0x6400
> Type=0x6402
> Type=0x6403
> Type=0x6404
> Type=0x6406
> Type=0x640a
> Type=0x640c
> Type=0x640d
> Type=0x640e
> Type=0x6411
> Type=0x6412
> Type=0x6413
> Type=0x6501
> Type=0x6508
> Type=0x6509
> Type=0x650c
> Type=0x650d
> Type=0x6511
> Type=0x6512
> Type=0x6513
> Type=0x6603
> Type=0x6604
> Type=0x6607
> Type=0x6608
> Type=0x660a
> Type=0x660d
> Type=0x6611
> Type=0x6613
> Type=0x6616
> Type=0x6617
> Type=0x7
> Type=0x700
> Type=0x8
> Type=0x800
> Type=0x9
> Type=0x900
> Type=0xa
> Type=0xa00
> Type=0xb
> Type=0xb00
> Type=0xc
> Type=0xc00
> Type=0xd
> Type=0xd00
> Type=0xe
> Type=0xe00
> Type=0xf00
> ==== end all codes combined ====
>
>
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups
> "nzopengis" group.
> To post to this group, send email to nzopengis at googlegroups.com
> To unsubscribe from this group, send email to
> nzopengis+unsubscribe at googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nzopengis?hl=en
> -~----------~----~----~----~------~----~------~--~---
>
>
--
Twitter: @Acrosscanada
Facebook: http://www.facebook.com/sam.vekemans
More information about the Talk-ca
mailing list