[Talk-us] NHD imports
bsupnik at xsquawkbox.net
Mon Oct 29 14:04:09 GMT 2012
First, I'm sorry that I didn't jump into this thread earlier. I am the
graphics lead on X-Plane, and for our latest version we switched from
TIGER+VMAP0+swbd to OSM for our vector water body and road data. In the
case of the US, this is perceived as a step backward by our users
because TIGER, while not as accurate as a lot of the OSM water data, is
reasonably complete. Now that we use OSM, we are "missing" lakes that
were present in older sim versions.
At the time the consensus on various lists about importing was that
individual authors should import relatively small areas (e.g. single
water sheds) and check the import against existing data, rather than
simply blasting in the entire data set. So I wanted to solve two
problems to make things easier for users who had time and motivation but
not data processing skills:
1. Getting NHD data. At the time the NHD data was not available in
shape-file format; some very nice person at the USGS basically burned me
DVDs of the whole data set and mailed them to me, because the data was
2. The conversion required running a non-GUI tool, which meant having
command line skills, etc.
So I ran the import script over the entire data set and uploaded the
resulting .osm files so individuals could use them. There was a third
step I was hoping could be solved: if you new to OSM, importing data
isn't trivial. Richard Fairhurst had a cool feature in Potlatch 2 where
PL2 could be told of the presence of a vector layer and import it
visually. But I was never able to get the data to work with PL2.
I had to put the whole project on hold as X-Plane's ship date loomed
closer and things went from crazy to really-freaking-crazy at work.
So to answer a few of Paul's questions:
> - They're of 2011 data, not the latest NHD data
Yep -- at the time I started the project it was a rip of 'latest', but
that was a while ago. :-)
> - I wasn't able to find a rules file from bsupnik for the conversion, so
> they can't be rerun with latest data. He hasn't been active on the lists
> since 2011 and hasn't replied
I used shp-to-osm-0.8.2, jar'd by someone else (I am not a big java
nerd, sorry) and the latest rules at the time from the Wiki. If it is
useful, I can send anyone who wants them a rip of the rules file I was
using at the time and/or the JAR, buuuut...
> - The NHD data model changed since 2011
Yeah, I was something like that a few months after I put this down and
went "oh hell...". I haven't had time to look at what happened; is the
data model a major revision to the point where everything I did makes no
sense, or is it just renaming of fields?
Other issues about the import:
- I wouldn't say it is "split" from multiple files - rather I would say
it is not joined - that is, the splits are the sub-basin splits that I
think persist in the NHD. Actually, I take that back - I think the tool
chain intentionally avoids really huge OSM entities to avoid imports
that can fail.
- There should be OSM codes like waterway=bla on at least -some- of the
data. The import program rules from the time did include FCODE and
FTYPE and had no facility to drop entities that had only NHD but no OSM
tags. At the time this was also considered not-so-good, but I wasn't in
a position to rebuild the tool.
Finally, just my 0.02: I am not an OSM expert and not an import expert
buuut in my uninformed opinion, having original 'foreign' keys on an
imported database isn't very useful. The import guideline is to sanely
merge the import with existing data, and the goal is to create something
'osm native' when done, hence the importance of OSM tags and the notion
of not importing data with no OSM meaning.
I think OSM really needs to be treated as one giant layer with
heterogenious tags. Imagine that a lake is missing from NHD. A user
opens potlatch and draws it, puts water=lake, and walks away, no FCODE
or GNIS_ID or anything like that. How can a consumer of OSM data handle
this case? Only by looking at semantic data like water=lake, and by
analyzing the union of all OSM data that has 'interesting' tags.
Now we go to re-import NHD - is this ever going to happen? (First:
given how hard it's been to get NHD in even once, I am skeptical. :-)
Every watershed has to be hand re-merged again.
Where I'm going with this is: I think imports should not be viewed as
'layers' into OSM; if you ever want to look at NHD with a layer, export
OSM into your favorite GIS, import NHD into your GIS too and now you
have layers. OSM imports are destructive merges and need to be
carefully managed as such, with the target data format being OSM-centric.
So at the time I was annoyed that we couldn't just import all of NHD at
once and 'get lakes everywhere' but while that would get a lot of
waterbodies into empty places, it's probably not good for OSM or the US
Okay, that's the end of my rant, which is probably either something
everyone else already knows or completely wrong or both. :-) Poke me if
you have more q's or need materials....I think I still have the original
raw 2011 NHD on DVD if anyone wants it, although it is now stale.
Scenery Home Page: http://scenery.x-plane.com/
Scenery blog: http://www.x-plane.com/blog/
Plugin SDK: http://www.xsquawkbox.net/xpsdk/
X-Plane Wiki: http://wiki.x-plane.com/
Scenery mailing list: x-plane-scenery at yahoogroups.com
Developer mailing list: x-plane-dev at yahoogroups.com
More information about the Talk-us