[Talk-us] NHD imports

Ben Supnik bsupnik at xsquawkbox.net
Mon Oct 29 14:04:09 GMT 2012

Hi Y'all,

First, I'm sorry that I didn't jump into this thread earlier.  I am the 
graphics lead on X-Plane, and for our latest version we switched from 
TIGER+VMAP0+swbd to OSM for our vector water body and road data.  In the 
case of the US, this is perceived as a step backward by our users 
because TIGER, while not as accurate as a lot of the OSM water data, is 
reasonably complete.  Now that we use OSM, we are "missing" lakes that 
were present in older sim versions.

At the time the consensus on various lists about importing was that 
individual authors should import relatively small areas (e.g. single 
water sheds) and check the import against existing data, rather than 
simply blasting in the entire data set.  So I wanted to solve two 
problems to make things easier for users who had time and motivation but 
not data processing skills:

1. Getting NHD data.  At the time the NHD data was not available in 
shape-file format; some very nice person at the USGS basically burned me 
DVDs of the whole data set and mailed them to me, because the data was 
not online.

2. The conversion required running a non-GUI tool, which meant having 
command line skills, etc.

So I ran the import script over the entire data set and uploaded the 
resulting .osm files so individuals could use them.  There was a third 
step I was hoping could be solved: if you new to OSM, importing data 
isn't trivial.  Richard Fairhurst had a cool feature in Potlatch 2 where 
PL2 could be told of the presence of a vector layer and import it 
visually.  But I was never able to get the data to work with PL2.

I had to put the whole project on hold as X-Plane's ship date loomed 
closer and things went from crazy to really-freaking-crazy at work.

So to answer a few of Paul's questions:

> - They're of 2011 data, not the latest NHD data

Yep -- at the time I started the project it was a rip of 'latest', but 
that was a while ago. :-)

> - I wasn't able to find a rules file from bsupnik for the conversion, so
> they can't be rerun with latest data. He hasn't been active on the lists
> since 2011 and hasn't replied

I used shp-to-osm-0.8.2, jar'd by someone else (I am not a big java 
nerd, sorry) and the latest rules at the time from the Wiki.  If it is 
useful, I can send anyone who wants them a rip of the rules file I was 
using at the time and/or the JAR, buuuut...

> - The NHD data model changed since 2011

Yeah, I was something like that a few months after I put this down and 
went "oh hell...".  I haven't had time to look at what happened; is the 
data model a major revision to the point where everything I did makes no 
sense, or is it just renaming of fields?

Other issues about the import:
- I wouldn't say it is "split" from multiple files - rather I would say 
it is not joined - that is, the splits are the sub-basin splits that I 
think persist in the NHD.  Actually, I take that back - I think the tool 
chain intentionally avoids really huge OSM entities to avoid imports 
that can fail.

- There should be OSM codes like waterway=bla on at least -some- of the 
data.  The import program rules from the time did include FCODE and 
FTYPE and had no facility to drop entities that had only NHD but no OSM 
tags.  At the time this was also considered not-so-good, but I wasn't in 
a position to rebuild the tool.

Finally, just my 0.02: I am not an OSM expert and not an import expert 
buuut in my uninformed opinion, having original 'foreign' keys on an 
imported database isn't very useful.  The import guideline is to sanely 
merge the import with existing data, and the goal is to create something 
'osm native' when done, hence the importance of OSM tags and the notion 
of not importing data with no OSM meaning.

I think OSM really needs to be treated as one giant layer with 
heterogenious tags.  Imagine that a lake is missing from NHD.  A user 
opens potlatch and draws it, puts water=lake, and walks away, no FCODE 
or GNIS_ID or anything like that.  How can a consumer of OSM data handle 
this case?  Only by looking at semantic data like water=lake, and by 
analyzing the union of all OSM data that has 'interesting' tags.

Now we go to re-import NHD - is this ever going to happen?  (First: 
given how hard it's been to get NHD in even once, I am skeptical. :-) 
Every watershed has to be hand re-merged again.

Where I'm going with this is: I think imports should not be viewed as 
'layers' into OSM; if you ever want to look at NHD with a layer, export 
OSM into your favorite GIS, import NHD into your GIS too and now you 
have layers.  OSM imports are destructive merges and need to be 
carefully managed as such, with the target data format being OSM-centric.

So at the time I was annoyed that we couldn't just import all of NHD at 
once and 'get lakes everywhere' but while that would get a lot of 
waterbodies into empty places, it's probably not good for OSM or the US 
mapping community.

Okay, that's the end of my rant, which is probably either something 
everyone else already knows or completely wrong or both. :-)  Poke me if 
you have more q's or need materials....I think I still have the original 
raw 2011 NHD on DVD if anyone wants it, although it is now stale.


Scenery Home Page: http://scenery.x-plane.com/
Scenery blog: http://www.x-plane.com/blog/
Plugin SDK: http://www.xsquawkbox.net/xpsdk/
X-Plane Wiki: http://wiki.x-plane.com/
Scenery mailing list: x-plane-scenery at yahoogroups.com
Developer mailing list: x-plane-dev at yahoogroups.com

More information about the Talk-us mailing list