[Talk-us] NHD imports

Paul Norman penorman at mac.com
Mon Oct 29 15:47:04 GMT 2012

This is only a partial reply - I should have more detail this afternoon when
I have more time.

> From: Ben Supnik [mailto:bsupnik at xsquawkbox.net]
> Subject: Re: [Talk-us] NHD imports
> 2. The conversion required running a non-GUI tool, which meant having
> command line skills, etc.

This is actually a plus for me - handling all of NHD without command line
scripts would be difficult. There's a *lot* of data.

> So I ran the import script over the entire data set and uploaded the
> resulting .osm files so individuals could use them.  There was a third
> step I was hoping could be solved: if you new to OSM, importing data
> isn't trivial.  Richard Fairhurst had a cool feature in Potlatch 2 where
> PL2 could be told of the presence of a vector layer and import it
> visually.  But I was never able to get the data to work with PL2.

This is actually mostly solved now -
http://www.cyclestreets.net/edit/locate/ has a PL2 instance which has a
vector layer background for all of the UK.

There are a few issues like loading many GB of data into the server
supplying the vector layer and making it available without deploying your
own PL2 instance, but these are fairly minor.
> I used shp-to-osm-0.8.2, jar'd by someone else (I am not a big java
> nerd, sorry) and the latest rules at the time from the Wiki.  If it is
> useful, I can send anyone who wants them a rip of the rules file I was
> using at the time and/or the JAR, buuuut...

Ah, good to know.

> > - The NHD data model changed since 2011
> Yeah, I was something like that a few months after I put this down and
> went "oh hell...".  I haven't had time to look at what happened; is the
> data model a major revision to the point where everything I did makes no
> sense, or is it just renaming of fields?

The common FCodes have remained the same but some of the less common ones
have changed.

> Other issues about the import:
> - I wouldn't say it is "split" from multiple files - rather I would say
> it is not joined - that is, the splits are the sub-basin splits that I
> think persist in the NHD.  Actually, I take that back - I think the tool
> chain intentionally avoids really huge OSM entities to avoid imports
> that can fail.

The splits I'm talking about are being split into layers. You have lakes on
one layer and rivers on another, etc.

The splits by area remain similar although I believe the pre-staged files
cover a larger area (HUC4 basins) than what you were given did.

> - There should be OSM codes like waterway=bla on at least -some- of the
> data.  The import program rules from the time did include FCODE and
> FTYPE and had no facility to drop entities that had only NHD but no OSM
> tags.  At the time this was also considered not-so-good, but I wasn't in
> a position to rebuild the tool.

Maybe I picked an odd region or layer. There are some areas in the NHD that
are quite different from all the others and use FCodes in different ways. I
should know the dangers of concluding something by looking only at one part
of NHD :)

> So at the time I was annoyed that we couldn't just import all of NHD at
> once and 'get lakes everywhere' but while that would get a lot of
> waterbodies into empty places, it's probably not good for OSM or the US
> mapping community.

That's my view too.

More information about the Talk-us mailing list