[Talk-us] NHD imports

Mon Oct 29 15:36:20 GMT 2012

Hi everybody,

as an active mapper focused on the topology consistence of the map, I don't
think that any automatic re-import would make any good. The initial bulk
imports left the map in the terrible state and I can't even imagine how
badly it would be damaged if all the NHD data is re-imported.

I think it might be nice to have a simple, even text-like (Russians mappers
have something similar<http://translate.google.com/translate?hl=en&sl=ru&tl=en&u=http%3A%2F%2Fvwo.osm.rambler.ru%2F>),
web interface to NHD data so mappers would be able to check water objects
manually and make a decision on re-importing and validating them.

Best regards,
Ivan.

On Mon, Oct 29, 2012 at 9:04 AM, Ben Supnik <bsupnik at xsquawkbox.net> wrote:

> Hi Y'all,
>
> First, I'm sorry that I didn't jump into this thread earlier.  I am the
> graphics lead on X-Plane, and for our latest version we switched from
> TIGER+VMAP0+swbd to OSM for our vector water body and road data.  In the
> case of the US, this is perceived as a step backward by our users because
> TIGER, while not as accurate as a lot of the OSM water data, is reasonably
> complete.  Now that we use OSM, we are "missing" lakes that were present in
> older sim versions.
>
> At the time the consensus on various lists about importing was that
> individual authors should import relatively small areas (e.g. single water
> sheds) and check the import against existing data, rather than simply
> blasting in the entire data set.  So I wanted to solve two problems to make
> things easier for users who had time and motivation but not data processing
> skills:
>
> 1. Getting NHD data.  At the time the NHD data was not available in
> shape-file format; some very nice person at the USGS basically burned me
> DVDs of the whole data set and mailed them to me, because the data was not
> online.
>
> 2. The conversion required running a non-GUI tool, which meant having
> command line skills, etc.
>
> So I ran the import script over the entire data set and uploaded the
> resulting .osm files so individuals could use them.  There was a third step
> I was hoping could be solved: if you new to OSM, importing data isn't
> trivial.  Richard Fairhurst had a cool feature in Potlatch 2 where PL2
> could be told of the presence of a vector layer and import it visually.
>  But I was never able to get the data to work with PL2.
>
> I had to put the whole project on hold as X-Plane's ship date loomed
> closer and things went from crazy to really-freaking-crazy at work.
>
> So to answer a few of Paul's questions:
>
>
>  - They're of 2011 data, not the latest NHD data
>>
>
> Yep -- at the time I started the project it was a rip of 'latest', but
> that was a while ago. :-)
>
>
>  - I wasn't able to find a rules file from bsupnik for the conversion, so
>> they can't be rerun with latest data. He hasn't been active on the lists
>> since 2011 and hasn't replied
>>
>
> I used shp-to-osm-0.8.2, jar'd by someone else (I am not a big java nerd,
> sorry) and the latest rules at the time from the Wiki.  If it is useful, I
> can send anyone who wants them a rip of the rules file I was using at the
> time and/or the JAR, buuuut...
>
>
>  - The NHD data model changed since 2011
>>
>
> Yeah, I was something like that a few months after I put this down and
> went "oh hell...".  I haven't had time to look at what happened; is the
> data model a major revision to the point where everything I did makes no
> sense, or is it just renaming of fields?
>
> Other issues about the import:
> - I wouldn't say it is "split" from multiple files - rather I would say it
> is not joined - that is, the splits are the sub-basin splits that I think
> persist in the NHD.  Actually, I take that back - I think the tool chain
> intentionally avoids really huge OSM entities to avoid imports that can
> fail.
>
> - There should be OSM codes like waterway=bla on at least -some- of the
> data.  The import program rules from the time did include FCODE and FTYPE
> and had no facility to drop entities that had only NHD but no OSM tags.  At
> the time this was also considered not-so-good, but I wasn't in a position
> to rebuild the tool.
>
> Finally, just my 0.02: I am not an OSM expert and not an import expert
> buuut in my uninformed opinion, having original 'foreign' keys on an
> imported database isn't very useful.  The import guideline is to sanely
> merge the import with existing data, and the goal is to create something
> 'osm native' when done, hence the importance of OSM tags and the notion of
> not importing data with no OSM meaning.
>
> I think OSM really needs to be treated as one giant layer with
> heterogenious tags.  Imagine that a lake is missing from NHD.  A user opens
> potlatch and draws it, puts water=lake, and walks away, no FCODE or GNIS_ID
> or anything like that.  How can a consumer of OSM data handle this case?
>  Only by looking at semantic data like water=lake, and by analyzing the
> union of all OSM data that has 'interesting' tags.
>
> Now we go to re-import NHD - is this ever going to happen?  (First: given
> how hard it's been to get NHD in even once, I am skeptical. :-) Every
> watershed has to be hand re-merged again.
>
> Where I'm going with this is: I think imports should not be viewed as
> 'layers' into OSM; if you ever want to look at NHD with a layer, export OSM
> into your favorite GIS, import NHD into your GIS too and now you have
> layers.  OSM imports are destructive merges and need to be carefully
> managed as such, with the target data format being OSM-centric.
>
> So at the time I was annoyed that we couldn't just import all of NHD at
> once and 'get lakes everywhere' but while that would get a lot of
> waterbodies into empty places, it's probably not good for OSM or the US
> mapping community.
>
> Okay, that's the end of my rant, which is probably either something
> everyone else already knows or completely wrong or both. :-)  Poke me if
> you have more q's or need materials....I think I still have the original
> raw 2011 NHD on DVD if anyone wants it, although it is now stale.
>
> cheers
> Ben
>
>
> --
> Scenery Home Page: http://scenery.x-plane.com/
> Scenery blog: http://www.x-plane.com/blog/
> Plugin SDK: http://www.xsquawkbox.net/**xpsdk/<http://www.xsquawkbox.net/xpsdk/>
> X-Plane Wiki: http://wiki.x-plane.com/
> Scenery mailing list: x-plane-scenery at yahoogroups.**com<x-plane-scenery at yahoogroups.com>
> Developer mailing list: x-plane-dev at yahoogroups.com
>
>
> ______________________________**_________________
> Talk-us mailing list
> Talk-us at openstreetmap.org
> http://lists.openstreetmap.**org/listinfo/talk-us<http://lists.openstreetmap.org/listinfo/talk-us>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20121029/a65e901c/attachment-0001.html>