[HOT] Database, OSM & HOT (Was: Request for information about common set of tags for HOT)

[Blake Girardot]

Hi Springfield,

Here is how I get useful thematic layers out OSM:

http://wiki.openstreetmap.org/wiki/User:Bgirardot/How_To_Convert_osm_.pbf_files_to_Esri_Shapefiles

And here is an example files generated through what I would guess is a 
similar process every 30 mins:

http://nepal.piensa.co/

Actually, I see they use a slightly different process with the same 
basic method, and the same software for the conversion/extraction:

https://github.com/GFDRR/osm-extract

(feedback on my thematic layers is always welcome, we want to create the 
most useful layers we can. Examples can be found in the wiki entry for 
Vanuatu typhoon response)

cheers,
Blake



On 5/22/2015 8:26 AM, Springfield Harrison wrote:
> Hello John,
>
> Thanks for your patient explanation, I'm beginning to see that OSM is a
> very different flavour of GIS.  At the outset, my assumption was that it
> was entirely emergency oriented.  I was puzzled by the references to
> hairdressers and gymnasiums but I guess they result from a different
> process.
>
> I do think that some emergency related features such as potential
> helipads, powerline crossings, towers, cable cars, landslides, glacial
> lakes, emergency shelters and such like might be better left to those
> with experience with those types of features.  They wouldn't necessarily
> need to be experienced with OSM, just familiar with identifying those
> features.  I'm surprised that there is no process for identifying and
> directing the more highly qualified mappers.
>
> I had intended to help with the helipad project but quickly became
> discouraged with the less than adequate imagery and the weirdness of
> leisure = common.  Merely verifying the leisure = common sites would
> probably overlook lots of other qualified sites.  And how many sites
> with this tag are actually sports fields as per the original intention?
> Then, mapping existing helipads marked with H in a circle, might be
> redundant as such official sites would probably be already mapped by a
> national agency.  I would recommend that potential helipads be tagged as
> aeroway = helipads_potential, verified = no.  Proper assessment of
> helipads requires an oblique, 3-D view.  I attempted to introduce Google
> Earth into the process but licensing fears put the kibosh on that.
>
> I found this surprising because Google Earth does have several other
> products and does make a lot of noise about community and not for profit
> mapping without any references to licensing.  They appear to actively
> promote user generated files being placed into the public domain.  I
> have spent some time attempting to talk to them about this but the best
> I could do was an e-mail.  Will advise.
>
> Thanks again for your time on this, I'm sure you have larger fish to
> fry, Cheers . . . . . . . . Spring Harrison
>
>
> At 20-05-2015 12:01 Wednesday, john whelan wrote:
>> OSM has a page of recommended tags,
>> http://wiki.openstreetmap.org/wiki/Map_Features. Sometimes these are
>> used sometimes not. http://taginfo.openstreetmap.org/ has information
>> on how common a tag is and is some times used to determine which tag
>> should be used. This is a bottom up approach rather than the top down
>> approach more usual in the business world.
>>
>> For example many of the mappers locally are enthusiastic cyclists but
>> the recommended tags for cycle paths have been formulated in Europe,
>> in Canada we have some cycle paths / lanes that are used for cyclists
>> in summer and we dump snow on them in winter. There is a local
>> convention for how these are tagged.
>>
>> With a conventional database you usually have a client in mind and
>> they have specific requirements. OSM doesn't. If someone wants to
>> map hairdressers that's fine as far as OSM is concerned they have
>> contributed to the map. Locally many tags were in OSM that wouldn't
>> render on conventional rendering systems, no one else used the tags
>> and the renderers just ignored them. Many mappers have their own
>> personal views about how something should be tagged and have no
>> interest in following any other suggestions at all. It is an issue.
>>
>> HOT is much more structured, we actually have clients with
>> requirements in mind so we map to those as best we can. Â We have
>> recommended tags and to a much larger extent people follow them. We
>> do have a lot of new mappers who may not even know about the map
>> features page or find that reading through more than two lines of
>> instructions boring. Having a two step process with validation helps
>> as well. However we still rely on locals on the ground mappers for
>> more detail and as far as I'm concerned if they want to map video
>> games, hairdresser, gymnastics, karate, volleyball or football fields
>> that's fine, they might map something else of use whilst they are
>> mapping or introduce someone else who might map something more useful
>> to us.
>>
>> What the agencies like is that we can map places very quickly which is
>> better than no maps. Also we are very cost effective, I was going to
>> say cheap but that has quality implications. They can add their own
>> specific tags without having to go through a formal standards
>> committee. Typically it takes five years to get something through the
>> ISO standards process. Currently in the background I believe HOT
>> going through a "standard's process" as we progress. Nepal has had a
>> big impact on HOT, Ebola came earlier in the spring and is still
>> around, but the scale of mapping in Nepal has caused a rethink about
>> how we do things including training, things are becoming more
>> formalised but having said that I don't think it will ever be totally
>> rigid.
>>
>> Cheerio John
>>
>> On 20 May 2015 at 11:32, althio <althio.forum at gmail.com
>> <mailto:althio.forum at gmail.com> > wrote:
>>
>>     Springfield,
>>
>>     Sorry for the partial answer and I don't mean to be harsh because I
>>     know things around here are not easy to find and understand. We all
>>     need pointers and FAQ or homepages and portals...
>>
>>     My point is... I do think that you are somehow confused between
>>     OpenStreetMap and HOT:
>>
>>     ***
>>     OSM aka OpenStreetMap, the project, its database, its goals, its
>>     community
>>     Â Â - [http <http://www.openstreetmap.org/welcome>
>>     ://www.openstreetmap.org/welcome
>>     <http://www.openstreetmap.org/welcome>] OpenStreetMap, the free
>>     and editable map of the world
>>     Â Â - [http <http://www.openstreetmap.org/about>
>>     ://www.openstreetmap.org/about
>>     <http://www.openstreetmap.org/about>] OpenStreetMap is built by a
>>     community of mappers that contribute and maintain data about roads,
>>     trails, cafés, railway stations, and much more (note: also video
>>     games, hairdresser, gymnastics, karate and volleyball...) (note2: also
>>     boundaries, hospitals, schools), all over the world.
>>
>>     see also OSM Foundation, the entity to support the project
>>     Â Â - http://wiki.osmfoundation.org/wiki/Main_Page
>>
>>     ***
>>     HOT aka Humanitarian OSM Team is using a subset of OSM database and
>>     building on it, its own goals (some overlap with OSM), its own
>>     community (some overlap with OSM)
>>     Â Â - [http <http://hotosm.org/>://hotosm.org/
>>     <http://hotosm.org/>] The Humanitarian OpenStreetMap Team [HOT]
>>     applies the principles of open source and open data sharing for
>>     humanitarian response and economic development.
>>     Â Â - [http <http://hotosm.org/about> ://hotosm.org/about
>>     <http://hotosm.org/about>]
>>     Â Â - [https://wiki.openstreetmap.org/wiki/Humanitarian_OSM_Team]
>>
>>     ***
>>
>>     Back to OSM and "tag soup" database. This is a rather hard and
>>     technical topic, and not really related to HOT and this list. You will
>>     not find the answer here, nor the most interested or skilled people.
>>     HOT uses and contributes to the database, HOT does not control it.
>>
>>     A few more words anyway? ("I am not a lawyer" and "I am not a
>>     database expert").
>>     OSM database is open, free, public, iterative and rather rich.
>>     OSM database is not fixed, not complete, not comprehensive, not
>>     homogeneous (spatially at least).
>>     The philosophy and structure have their own advantages and
>>     disadvantages compared to existing datasets.
>>     Please appreciate the uniqueness, value and potential of OSM database
>>     before you try to make it a clone of something existing.
>>
>>
>>     All the best,
>>
>>     Â - althio
>>
>>
>>     On 19 May 2015 at 21:38, Springfield Harrison
>>     <stellargps at gmail.com <mailto:stellargps at gmail.com>> wrote:
>>     > Hello Stefan & Blake,
>>     >
>>     > I concur with the comments about the "tag soup" mess. As I have
>>     mentioned
>>     > before, I am new to this OSM environment but have some years
>>     experience with
>>     > GPS and GIS mapping and database design.
>>     >
>>     > To be honest, I was appalled when I discovered that the OSM
>>     database design
>>     > looked like a glorified scratchpad. I just downloaded and
>>     inspected 366,017
>>     > OSM database records. There were 18 Key Terms and scores of
>>     values. I
>>     > extracted the unique combinations of keys/values and ended up
>>     with 388
>>     > records of those.
>>     >
>>     > It is difficult to describe the results in detail as patterns
>>     are very hard
>>     > to see with this system. Suffice it to say, there is an
>>     abundance of
>>     > overlap, redundancy, ambiguity and a confusing intermingling of
>>     features and
>>     > attributes. Using traditional methods of querying a database,
>>     it would be
>>     > impossible to definitively extract a meaningful subset of any of
>>     the 366,000
>>     > records. Generally speaking, the problem is that one feature may be
>>     > described in many different ways that are not consistent.
>>     >
>>     > Having said all that, since I frequently hear how well all this
>>     mapping
>>     > information is received in the field, I must conclude that this
>>     mishmash of
>>     > tagging somehow creates a usable end product. It may well be
>>     that I am not
>>     > aware of magic techniques that bring order to all this chaotic
>>     tagging.
>>     > However, if it works, it is good. However I do believe that it
>>     will work
>>     > better with a more robust database.
>>     >
>>     > Sorry to offer this harsh critique, but in decades of looking at
>>     database
>>     > structures for both geographical and administrative
>>     applications, I have
>>     > never seen such a jumble of terminology.
>>     >
>>     > Anyway, I have put together what I believe is a more appropriate
>>     Data
>>     > Dictionary that generally parallels the best practices in
>>     database design.
>>     > I have found this approach to be very useful, and also useful in
>>     the field,
>>     > since being introduced to it by Trimble Navigation in the early 90s.
>>     >
>>     > I am impressed with the enthusiasm that permeates the crowd GIS
>>     initiative
>>     > but concerned that the geographical and database underpinnings
>>     may be less
>>     > than ideal. My observation from creating a few software
>>     applications, is
>>     > that the lesser trained are the users, the much greater
>>     investment there
>>     > needs to be in the user interface and training. GIS and GPS
>>     data collection
>>     > is not particularly intuitive.
>>     >
>>     > My approach in projects of this kind is always to start at the
>>     far end with
>>     > the users - what information are they wanting for whatever it is
>>     that they
>>     > do? Then I look at the reporting requirements and finally
>>     design the data
>>     > collection process to feed into that.
>>     >
>>     > In the case of this emergency relief operation, I'm hard-pressed
>>     to see the
>>     > value in mapping video games, hairdresser, gymnastics, karate and
>>     > volleyball. To be fair, many of the other attributes could have
>>     value in
>>     > providing relief services but in the record set that I
>>     downloaded, there
>>     > seems to be little information related to the emergency relief
>>     effort. In
>>     > over 366,000 records there are only 19 marked as aeroway = helipad.
>>     >
>>     > I'm not sure just how thorough you intend to be with the "updating,
>>     > streamlining and regularizing" but I would be happy to help
>>     where possible.
>>     >
>>     > It would probably not be overly difficult to substitute a new
>>     > feature/attribute catalogue into the OSM database. Translating
>>     the existing
>>     > mass of keys and values to their new equivalent might be more
>>     challenging.
>>     > Databases succeed because they conform to standard pattern sets.
>>     >
>>     > Again, sorry to be less than enthusiastic but perhaps things can be
>>     > improved.
>>     >
>>     >Â Â Â Â Â Thanks for your patience, Cheers . . . . . . . . Spring
>>     Harrison
>>
>>     _______________________________________________
>>     HOT mailing list
>>     HOT at openstreetmap.org <mailto:HOT at openstreetmap.org>
>>     https://lists.openstreetmap.org/listinfo/hot
>>
>
>
> _______________________________________________
> HOT mailing list
> HOT at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot
>