[HOT] Live stats for mapathons / RedCross API / JOSM plugin (was: Mapathon stats analysis)

Violaine Doutreleau v_doutreleau at cartong.org
Thu Jun 22 10:31:30 UTC 2017


Hi all,

Here is the tool we use, developed by one of our volunteers : 
http://mapathon.cartong.org/

It works pretty well in area where there is no much mappers external to 
the TM project, directly editing in OSM, as we are trying to figure out 
how to best take into account changeset comments (for now it works on 
bbox+timestamps). Thanks Bjoern for the ref., we will have a deeper look 
into this !

Hope this can help !

Bests,
Violaine for Missing Maps
CartONG
Le 20/06/2017 à 21:09, Bjoern Hassler a écrit :
> Hi Erwin, dear friends,
>
> I have some nodejs code here: 
> https://github.com/OSM-Utilities/JOSM-Scripts-HOT/blob/master/lib/mmstats.js 
> <https://github.com/OSM-Utilities/JOSM-Scripts-HOT/blob/master/lib/mmstats.js>. 
> It shows users contributing under a hashtag (using the osmstats api) 
> and then shows edits by those users (buildings, roads, but also other 
> objects), as well as common issues (e.g. isolated untagged notes, 
> untagged ways, non-square 4-node buildings). E.g. you'll see that for 
> some users the non-square buildings are a few % of their edits, which 
> means they are just omissions. For other users, it's more like 50%, 
> which means they are not squaring systematically.
>
> On a technical note: In terms of changed and diff: That works for 
> stats, but obviously also interested in validation. What I'd really 
> like is a query that gives me the "current state of everything that 
> changed between these times". In the above code, you'll see some 
> recursion, that should do that (though I have to do some more testing, 
> and there may be some more recursion required). I was trying to keep 
> it all within one query as well, as the queries do take time.
>
> Let me know if you have any questions or comments!
> Bjoern
>
> On 20 June 2017 at 07:48, Erwin Olario <govvin at gmail.com 
> <mailto:govvin at gmail.com>> wrote:
>
>
>     Instead of using changed, I run a query [0]: using diff and filter
>     the edits made by only by users who are present in the mapathon:
>
>     Caveat: there's an existing bug when querying dates with UTC
>     offsets, so use the actual offset time instead.
>
>     Also, instead of filtering by "uid:", you may use "user:" for
>     usernames. However, some new users eventually change their names
>     to something else, so I prefer to use the former.
>
>     [0]:
>     https://gist.github.com/govvin/8cc3cfff204314536d744fe82c52e6f3
>     <https://gist.github.com/govvin/8cc3cfff204314536d744fe82c52e6f3>
>
>
>     On Sat, Jun 17, 2017 at 7:45 PM Bjoern Hassler
>     <bjohas+mw at gmail.com <mailto:bjohas%2Bmw at gmail.com>> wrote:
>
>         Dear friends,
>
>         I looked into https://github.com/AmericanRedCross/osm-stats
>         <https://github.com/AmericanRedCross/osm-stats> a little, and
>         while I can't quite get overpass results to match with what
>         the API outputs, it does have very useful features.
>
>         Is anybody interested in (or already working on) getting
>         (live) stats for mapathons, via enhancements to osm-stats or
>         otherwise?
>
>         E.g. you could
>         1. Use osm-stats to get users making changes with specific hastags
>         2. Use overpass to load objects changed by those users since
>         start of mapathon
>
>         You could then run some stats (similar to the /user endpoint
>         in osm-stats). These stats could also include results from
>         running queries on bounding boxes, or on user-lists. This
>         could all be done in combination with osm-stats and overpass,
>         but there may be a case for integrating it into osm-stats, see
>         https://github.com/AmericanRedCross/osm-stats/issues/45
>         <https://github.com/AmericanRedCross/osm-stats/issues/45>
>
>         Either with the process above, or an extended osm-stats, you
>         could also systematically check for common issues, e.g.
>         untagged single nodes, ways with four nodes which are not
>         squared, etc. I.e. you could automatically check for such
>         common issues across the whole mapathon, with output like:
>         "OSM user ABC added X untagged nodes and Y non-squared
>         buildings." rather than wait for a validator to spot the
>         issues manually / later.
>
>         Of course, some of this this can be done via
>         MapPaint/MapCSS-styles in JOSM too (e.g. single nodes), but I
>         am not sure how you could do squared buildings. Also across a
>         large area, the above could also be useful for an overview
>         (and giving an indication of somebody making systematic
>         mistake vs. one-off slips). The JOSM plugin could also take
>         you to those places where there are issues, to inspect this
>         (or even within tasks to fix it). Definitely going beyond what
>         you can do with MapCSS or validation rule, you could use the
>         JOSM plugin to visit new buildings (and whatever other
>         objects) to check the tracing was done accurately.
>
>         Of course, there are different approaches, that give you
>         almost the same (i.e. using MapCSS or validation rules, To Do
>         plugin etc), but a dedicated "MM mapathon validation" plugin
>         would be easy to set up and could be made user friendly. It
>         wouldn't replace the usual validation process, but would
>         enable a "key validator" to keep an overview during a mapathon.
>
>         I'd be keen to hear people's views, and whether anybody else
>         feels this would be worth-while and useful.
>
>         What do you think?
>         Bjoern
>
>
>         On 16 June 2017 at 17:07, Bjoern Hassler <bjohas+mw at gmail.com
>         <mailto:bjohas+mw at gmail.com>> wrote:
>
>             Hi Mike,
>
>             Thanks for that. Yeah, at the moment I'm using overpass
>             queries with
>             (changed:"2017-06-15T17:00:00Z","2017-06-15T21:00:00Z");
>             to work out changes during the mapathon. You could go by
>             bbox (and perhaps catch some other MM mappers who were
>             mapping there independently or, as you say unlikely, some
>             other mappers). Or you can go by user (no bbox) in case
>             users contributed to several projects, or by accident
>             picked another project.
>
>             Is there a simple way of getting the bbox from the tasking
>             manager? That would be great. (Of course, it can be
>             computed, e.g. by downloading the gpx box from the NE and
>             SW tiles... I know it's simple... but not as simple as
>             clicking a button on the tasking manager that takes you
>             from a task to the overpass query with the bbox filled in :)
>
>             The leaderboard uses this
>             https://github.com/AmericanRedCross
>             <https://github.com/AmericanRedCross>, which is helpful,
>             as the Overpass API cannot query on the changeset
>             comments, but the redcross api can (and so you don't need
>             to use the main OSM API).
>
>             Bjoern
>
>             On 16 June 2017 at 16:27, Mike Thompson
>             <miketho16 at gmail.com <mailto:miketho16 at gmail.com>> wrote:
>
>                 One can use Overpass to get all of the OSM elements
>                 which were added, changed or deleted during the time
>                 of your mapathon in your AOI.  These will be tagged
>                 with the user who last touched them, so you can get
>                 stats by user. The disadvantage is that you will
>                 include edits by users who were not part of your
>                 mapathon but who happened to made edits to OSM in that
>                 AOI during the time period of your mapathon.  However,
>                 unless you are working on a crisis task, this is not
>                 likely to skew your results too much in my experience.
>
>                 I sometimes make a Carto map of the results, e.g.:
>                 https://tekim.carto.com/viz/baf5c3a2-2aa8-11e7-88b2-0ef24382571b/embed_map
>                 <https://tekim.carto.com/viz/baf5c3a2-2aa8-11e7-88b2-0ef24382571b/embed_map>
>
>                 I have more detailed instructions should someone be
>                 interested.
>
>                 Mike
>
>
>                 On Fri, Jun 16, 2017 at 9:15 AM, Donal Hunt
>                 <donal.hunt at gmail.com <mailto:donal.hunt at gmail.com>>
>                 wrote:
>
>                     /[I don't speak for the team (I'm just a volunteer
>                     contributor to missing maps) so someone with more
>                     context may chime in with better info...]/
>
>                     I'm not familiar with the implementation of the
>                     leaderboard at all. Someone from the HOTOSM web
>                     team probably knows. Or you can file a feature
>                     request here I think:
>                     https://github.com/MissingMaps/missingmaps.github.io/issues
>                     <https://github.com/MissingMaps/missingmaps.github.io/issues>
>
>                     For the features specific to the tasking manager,
>                     I suspect that filing them here
>                     <https://github.com/hotosm/osm-tasking-manager2/issues> (https://github.com/hotosm/osm-tasking-manager2/issues
>                     <https://github.com/hotosm/osm-tasking-manager2/issues>)
>                     is the right thing to do. That way one of the
>                     regular maintainers or a volunteer developer can
>                     see the priority of the feature being developed
>                     and contribute to it being implemented.
>
>                     For the stats related requests, I suspect
>                     something probably exists already within the OSM
>                     community / ecosystem that could be tweaked for
>                     your needs. I don't have enough context right now
>                     to suggest something (maybe in the future).
>
>                     Regards
>
>                     Donal
>
>                     On Fri, Jun 16, 2017 at 3:38 PM, Bjoern Hassler
>                     <bjohas+mw at gmail.com <mailto:bjohas+mw at gmail.com>>
>                     wrote:
>
>                         Dear Jan, dear Donal, dear friends,
>
>                         Is there an instance of
>                         https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped
>                         <https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped>
>                         running somewhere?
>
>                         This is great:
>                         http://www.missingmaps.org/leaderboards/#/HASHTAG
>                         <http://www.missingmaps.org/leaderboards/#/HASHTAG>,
>                         though there's some discrepancy between the
>                         data it shows and what I can see from the
>                         overpass api directly and from
>                         http://resultmaps.neis-one.org/
>                         <http://resultmaps.neis-one.org/>. How does
>                         the leaderboards tool collect the information?
>
>                         Ideally, what I would like is:
>                         - all users who contributed to tasking
>                         managers under id #1234 (from
>                         date/time-date/time), including incomplete tasks
>                         - all users who contributed to the bbox of
>                         project #1234  (from date/time-date/time)
>                         - all users who used hashtag in a variety of
>                         spellings (#hashtag | #Hashtag | etc)  (from
>                         date/time-date/time)
>                         (- all users who attended a mapathon.... see
>                         below)
>
>                         Ideally as GET parameters or similar, i.e.
>                         http://someservice/?project=1234|1235|1236&includeincompletetasks=yes&includeprojectareas=yes&hastag=hashtag|hashtag2&hastagcasesensitive=no&start=...&end=.
>                         <http://someservice/?project=1234%7C1235%7C1236&includeincompletetasks=yes&includeprojectareas=yes&hastag=hashtag%7Chashtag2&hastagcasesensitive=no&start=...&end=.>..
>
>                         which would return a list of users, with
>                         projects contributed to and how that fact was
>                         determined (via project id1/2/3, project area
>                         for project id1/2/3 or hashtag1, hashtag2,
>                         etc), plus number of contributions per user in
>                         different categories (nodes, ways,
>                         way[building], way[highway]), plus (one can
>                         but dream!) the age of their OSM account and
>                         total changesets.
>
>                         Clearly there is redundancy in the query - but
>                         that's on purpose. It would e.g. find people
>                         who are mapping in an area, but not working
>                         through the task manager.
>
>                         It would also be amazing if on the tasking
>                         manager, people can register their attendance
>                         at a mapathon. E.g. as you go to project page
>                         it says: "A mapathon for this project is in
>                         progress. Click here if you are participating
>                         in this mapathon in person or remotely."
>
>                         (E.g. In terms of the query, add
>                         includeattendeelist=yes
>                         http://someservice/?....&includeattendeelist=yes
>                         <http://someservice/?....&includeattendeelist=yes>
>                         )
>
>                         If you had such an attendee list tool, maybe
>                         you could even put in your name and table
>                         number. Then we'd immediately know who is
>                         there, and could start looking at their edits,
>                         and support them. I sometimes wander round a
>                         mapathon trying to find a user who needs a bit
>                         of extra support.
>
>                         Having said all of this, I'd be very happy to
>                         help build such tools, but would prefer to do
>                         this as part of a small team!
>                         Bjoern
>
>
>                         On 16 June 2017 at 14:09, Jan Martinec
>                         <jan at martinec.name <mailto:jan at martinec.name>>
>                         wrote:
>
>                             Dne 16.6.2017 v 14:13 Bjoern Hassler
>                             napsal(a):
>
>                                 Dear friends,
>
>
>                                 What tools do we normally use to get
>                                 statistics on a mapathon?
>
>                                 Clearly the tasking manager provides
>                                 contributors to the task in the 'stats'
>                                 section, and also I can run overpass
>                                 to look for changes made by those users.
>
>                                 I don't think it's possible to get
>                                 changesets by #hashtag? You'd have to
>                                 use the
>                                 main API to get all change sets for
>                                 the period of the event, and then
>                                 select the
>                                 ones that have the right hashtag(s)?
>
>                                 Does anybody have some tools they
>                                 could point me to?
>
>                                 (Something like this
>                                 http://resultmaps.neis-one.org/osm-changesets
>                                 <http://resultmaps.neis-one.org/osm-changesets>
>                                 would be
>                                 great, if it listed the changesets and
>                                 users...)
>
>                                 Many thanks!
>                                 Bjoern
>
>
>
>                                 _______________________________________________
>                                 HOT mailing list
>                                 HOT at openstreetmap.org
>                                 <mailto:HOT at openstreetmap.org>
>                                 https://lists.openstreetmap.org/listinfo/hot
>                                 <https://lists.openstreetmap.org/listinfo/hot>
>
>
>                             Hello,
>
>                             would this be useful? I think it provides
>                             a numerical output as well as pretty pictures:
>                             https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped
>                             <https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped>
>
>                             Cheers,
>
>                             Jan "Piskvor" Martinec
>
>                             _______________________________________________
>                             HOT mailing list
>                             HOT at openstreetmap.org
>                             <mailto:HOT at openstreetmap.org>
>                             https://lists.openstreetmap.org/listinfo/hot
>                             <https://lists.openstreetmap.org/listinfo/hot>
>
>
>
>
>                     _______________________________________________
>                     HOT mailing list
>                     HOT at openstreetmap.org <mailto:HOT at openstreetmap.org>
>                     https://lists.openstreetmap.org/listinfo/hot
>                     <https://lists.openstreetmap.org/listinfo/hot>
>
>
>
>
>         _______________________________________________
>         HOT mailing list
>         HOT at openstreetmap.org <mailto:HOT at openstreetmap.org>
>         https://lists.openstreetmap.org/listinfo/hot
>         <https://lists.openstreetmap.org/listinfo/hot>
>
>     -- 
>
>     /Erwin Olario
>
>     e: erwin at ngnuity.xyz <mailto:erwin at ngnuity.xyz> | v/m:
>     https://t.me/GOwin | s: https://mstdn.io/@GOwin
>
>
>
>
> _______________________________________________
> HOT mailing list
> HOT at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot

-- 
Fwd: Re: Disponibilité de votre lieu pour un mapathon Missing Maps
*Violaine Doutreleau*
Coordinatrice Missing Maps
CartONG 
<file:///%5C%5Cmedia%5Cviolaine%5CData%5C0.CartONG%5Cwww.cartong.org>
mobile : 06.95.02.42.44
skype : doutreleau.violaine


_P Help save paper - do you need to print this email?_
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20170622/2bfeb7ab/attachment.html>


More information about the HOT mailing list