[HOT] Live stats for mapathons / RedCross API / JOSM plugin (was: Mapathon stats analysis)

Thu Jun 22 14:57:36 UTC 2017

Thanks so much for sharing that Violaine, really useful for mapathons
especially if there is an issue with the overall leaderboard - thank you to
the volunteer who made it too!

On Thu, Jun 22, 2017 at 5:31 AM, Violaine Doutreleau <
v_doutreleau at cartong.org> wrote:

> Hi all,
> Here is the tool we use, developed by one of our volunteers :
> http://mapathon.cartong.org/
>
> It works pretty well in area where there is no much mappers external to
> the TM project, directly editing in OSM, as we are trying to figure out how
> to best take into account changeset comments (for now it works on
> bbox+timestamps). Thanks Bjoern for the ref., we will have a deeper look
> into this !
>
> Hope this can help !
>
> Bests,
> Violaine for Missing Maps
> CartONG
>
> Le 20/06/2017 à 21:09, Bjoern Hassler a écrit :
>
> Hi Erwin, dear friends,
>
> I have some nodejs code here: https://github.com/OSM-U
> tilities/JOSM-Scripts-HOT/blob/master/lib/mmstats.js. It shows users
> contributing under a hashtag (using the osmstats api) and then shows edits
> by those users (buildings, roads, but also other objects), as well as
> common issues (e.g. isolated untagged notes, untagged ways, non-square
> 4-node buildings). E.g. you'll see that for some users the non-square
> buildings are a few % of their edits, which means they are just omissions.
> For other users, it's more like 50%, which means they are not squaring
> systematically.
>
> On a technical note: In terms of changed and diff: That works for stats,
> but obviously also interested in validation. What I'd really like is a
> query that gives me the "current state of everything that changed between
> these times". In the above code, you'll see some recursion, that should do
> that (though I have to do some more testing, and there may be some more
> recursion required). I was trying to keep it all within one query as well,
> as the queries do take time.
>
> Let me know if you have any questions or comments!
> Bjoern
>
> On 20 June 2017 at 07:48, Erwin Olario <govvin at gmail.com> wrote:
>
>>
>> Instead of using changed, I run a query [0]: using diff and filter the
>> edits made by only by users who are present in the mapathon:
>>
>> Caveat: there's an existing bug when querying dates with UTC offsets, so
>> use the actual offset time instead.
>>
>> Also, instead of filtering by "uid:", you may use "user:" for usernames.
>> However, some new users eventually change their names to something else, so
>> I prefer to use the former.
>>
>> [0]: https://gist.github.com/govvin/8cc3cfff204314536d744fe82c52e6f3
>>
>>
>> On Sat, Jun 17, 2017 at 7:45 PM Bjoern Hassler <bjohas+mw at gmail.com>
>> wrote:
>>
>>> Dear friends,
>>>
>>> I looked into https://github.com/AmericanRedCross/osm-stats a little,
>>> and while I can't quite get overpass results to match with what the API
>>> outputs, it does have very useful features.
>>>
>>> Is anybody interested in (or already working on) getting (live) stats
>>> for mapathons, via enhancements to osm-stats or otherwise?
>>>
>>> E.g. you could
>>> 1. Use osm-stats to get users making changes with specific hastags
>>> 2. Use overpass to load objects changed by those users since start of
>>> mapathon
>>>
>>> You could then run some stats (similar to the /user endpoint in
>>> osm-stats). These stats could also include results from running queries on
>>> bounding boxes, or on user-lists. This could all be done in combination
>>> with osm-stats and overpass, but there may be a case for integrating it
>>> into osm-stats, see https://github.com/AmericanRed
>>> Cross/osm-stats/issues/45
>>>
>>> Either with the process above, or an extended osm-stats, you could also
>>> systematically check for common issues, e.g. untagged single nodes, ways
>>> with four nodes which are not squared, etc. I.e. you could automatically
>>> check for such common issues across the whole mapathon, with output like:
>>> "OSM user ABC added X untagged nodes and Y non-squared buildings." rather
>>> than wait for a validator to spot the issues manually / later.
>>>
>>> Of course, some of this this can be done via MapPaint/MapCSS-styles in
>>> JOSM too (e.g. single nodes), but I am not sure how you could do squared
>>> buildings. Also across a large area, the above could also be useful for an
>>> overview (and giving an indication of somebody making systematic mistake
>>> vs. one-off slips). The JOSM plugin could also take you to those places
>>> where there are issues, to inspect this (or even within tasks to fix it).
>>> Definitely going beyond what you can do with MapCSS or validation rule, you
>>> could use the JOSM plugin to visit new buildings (and whatever other
>>> objects) to check the tracing was done accurately.
>>>
>>> Of course, there are different approaches, that give you almost the same
>>> (i.e. using MapCSS or validation rules, To Do plugin etc), but a dedicated
>>> "MM mapathon validation" plugin would be easy to set up and could be made
>>> user friendly. It wouldn't replace the usual validation process, but would
>>> enable a "key validator" to keep an overview during a mapathon.
>>>
>>> I'd be keen to hear people's views, and whether anybody else feels this
>>> would be worth-while and useful.
>>>
>>> What do you think?
>>> Bjoern
>>>
>>>
>>> On 16 June 2017 at 17:07, Bjoern Hassler <bjohas+mw at gmail.com> wrote:
>>>
>>>> Hi Mike,
>>>>
>>>> Thanks for that. Yeah, at the moment I'm using overpass queries with
>>>> (changed:"2017-06-15T17:00:00Z","2017-06-15T21:00:00Z"); to work out
>>>> changes during the mapathon. You could go by bbox (and perhaps catch some
>>>> other MM mappers who were mapping there independently or, as you say
>>>> unlikely, some other mappers). Or you can go by user (no bbox) in case
>>>> users contributed to several projects, or by accident picked another
>>>> project.
>>>>
>>>> Is there a simple way of getting the bbox from the tasking manager?
>>>> That would be great. (Of course, it can be computed, e.g. by downloading
>>>> the gpx box from the NE and SW tiles... I know it's simple... but not as
>>>> simple as clicking a button on the tasking manager that takes you from a
>>>> task to the overpass query with the bbox filled in :)
>>>>
>>>> The leaderboard uses this https://github.com/AmericanRedCross, which
>>>> is helpful, as the Overpass API cannot query on the changeset comments, but
>>>> the redcross api can (and so you don't need to use the main OSM API).
>>>>
>>>> Bjoern
>>>>
>>>> On 16 June 2017 at 16:27, Mike Thompson <miketho16 at gmail.com> wrote:
>>>>
>>>>> One can use Overpass to get all of the OSM elements which were added,
>>>>> changed or deleted during the time of your mapathon in your AOI.  These
>>>>> will be tagged with the user who last touched them, so you can get stats by
>>>>> user.  The disadvantage is that you will include edits by users who were
>>>>> not part of your mapathon but who happened to made edits to OSM in that AOI
>>>>> during the time period of your mapathon.  However, unless you are working
>>>>> on a crisis task, this is not likely to skew your results too much in my
>>>>> experience.
>>>>>
>>>>> I sometimes make a Carto map of the results, e.g.:
>>>>> https://tekim.carto.com/viz/baf5c3a2-2aa8-11e7-88b2-0ef24382
>>>>> 571b/embed_map
>>>>>
>>>>> I have more detailed instructions should someone be interested.
>>>>>
>>>>> Mike
>>>>>
>>>>>
>>>>> On Fri, Jun 16, 2017 at 9:15 AM, Donal Hunt <donal.hunt at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> *[I don't speak for the team (I'm just a volunteer contributor to
>>>>>> missing maps) so someone with more context may chime in with better
>>>>>> info...]*
>>>>>>
>>>>>> I'm not familiar with the implementation of the leaderboard at all.
>>>>>> Someone from the HOTOSM web team probably knows. Or you can file a feature
>>>>>> request here I think: https://github.com/Miss
>>>>>> ingMaps/missingmaps.github.io/issues
>>>>>>
>>>>>> For the features specific to the tasking manager, I suspect that
>>>>>> filing them here
>>>>>> <https://github.com/hotosm/osm-tasking-manager2/issues> (
>>>>>> https://github.com/hotosm/osm-tasking-manager2/issues) is the right
>>>>>> thing to do. That way one of the regular maintainers or a volunteer
>>>>>> developer can see the priority of the feature being developed and
>>>>>> contribute to it being implemented.
>>>>>>
>>>>>> For the stats related requests, I suspect something probably exists
>>>>>> already within the OSM community / ecosystem that could be tweaked for your
>>>>>> needs. I don't have enough context right now to suggest something (maybe in
>>>>>> the future).
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Donal
>>>>>>
>>>>>> On Fri, Jun 16, 2017 at 3:38 PM, Bjoern Hassler <bjohas+mw at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Dear Jan, dear Donal, dear friends,
>>>>>>>
>>>>>>> Is there an instance of https://github.com/tgrippa/
>>>>>>> Mapathon_HOT_OSM_WhatWeMapped running somewhere?
>>>>>>>
>>>>>>> This is great: http://www.missingmaps.org/leaderboards/#/HASHTAG,
>>>>>>> though there's some discrepancy between the data it shows and what I can
>>>>>>> see from the overpass api directly and from
>>>>>>> http://resultmaps.neis-one.org/. How does the leaderboards tool
>>>>>>> collect the information?
>>>>>>>
>>>>>>> Ideally, what I would like is:
>>>>>>> - all users who contributed to tasking managers under id #1234 (from
>>>>>>> date/time-date/time), including incomplete tasks
>>>>>>> - all users who contributed to the bbox of project #1234  (from
>>>>>>> date/time-date/time)
>>>>>>> - all users who used hashtag in a variety of spellings (#hashtag |
>>>>>>> #Hashtag | etc)  (from date/time-date/time)
>>>>>>> (- all users who attended a mapathon.... see below)
>>>>>>>
>>>>>>> Ideally as GET parameters or similar, i.e.
>>>>>>> http://someservice/?project=1234|1235|1236&includeincomplete
>>>>>>> tasks=yes&includeprojectareas=yes&hastag=hashtag|hashtag2&ha
>>>>>>> stagcasesensitive=no&start=...&end=...
>>>>>>>
>>>>>>> which would return a list of users, with projects contributed to and
>>>>>>> how that fact was determined (via project id1/2/3, project area for project
>>>>>>> id1/2/3 or hashtag1, hashtag2, etc), plus number of contributions per user
>>>>>>> in different categories (nodes, ways, way[building], way[highway]), plus
>>>>>>> (one can but dream!) the age of their OSM account and total changesets.
>>>>>>>
>>>>>>> Clearly there is redundancy in the query - but that's on purpose. It
>>>>>>> would e.g. find people who are mapping in an area, but not working through
>>>>>>> the task manager.
>>>>>>>
>>>>>>> It would also be amazing if on the tasking manager, people can
>>>>>>> register their attendance at a mapathon. E.g. as you go to project page it
>>>>>>> says: "A mapathon for this project is in progress. Click here if you are
>>>>>>> participating in this mapathon in person or remotely."
>>>>>>>
>>>>>>> (E.g. In terms of the query, add includeattendeelist=yes
>>>>>>> http://someservice/?....&includeattendeelist=yes )
>>>>>>>
>>>>>>> If you had such an attendee list tool, maybe you could even put in
>>>>>>> your name and table number. Then we'd immediately know who is there, and
>>>>>>> could start looking at their edits, and support them. I sometimes wander
>>>>>>> round a mapathon trying to find a user who needs a bit of extra support.
>>>>>>>
>>>>>>> Having said all of this, I'd be very happy to help build such tools,
>>>>>>> but would prefer to do this as part of a small team!
>>>>>>> Bjoern
>>>>>>>
>>>>>>>
>>>>>>> On 16 June 2017 at 14:09, Jan Martinec <jan at martinec.name> wrote:
>>>>>>>
>>>>>>>> Dne 16.6.2017 v 14:13 Bjoern Hassler napsal(a):
>>>>>>>>
>>>>>>>>> Dear friends,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What tools do we normally use to get statistics on a mapathon?
>>>>>>>>>
>>>>>>>>> Clearly the tasking manager provides contributors to the task in
>>>>>>>>> the 'stats'
>>>>>>>>> section, and also I can run overpass to look for changes made by
>>>>>>>>> those users.
>>>>>>>>>
>>>>>>>>> I don't think it's possible to get changesets by #hashtag? You'd
>>>>>>>>> have to use the
>>>>>>>>> main API to get all change sets for the period of the event, and
>>>>>>>>> then select the
>>>>>>>>> ones that have the right hashtag(s)?
>>>>>>>>>
>>>>>>>>> Does anybody have some tools they could point me to?
>>>>>>>>>
>>>>>>>>> (Something like this http://resultmaps.neis-one.org/osm-changesets
>>>>>>>>> would be
>>>>>>>>> great, if it listed the changesets and users...)
>>>>>>>>>
>>>>>>>>> Many thanks!
>>>>>>>>> Bjoern
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> HOT mailing list
>>>>>>>>> HOT at openstreetmap.org
>>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> would this be useful? I think it provides a numerical output as
>>>>>>>> well as pretty pictures:
>>>>>>>> https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Jan "Piskvor" Martinec
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> HOT mailing list
>>>>>>>> HOT at openstreetmap.org
>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> HOT mailing list
>>>>>> HOT at openstreetmap.org
>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>
>>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> HOT mailing list
>>> HOT at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>>>
>> --
>>
>> /Erwin Olario
>>
>> e: erwin at ngnuity.xyz | v/m: https://t.me/GOwin | s:
>> https://mstdn.io/@GOwin
>>
>
>
>
> _______________________________________________
> HOT mailing listHOT at openstreetmap.orghttps://lists.openstreetmap.org/listinfo/hot
>
>
> --
> *Violaine Doutreleau*
> Coordinatrice Missing Maps
> CartONG
> mobile : 06.95.02.42.44
> skype : doutreleau.violaine
>
>
> *P Help save paper - do you need to print this email?*
>
> _______________________________________________
> HOT mailing list
> HOT at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot
>
>

-- 
*Rebecca Firth*
Community Partnerships Manager
rebecca.firth at hotosm.org <tyler.radford at hotosm.org>
@RebeccaFirthy
Skype: rebeccafirth

*Humanitarian OpenStreetMap Team*
*Using OpenStreetMap for Humanitarian Response & Economic Development*
web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> | facebook
<https://www.facebook.com/hotosm> | donate <http://donate.hotosm.org/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20170622/50158d4f/attachment.html>