[HOT] Live stats for mapathons / RedCross API / JOSM plugin(was: Mapathon stats analysis)

Tue Jun 27 07:16:36 UTC 2017

I have also tried it out and it's a good one. Am going to take it out on a
test run on the next Mapathon.

On Tue, Jun 27, 2017 at 9:19 AM, Erwin Olario <govvin at gmail.com> wrote:

> [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible
> for Automatic Cleanup! (govvin at gmail.com) Add cleanup rule
> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DG6sasPvMKhaW%252BLYmSQrH6gO95zP%252BX1q5hjW5Y42VI14%253D%26token%3DtXiqykbpCfGK67Lsld621pcdxcWVJu5dBfVGPmb6S8I1ogfAASQujlF0j3%252Ba4o9zB0w7M%252FZmokseFEcypoI%252F0fbFu%252F8Im%252BxOKAVleGuN736ZuKD%252Fcdx8aeQ2MrB05spnh4J28VSp%252BxQ%253D&tc_serial=31125916218&tc_rand=1828961458&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
> | More info
> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=31125916218&tc_rand=1828961458&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>
> This tool from CartONG looks very interesting.  I just tried it on older
> TMs we've worked in the past but it's throwing me download errors.
>
> I'll give that script a run one of these days, Bjoern.
>
> On Thu, Jun 22, 2017, 22:57 Rebecca Firth <rebecca.firth at hotosm.org>
> wrote:
>
>> Thanks so much for sharing that Violaine, really useful for mapathons
>> especially if there is an issue with the overall leaderboard - thank you to
>> the volunteer who made it too!
>>
>> On Thu, Jun 22, 2017 at 5:31 AM, Violaine Doutreleau <
>> v_doutreleau at cartong.org> wrote:
>>
>>> Hi all,
>>> Here is the tool we use, developed by one of our volunteers :
>>> http://mapathon.cartong.org/
>>>
>>> It works pretty well in area where there is no much mappers external to
>>> the TM project, directly editing in OSM, as we are trying to figure out how
>>> to best take into account changeset comments (for now it works on
>>> bbox+timestamps). Thanks Bjoern for the ref., we will have a deeper look
>>> into this !
>>>
>>> Hope this can help !
>>>
>>> Bests,
>>> Violaine for Missing Maps
>>> CartONG
>>>
>>> Le 20/06/2017 à 21:09, Bjoern Hassler a écrit :
>>>
>>> Hi Erwin, dear friends,
>>>
>>> I have some nodejs code here: https://github.com/OSM-
>>> Utilities/JOSM-Scripts-HOT/blob/master/lib/mmstats.js. It shows users
>>> contributing under a hashtag (using the osmstats api) and then shows edits
>>> by those users (buildings, roads, but also other objects), as well as
>>> common issues (e.g. isolated untagged notes, untagged ways, non-square
>>> 4-node buildings). E.g. you'll see that for some users the non-square
>>> buildings are a few % of their edits, which means they are just omissions.
>>> For other users, it's more like 50%, which means they are not squaring
>>> systematically.
>>>
>>> On a technical note: In terms of changed and diff: That works for stats,
>>> but obviously also interested in validation. What I'd really like is a
>>> query that gives me the "current state of everything that changed between
>>> these times". In the above code, you'll see some recursion, that should do
>>> that (though I have to do some more testing, and there may be some more
>>> recursion required). I was trying to keep it all within one query as well,
>>> as the queries do take time.
>>>
>>> Let me know if you have any questions or comments!
>>> Bjoern
>>>
>>> On 20 June 2017 at 07:48, Erwin Olario <govvin at gmail.com> wrote:
>>>
>>>>
>>>> Instead of using changed, I run a query [0]: using diff and filter the
>>>> edits made by only by users who are present in the mapathon:
>>>>
>>>> Caveat: there's an existing bug when querying dates with UTC offsets,
>>>> so use the actual offset time instead.
>>>>
>>>> Also, instead of filtering by "uid:", you may use "user:" for
>>>> usernames. However, some new users eventually change their names to
>>>> something else, so I prefer to use the former.
>>>>
>>>> [0]: https://gist.github.com/govvin/8cc3cfff204314536d744fe82c52e6f3
>>>>
>>>>
>>>> On Sat, Jun 17, 2017 at 7:45 PM Bjoern Hassler <bjohas+mw at gmail.com>
>>>> wrote:
>>>>
>>>>> Dear friends,
>>>>>
>>>>> I looked into https://github.com/AmericanRedCross/osm-stats a little,
>>>>> and while I can't quite get overpass results to match with what the API
>>>>> outputs, it does have very useful features.
>>>>>
>>>>> Is anybody interested in (or already working on) getting (live) stats
>>>>> for mapathons, via enhancements to osm-stats or otherwise?
>>>>>
>>>>> E.g. you could
>>>>> 1. Use osm-stats to get users making changes with specific hastags
>>>>> 2. Use overpass to load objects changed by those users since start of
>>>>> mapathon
>>>>>
>>>>> You could then run some stats (similar to the /user endpoint in
>>>>> osm-stats). These stats could also include results from running queries on
>>>>> bounding boxes, or on user-lists. This could all be done in combination
>>>>> with osm-stats and overpass, but there may be a case for integrating it
>>>>> into osm-stats, see https://github.com/AmericanRedCross/osm-stats/
>>>>> issues/45
>>>>>
>>>>> Either with the process above, or an extended osm-stats, you could
>>>>> also systematically check for common issues, e.g. untagged single nodes,
>>>>> ways with four nodes which are not squared, etc. I.e. you could
>>>>> automatically check for such common issues across the whole mapathon, with
>>>>> output like: "OSM user ABC added X untagged nodes and Y non-squared
>>>>> buildings." rather than wait for a validator to spot the issues manually /
>>>>> later.
>>>>>
>>>>> Of course, some of this this can be done via MapPaint/MapCSS-styles in
>>>>> JOSM too (e.g. single nodes), but I am not sure how you could do squared
>>>>> buildings. Also across a large area, the above could also be useful for an
>>>>> overview (and giving an indication of somebody making systematic mistake
>>>>> vs. one-off slips). The JOSM plugin could also take you to those places
>>>>> where there are issues, to inspect this (or even within tasks to fix it).
>>>>> Definitely going beyond what you can do with MapCSS or validation rule, you
>>>>> could use the JOSM plugin to visit new buildings (and whatever other
>>>>> objects) to check the tracing was done accurately.
>>>>>
>>>>> Of course, there are different approaches, that give you almost the
>>>>> same (i.e. using MapCSS or validation rules, To Do plugin etc), but a
>>>>> dedicated "MM mapathon validation" plugin would be easy to set up and could
>>>>> be made user friendly. It wouldn't replace the usual validation process,
>>>>> but would enable a "key validator" to keep an overview during a mapathon.
>>>>>
>>>>> I'd be keen to hear people's views, and whether anybody else feels
>>>>> this would be worth-while and useful.
>>>>>
>>>>> What do you think?
>>>>> Bjoern
>>>>>
>>>>>
>>>>> On 16 June 2017 at 17:07, Bjoern Hassler <bjohas+mw at gmail.com> wrote:
>>>>>
>>>>>> Hi Mike,
>>>>>>
>>>>>> Thanks for that. Yeah, at the moment I'm using overpass queries with
>>>>>> (changed:"2017-06-15T17:00:00Z","2017-06-15T21:00:00Z"); to work out
>>>>>> changes during the mapathon. You could go by bbox (and perhaps catch some
>>>>>> other MM mappers who were mapping there independently or, as you say
>>>>>> unlikely, some other mappers). Or you can go by user (no bbox) in case
>>>>>> users contributed to several projects, or by accident picked another
>>>>>> project.
>>>>>>
>>>>>> Is there a simple way of getting the bbox from the tasking manager?
>>>>>> That would be great. (Of course, it can be computed, e.g. by downloading
>>>>>> the gpx box from the NE and SW tiles... I know it's simple... but not as
>>>>>> simple as clicking a button on the tasking manager that takes you from a
>>>>>> task to the overpass query with the bbox filled in :)
>>>>>>
>>>>>> The leaderboard uses this https://github.com/AmericanRedCross, which
>>>>>> is helpful, as the Overpass API cannot query on the changeset comments, but
>>>>>> the redcross api can (and so you don't need to use the main OSM API).
>>>>>>
>>>>>> Bjoern
>>>>>>
>>>>>> On 16 June 2017 at 16:27, Mike Thompson <miketho16 at gmail.com> wrote:
>>>>>>
>>>>>>> One can use Overpass to get all of the OSM elements which were
>>>>>>> added, changed or deleted during the time of your mapathon in your AOI.
>>>>>>> These will be tagged with the user who last touched them, so you can get
>>>>>>> stats by user.  The disadvantage is that you will include edits by users
>>>>>>> who were not part of your mapathon but who happened to made edits to OSM in
>>>>>>> that AOI during the time period of your mapathon.  However, unless you are
>>>>>>> working on a crisis task, this is not likely to skew your results too much
>>>>>>> in my experience.
>>>>>>>
>>>>>>> I sometimes make a Carto map of the results, e.g.:
>>>>>>> https://tekim.carto.com/viz/baf5c3a2-2aa8-11e7-88b2-
>>>>>>> 0ef24382571b/embed_map
>>>>>>>
>>>>>>> I have more detailed instructions should someone be interested.
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 16, 2017 at 9:15 AM, Donal Hunt <donal.hunt at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> *[I don't speak for the team (I'm just a volunteer contributor to
>>>>>>>> missing maps) so someone with more context may chime in with better
>>>>>>>> info...]*
>>>>>>>>
>>>>>>>> I'm not familiar with the implementation of the leaderboard at all.
>>>>>>>> Someone from the HOTOSM web team probably knows. Or you can file a feature
>>>>>>>> request here I think: https://github.com/MissingMaps/missingmaps.
>>>>>>>> github.io/issues
>>>>>>>>
>>>>>>>> For the features specific to the tasking manager, I suspect that
>>>>>>>> filing them here
>>>>>>>> <https://github.com/hotosm/osm-tasking-manager2/issues> (
>>>>>>>> https://github.com/hotosm/osm-tasking-manager2/issues) is the
>>>>>>>> right thing to do. That way one of the regular maintainers or a volunteer
>>>>>>>> developer can see the priority of the feature being developed and
>>>>>>>> contribute to it being implemented.
>>>>>>>>
>>>>>>>> For the stats related requests, I suspect something probably exists
>>>>>>>> already within the OSM community / ecosystem that could be tweaked for your
>>>>>>>> needs. I don't have enough context right now to suggest something (maybe in
>>>>>>>> the future).
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Donal
>>>>>>>>
>>>>>>>> On Fri, Jun 16, 2017 at 3:38 PM, Bjoern Hassler <
>>>>>>>> bjohas+mw at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Dear Jan, dear Donal, dear friends,
>>>>>>>>>
>>>>>>>>> Is there an instance of https://github.com/tgrippa/
>>>>>>>>> Mapathon_HOT_OSM_WhatWeMapped running somewhere?
>>>>>>>>>
>>>>>>>>> This is great: http://www.missingmaps.org/leaderboards/#/HASHTAG,
>>>>>>>>> though there's some discrepancy between the data it shows and what I can
>>>>>>>>> see from the overpass api directly and from
>>>>>>>>> http://resultmaps.neis-one.org/. How does the leaderboards tool
>>>>>>>>> collect the information?
>>>>>>>>>
>>>>>>>>> Ideally, what I would like is:
>>>>>>>>> - all users who contributed to tasking managers under id #1234
>>>>>>>>> (from date/time-date/time), including incomplete tasks
>>>>>>>>> - all users who contributed to the bbox of project #1234  (from
>>>>>>>>> date/time-date/time)
>>>>>>>>> - all users who used hashtag in a variety of spellings (#hashtag |
>>>>>>>>> #Hashtag | etc)  (from date/time-date/time)
>>>>>>>>> (- all users who attended a mapathon.... see below)
>>>>>>>>>
>>>>>>>>> Ideally as GET parameters or similar, i.e.
>>>>>>>>> http://someservice/?project=1234|1235|1236&
>>>>>>>>> includeincompletetasks=yes&includeprojectareas=yes&
>>>>>>>>> hastag=hashtag|hashtag2&hastagcasesensitive=no&start=...&end=...
>>>>>>>>>
>>>>>>>>> which would return a list of users, with projects contributed to
>>>>>>>>> and how that fact was determined (via project id1/2/3, project area for
>>>>>>>>> project id1/2/3 or hashtag1, hashtag2, etc), plus number of contributions
>>>>>>>>> per user in different categories (nodes, ways, way[building],
>>>>>>>>> way[highway]), plus (one can but dream!) the age of their OSM account and
>>>>>>>>> total changesets.
>>>>>>>>>
>>>>>>>>> Clearly there is redundancy in the query - but that's on purpose.
>>>>>>>>> It would e.g. find people who are mapping in an area, but not working
>>>>>>>>> through the task manager.
>>>>>>>>>
>>>>>>>>> It would also be amazing if on the tasking manager, people can
>>>>>>>>> register their attendance at a mapathon. E.g. as you go to project page it
>>>>>>>>> says: "A mapathon for this project is in progress. Click here if you are
>>>>>>>>> participating in this mapathon in person or remotely."
>>>>>>>>>
>>>>>>>>> (E.g. In terms of the query, add includeattendeelist=yes
>>>>>>>>> http://someservice/?....&includeattendeelist=yes )
>>>>>>>>>
>>>>>>>>> If you had such an attendee list tool, maybe you could even put in
>>>>>>>>> your name and table number. Then we'd immediately know who is there, and
>>>>>>>>> could start looking at their edits, and support them. I sometimes wander
>>>>>>>>> round a mapathon trying to find a user who needs a bit of extra support.
>>>>>>>>>
>>>>>>>>> Having said all of this, I'd be very happy to help build such
>>>>>>>>> tools, but would prefer to do this as part of a small team!
>>>>>>>>> Bjoern
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 16 June 2017 at 14:09, Jan Martinec <jan at martinec.name> wrote:
>>>>>>>>>
>>>>>>>>>> Dne 16.6.2017 v 14:13 Bjoern Hassler napsal(a):
>>>>>>>>>>
>>>>>>>>>>> Dear friends,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What tools do we normally use to get statistics on a mapathon?
>>>>>>>>>>>
>>>>>>>>>>> Clearly the tasking manager provides contributors to the task in
>>>>>>>>>>> the 'stats'
>>>>>>>>>>> section, and also I can run overpass to look for changes made by
>>>>>>>>>>> those users.
>>>>>>>>>>>
>>>>>>>>>>> I don't think it's possible to get changesets by #hashtag? You'd
>>>>>>>>>>> have to use the
>>>>>>>>>>> main API to get all change sets for the period of the event, and
>>>>>>>>>>> then select the
>>>>>>>>>>> ones that have the right hashtag(s)?
>>>>>>>>>>>
>>>>>>>>>>> Does anybody have some tools they could point me to?
>>>>>>>>>>>
>>>>>>>>>>> (Something like this http://resultmaps.neis-one.
>>>>>>>>>>> org/osm-changesets would be
>>>>>>>>>>> great, if it listed the changesets and users...)
>>>>>>>>>>>
>>>>>>>>>>> Many thanks!
>>>>>>>>>>> Bjoern
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> HOT mailing list
>>>>>>>>>>> HOT at openstreetmap.org
>>>>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> would this be useful? I think it provides a numerical output as
>>>>>>>>>> well as pretty pictures:
>>>>>>>>>> https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Jan "Piskvor" Martinec
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> HOT mailing list
>>>>>>>>>> HOT at openstreetmap.org
>>>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> HOT mailing list
>>>>>>>> HOT at openstreetmap.org
>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> HOT mailing list
>>>>> HOT at openstreetmap.org
>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>
>>>> --
>>>>
>>>> /Erwin Olario
>>>>
>>>> e: erwin at ngnuity.xyz | v/m: https://t.me/GOwin | s:
>>>> https://mstdn.io/@GOwin
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> HOT mailing listHOT at openstreetmap.orghttps://lists.openstreetmap.org/listinfo/hot
>>>
>>>
>>> --
>>> *Violaine Doutreleau*
>>> Coordinatrice Missing Maps
>>> CartONG
>>> mobile : 06.95.02.42.44
>>> skype : doutreleau.violaine
>>>
>>>
>>> *P Help save paper - do you need to print this email?*
>>>
>>> _______________________________________________
>>> HOT mailing list
>>> HOT at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>>>
>>>
>>
>>
>> --
>> *Rebecca Firth*
>> Community Partnerships Manager
>> rebecca.firth at hotosm.org <tyler.radford at hotosm.org>
>> @RebeccaFirthy
>> Skype: rebeccafirth
>>
>> *Humanitarian OpenStreetMap Team*
>> *Using OpenStreetMap for Humanitarian Response & Economic Development*
>> web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> |
>> facebook <https://www.facebook.com/hotosm> | donate
>> <http://donate.hotosm.org/>
>>
> --
>
> /Erwin Olario
>
> e: erwin at ngnuity.xyz | v/m: https://t.me/GOwin | s:
> https://mstdn.io/@GOwin
>
> _______________________________________________
> HOT mailing list
> HOT at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot
>
>

-- 

*Humanitarian OpenStreetMap Team
*Mapping Trainer, Uganda

Email: kigguddedeogratias at gmail.com
Skype: deogratius kiggudde
Twitter: @KigguddeD
UG Mobile: +256 703 143 225

*Using OpenStreetMap for Humanitarian Response & Economic Development*
web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> |
facebook <https://www.facebook.com/hotosm> | donate
<http://hotosm.org/donate>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20170627/fb6295c2/attachment.html>