[HOT] Live stats for mapathons / RedCross API / JOSM plugin (was: Mapathon stats analysis)

Tue Jun 27 06:19:15 UTC 2017

This tool from CartONG looks very interesting.  I just tried it on older
TMs we've worked in the past but it's throwing me download errors.

I'll give that script a run one of these days, Bjoern.

On Thu, Jun 22, 2017, 22:57 Rebecca Firth <rebecca.firth at hotosm.org> wrote:

> Thanks so much for sharing that Violaine, really useful for mapathons
> especially if there is an issue with the overall leaderboard - thank you to
> the volunteer who made it too!
>
> On Thu, Jun 22, 2017 at 5:31 AM, Violaine Doutreleau <
> v_doutreleau at cartong.org> wrote:
>
>> Hi all,
>> Here is the tool we use, developed by one of our volunteers :
>> http://mapathon.cartong.org/
>>
>> It works pretty well in area where there is no much mappers external to
>> the TM project, directly editing in OSM, as we are trying to figure out how
>> to best take into account changeset comments (for now it works on
>> bbox+timestamps). Thanks Bjoern for the ref., we will have a deeper look
>> into this !
>>
>> Hope this can help !
>>
>> Bests,
>> Violaine for Missing Maps
>> CartONG
>>
>> Le 20/06/2017 à 21:09, Bjoern Hassler a écrit :
>>
>> Hi Erwin, dear friends,
>>
>> I have some nodejs code here:
>> https://github.com/OSM-Utilities/JOSM-Scripts-HOT/blob/master/lib/mmstats.js.
>> It shows users contributing under a hashtag (using the osmstats api) and
>> then shows edits by those users (buildings, roads, but also other objects),
>> as well as common issues (e.g. isolated untagged notes, untagged ways,
>> non-square 4-node buildings). E.g. you'll see that for some users the
>> non-square buildings are a few % of their edits, which means they are just
>> omissions. For other users, it's more like 50%, which means they are not
>> squaring systematically.
>>
>> On a technical note: In terms of changed and diff: That works for stats,
>> but obviously also interested in validation. What I'd really like is a
>> query that gives me the "current state of everything that changed between
>> these times". In the above code, you'll see some recursion, that should do
>> that (though I have to do some more testing, and there may be some more
>> recursion required). I was trying to keep it all within one query as well,
>> as the queries do take time.
>>
>> Let me know if you have any questions or comments!
>> Bjoern
>>
>> On 20 June 2017 at 07:48, Erwin Olario <govvin at gmail.com> wrote:
>>
>>>
>>> Instead of using changed, I run a query [0]: using diff and filter the
>>> edits made by only by users who are present in the mapathon:
>>>
>>> Caveat: there's an existing bug when querying dates with UTC offsets, so
>>> use the actual offset time instead.
>>>
>>> Also, instead of filtering by "uid:", you may use "user:" for usernames.
>>> However, some new users eventually change their names to something else, so
>>> I prefer to use the former.
>>>
>>> [0]: https://gist.github.com/govvin/8cc3cfff204314536d744fe82c52e6f3
>>>
>>>
>>> On Sat, Jun 17, 2017 at 7:45 PM Bjoern Hassler <bjohas+mw at gmail.com>
>>> wrote:
>>>
>>>> Dear friends,
>>>>
>>>> I looked into https://github.com/AmericanRedCross/osm-stats a little,
>>>> and while I can't quite get overpass results to match with what the API
>>>> outputs, it does have very useful features.
>>>>
>>>> Is anybody interested in (or already working on) getting (live) stats
>>>> for mapathons, via enhancements to osm-stats or otherwise?
>>>>
>>>> E.g. you could
>>>> 1. Use osm-stats to get users making changes with specific hastags
>>>> 2. Use overpass to load objects changed by those users since start of
>>>> mapathon
>>>>
>>>> You could then run some stats (similar to the /user endpoint in
>>>> osm-stats). These stats could also include results from running queries on
>>>> bounding boxes, or on user-lists. This could all be done in combination
>>>> with osm-stats and overpass, but there may be a case for integrating it
>>>> into osm-stats, see
>>>> https://github.com/AmericanRedCross/osm-stats/issues/45
>>>>
>>>> Either with the process above, or an extended osm-stats, you could also
>>>> systematically check for common issues, e.g. untagged single nodes, ways
>>>> with four nodes which are not squared, etc. I.e. you could automatically
>>>> check for such common issues across the whole mapathon, with output like:
>>>> "OSM user ABC added X untagged nodes and Y non-squared buildings." rather
>>>> than wait for a validator to spot the issues manually / later.
>>>>
>>>> Of course, some of this this can be done via MapPaint/MapCSS-styles in
>>>> JOSM too (e.g. single nodes), but I am not sure how you could do squared
>>>> buildings. Also across a large area, the above could also be useful for an
>>>> overview (and giving an indication of somebody making systematic mistake
>>>> vs. one-off slips). The JOSM plugin could also take you to those places
>>>> where there are issues, to inspect this (or even within tasks to fix it).
>>>> Definitely going beyond what you can do with MapCSS or validation rule, you
>>>> could use the JOSM plugin to visit new buildings (and whatever other
>>>> objects) to check the tracing was done accurately.
>>>>
>>>> Of course, there are different approaches, that give you almost the
>>>> same (i.e. using MapCSS or validation rules, To Do plugin etc), but a
>>>> dedicated "MM mapathon validation" plugin would be easy to set up and could
>>>> be made user friendly. It wouldn't replace the usual validation process,
>>>> but would enable a "key validator" to keep an overview during a mapathon.
>>>>
>>>> I'd be keen to hear people's views, and whether anybody else feels this
>>>> would be worth-while and useful.
>>>>
>>>> What do you think?
>>>> Bjoern
>>>>
>>>>
>>>> On 16 June 2017 at 17:07, Bjoern Hassler <bjohas+mw at gmail.com> wrote:
>>>>
>>>>> Hi Mike,
>>>>>
>>>>> Thanks for that. Yeah, at the moment I'm using overpass queries with
>>>>> (changed:"2017-06-15T17:00:00Z","2017-06-15T21:00:00Z"); to work out
>>>>> changes during the mapathon. You could go by bbox (and perhaps catch some
>>>>> other MM mappers who were mapping there independently or, as you say
>>>>> unlikely, some other mappers). Or you can go by user (no bbox) in case
>>>>> users contributed to several projects, or by accident picked another
>>>>> project.
>>>>>
>>>>> Is there a simple way of getting the bbox from the tasking manager?
>>>>> That would be great. (Of course, it can be computed, e.g. by downloading
>>>>> the gpx box from the NE and SW tiles... I know it's simple... but not as
>>>>> simple as clicking a button on the tasking manager that takes you from a
>>>>> task to the overpass query with the bbox filled in :)
>>>>>
>>>>> The leaderboard uses this https://github.com/AmericanRedCross, which
>>>>> is helpful, as the Overpass API cannot query on the changeset comments, but
>>>>> the redcross api can (and so you don't need to use the main OSM API).
>>>>>
>>>>> Bjoern
>>>>>
>>>>> On 16 June 2017 at 16:27, Mike Thompson <miketho16 at gmail.com> wrote:
>>>>>
>>>>>> One can use Overpass to get all of the OSM elements which were added,
>>>>>> changed or deleted during the time of your mapathon in your AOI.  These
>>>>>> will be tagged with the user who last touched them, so you can get stats by
>>>>>> user.  The disadvantage is that you will include edits by users who were
>>>>>> not part of your mapathon but who happened to made edits to OSM in that AOI
>>>>>> during the time period of your mapathon.  However, unless you are working
>>>>>> on a crisis task, this is not likely to skew your results too much in my
>>>>>> experience.
>>>>>>
>>>>>> I sometimes make a Carto map of the results, e.g.:
>>>>>>
>>>>>> https://tekim.carto.com/viz/baf5c3a2-2aa8-11e7-88b2-0ef24382571b/embed_map
>>>>>>
>>>>>> I have more detailed instructions should someone be interested.
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 16, 2017 at 9:15 AM, Donal Hunt <donal.hunt at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> *[I don't speak for the team (I'm just a volunteer contributor to
>>>>>>> missing maps) so someone with more context may chime in with better
>>>>>>> info...]*
>>>>>>>
>>>>>>> I'm not familiar with the implementation of the leaderboard at all.
>>>>>>> Someone from the HOTOSM web team probably knows. Or you can file a feature
>>>>>>> request here I think:
>>>>>>> https://github.com/MissingMaps/missingmaps.github.io/issues
>>>>>>>
>>>>>>> For the features specific to the tasking manager, I suspect that
>>>>>>> filing them here
>>>>>>> <https://github.com/hotosm/osm-tasking-manager2/issues> (
>>>>>>> https://github.com/hotosm/osm-tasking-manager2/issues) is the right
>>>>>>> thing to do. That way one of the regular maintainers or a volunteer
>>>>>>> developer can see the priority of the feature being developed and
>>>>>>> contribute to it being implemented.
>>>>>>>
>>>>>>> For the stats related requests, I suspect something probably exists
>>>>>>> already within the OSM community / ecosystem that could be tweaked for your
>>>>>>> needs. I don't have enough context right now to suggest something (maybe in
>>>>>>> the future).
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Donal
>>>>>>>
>>>>>>> On Fri, Jun 16, 2017 at 3:38 PM, Bjoern Hassler <bjohas+mw at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Dear Jan, dear Donal, dear friends,
>>>>>>>>
>>>>>>>> Is there an instance of
>>>>>>>> https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped running
>>>>>>>> somewhere?
>>>>>>>>
>>>>>>>> This is great: http://www.missingmaps.org/leaderboards/#/HASHTAG,
>>>>>>>> though there's some discrepancy between the data it shows and what I can
>>>>>>>> see from the overpass api directly and from
>>>>>>>> http://resultmaps.neis-one.org/. How does the leaderboards tool
>>>>>>>> collect the information?
>>>>>>>>
>>>>>>>> Ideally, what I would like is:
>>>>>>>> - all users who contributed to tasking managers under id #1234
>>>>>>>> (from date/time-date/time), including incomplete tasks
>>>>>>>> - all users who contributed to the bbox of project #1234  (from
>>>>>>>> date/time-date/time)
>>>>>>>> - all users who used hashtag in a variety of spellings (#hashtag |
>>>>>>>> #Hashtag | etc)  (from date/time-date/time)
>>>>>>>> (- all users who attended a mapathon.... see below)
>>>>>>>>
>>>>>>>> Ideally as GET parameters or similar, i.e.
>>>>>>>>
>>>>>>>> http://someservice/?project=1234|1235|1236&includeincompletetasks=yes&includeprojectareas=yes&hastag=hashtag|hashtag2&hastagcasesensitive=no&start=...&end=.
>>>>>>>> ..
>>>>>>>>
>>>>>>>> which would return a list of users, with projects contributed to
>>>>>>>> and how that fact was determined (via project id1/2/3, project area for
>>>>>>>> project id1/2/3 or hashtag1, hashtag2, etc), plus number of contributions
>>>>>>>> per user in different categories (nodes, ways, way[building],
>>>>>>>> way[highway]), plus (one can but dream!) the age of their OSM account and
>>>>>>>> total changesets.
>>>>>>>>
>>>>>>>> Clearly there is redundancy in the query - but that's on purpose.
>>>>>>>> It would e.g. find people who are mapping in an area, but not working
>>>>>>>> through the task manager.
>>>>>>>>
>>>>>>>> It would also be amazing if on the tasking manager, people can
>>>>>>>> register their attendance at a mapathon. E.g. as you go to project page it
>>>>>>>> says: "A mapathon for this project is in progress. Click here if you are
>>>>>>>> participating in this mapathon in person or remotely."
>>>>>>>>
>>>>>>>> (E.g. In terms of the query, add includeattendeelist=yes
>>>>>>>> http://someservice/?....&includeattendeelist=yes )
>>>>>>>>
>>>>>>>> If you had such an attendee list tool, maybe you could even put in
>>>>>>>> your name and table number. Then we'd immediately know who is there, and
>>>>>>>> could start looking at their edits, and support them. I sometimes wander
>>>>>>>> round a mapathon trying to find a user who needs a bit of extra support.
>>>>>>>>
>>>>>>>> Having said all of this, I'd be very happy to help build such
>>>>>>>> tools, but would prefer to do this as part of a small team!
>>>>>>>> Bjoern
>>>>>>>>
>>>>>>>>
>>>>>>>> On 16 June 2017 at 14:09, Jan Martinec <jan at martinec.name> wrote:
>>>>>>>>
>>>>>>>>> Dne 16.6.2017 v 14:13 Bjoern Hassler napsal(a):
>>>>>>>>>
>>>>>>>>>> Dear friends,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What tools do we normally use to get statistics on a mapathon?
>>>>>>>>>>
>>>>>>>>>> Clearly the tasking manager provides contributors to the task in
>>>>>>>>>> the 'stats'
>>>>>>>>>> section, and also I can run overpass to look for changes made by
>>>>>>>>>> those users.
>>>>>>>>>>
>>>>>>>>>> I don't think it's possible to get changesets by #hashtag? You'd
>>>>>>>>>> have to use the
>>>>>>>>>> main API to get all change sets for the period of the event, and
>>>>>>>>>> then select the
>>>>>>>>>> ones that have the right hashtag(s)?
>>>>>>>>>>
>>>>>>>>>> Does anybody have some tools they could point me to?
>>>>>>>>>>
>>>>>>>>>> (Something like this
>>>>>>>>>> http://resultmaps.neis-one.org/osm-changesets would be
>>>>>>>>>> great, if it listed the changesets and users...)
>>>>>>>>>>
>>>>>>>>>> Many thanks!
>>>>>>>>>> Bjoern
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> HOT mailing list
>>>>>>>>>> HOT at openstreetmap.org
>>>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> would this be useful? I think it provides a numerical output as
>>>>>>>>> well as pretty pictures:
>>>>>>>>> https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Jan "Piskvor" Martinec
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> HOT mailing list
>>>>>>>>> HOT at openstreetmap.org
>>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> HOT mailing list
>>>>>>> HOT at openstreetmap.org
>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> HOT mailing list
>>>> HOT at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>
>>> --
>>>
>>> /Erwin Olario
>>>
>>> e: erwin at ngnuity.xyz | v/m: https://t.me/GOwin | s:
>>> https://mstdn.io/@GOwin
>>>
>>
>>
>>
>> _______________________________________________
>> HOT mailing listHOT at openstreetmap.orghttps://lists.openstreetmap.org/listinfo/hot
>>
>>
>> --
>> *Violaine Doutreleau*
>> Coordinatrice Missing Maps
>> CartONG
>> mobile : 06.95.02.42.44
>> skype : doutreleau.violaine
>>
>>
>> *P Help save paper - do you need to print this email?*
>>
>> _______________________________________________
>> HOT mailing list
>> HOT at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/hot
>>
>>
>
>
> --
> *Rebecca Firth*
> Community Partnerships Manager
> rebecca.firth at hotosm.org <tyler.radford at hotosm.org>
> @RebeccaFirthy
> Skype: rebeccafirth
>
> *Humanitarian OpenStreetMap Team*
> *Using OpenStreetMap for Humanitarian Response & Economic Development*
> web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> | facebook
> <https://www.facebook.com/hotosm> | donate <http://donate.hotosm.org/>
>
-- 

/Erwin Olario

e: erwin at ngnuity.xyz | v/m: https://t.me/GOwin | s: https://mstdn.io/@GOwin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20170627/ff80cd33/attachment.html>