[HOT] Live stats for mapathons / RedCross API / JOSM plugin (was: Mapathon stats analysis)

Bjoern Hassler bjohas+mw at gmail.com
Tue Jun 20 19:09:24 UTC 2017


Hi Erwin, dear friends,

I have some nodejs code here: https://github.com/OSM-
Utilities/JOSM-Scripts-HOT/blob/master/lib/mmstats.js. It shows users
contributing under a hashtag (using the osmstats api) and then shows edits
by those users (buildings, roads, but also other objects), as well as
common issues (e.g. isolated untagged notes, untagged ways, non-square
4-node buildings). E.g. you'll see that for some users the non-square
buildings are a few % of their edits, which means they are just omissions.
For other users, it's more like 50%, which means they are not squaring
systematically.

On a technical note: In terms of changed and diff: That works for stats,
but obviously also interested in validation. What I'd really like is a
query that gives me the "current state of everything that changed between
these times". In the above code, you'll see some recursion, that should do
that (though I have to do some more testing, and there may be some more
recursion required). I was trying to keep it all within one query as well,
as the queries do take time.

Let me know if you have any questions or comments!
Bjoern

On 20 June 2017 at 07:48, Erwin Olario <govvin at gmail.com> wrote:

>
> Instead of using changed, I run a query [0]: using diff and filter the
> edits made by only by users who are present in the mapathon:
>
> Caveat: there's an existing bug when querying dates with UTC offsets, so
> use the actual offset time instead.
>
> Also, instead of filtering by "uid:", you may use "user:" for usernames.
> However, some new users eventually change their names to something else, so
> I prefer to use the former.
>
> [0]: https://gist.github.com/govvin/8cc3cfff204314536d744fe82c52e6f3
>
>
> On Sat, Jun 17, 2017 at 7:45 PM Bjoern Hassler <bjohas+mw at gmail.com>
> wrote:
>
>> Dear friends,
>>
>> I looked into https://github.com/AmericanRedCross/osm-stats a little,
>> and while I can't quite get overpass results to match with what the API
>> outputs, it does have very useful features.
>>
>> Is anybody interested in (or already working on) getting (live) stats for
>> mapathons, via enhancements to osm-stats or otherwise?
>>
>> E.g. you could
>> 1. Use osm-stats to get users making changes with specific hastags
>> 2. Use overpass to load objects changed by those users since start of
>> mapathon
>>
>> You could then run some stats (similar to the /user endpoint in
>> osm-stats). These stats could also include results from running queries on
>> bounding boxes, or on user-lists. This could all be done in combination
>> with osm-stats and overpass, but there may be a case for integrating it
>> into osm-stats, see https://github.com/AmericanRedCross/osm-stats/
>> issues/45
>>
>> Either with the process above, or an extended osm-stats, you could also
>> systematically check for common issues, e.g. untagged single nodes, ways
>> with four nodes which are not squared, etc. I.e. you could automatically
>> check for such common issues across the whole mapathon, with output like:
>> "OSM user ABC added X untagged nodes and Y non-squared buildings." rather
>> than wait for a validator to spot the issues manually / later.
>>
>> Of course, some of this this can be done via MapPaint/MapCSS-styles in
>> JOSM too (e.g. single nodes), but I am not sure how you could do squared
>> buildings. Also across a large area, the above could also be useful for an
>> overview (and giving an indication of somebody making systematic mistake
>> vs. one-off slips). The JOSM plugin could also take you to those places
>> where there are issues, to inspect this (or even within tasks to fix it).
>> Definitely going beyond what you can do with MapCSS or validation rule, you
>> could use the JOSM plugin to visit new buildings (and whatever other
>> objects) to check the tracing was done accurately.
>>
>> Of course, there are different approaches, that give you almost the same
>> (i.e. using MapCSS or validation rules, To Do plugin etc), but a dedicated
>> "MM mapathon validation" plugin would be easy to set up and could be made
>> user friendly. It wouldn't replace the usual validation process, but would
>> enable a "key validator" to keep an overview during a mapathon.
>>
>> I'd be keen to hear people's views, and whether anybody else feels this
>> would be worth-while and useful.
>>
>> What do you think?
>> Bjoern
>>
>>
>> On 16 June 2017 at 17:07, Bjoern Hassler <bjohas+mw at gmail.com> wrote:
>>
>>> Hi Mike,
>>>
>>> Thanks for that. Yeah, at the moment I'm using overpass queries with
>>> (changed:"2017-06-15T17:00:00Z","2017-06-15T21:00:00Z"); to work out
>>> changes during the mapathon. You could go by bbox (and perhaps catch some
>>> other MM mappers who were mapping there independently or, as you say
>>> unlikely, some other mappers). Or you can go by user (no bbox) in case
>>> users contributed to several projects, or by accident picked another
>>> project.
>>>
>>> Is there a simple way of getting the bbox from the tasking manager? That
>>> would be great. (Of course, it can be computed, e.g. by downloading the gpx
>>> box from the NE and SW tiles... I know it's simple... but not as simple as
>>> clicking a button on the tasking manager that takes you from a task to the
>>> overpass query with the bbox filled in :)
>>>
>>> The leaderboard uses this https://github.com/AmericanRedCross, which is
>>> helpful, as the Overpass API cannot query on the changeset comments, but
>>> the redcross api can (and so you don't need to use the main OSM API).
>>>
>>> Bjoern
>>>
>>> On 16 June 2017 at 16:27, Mike Thompson <miketho16 at gmail.com> wrote:
>>>
>>>> One can use Overpass to get all of the OSM elements which were added,
>>>> changed or deleted during the time of your mapathon in your AOI.  These
>>>> will be tagged with the user who last touched them, so you can get stats by
>>>> user.  The disadvantage is that you will include edits by users who were
>>>> not part of your mapathon but who happened to made edits to OSM in that AOI
>>>> during the time period of your mapathon.  However, unless you are working
>>>> on a crisis task, this is not likely to skew your results too much in my
>>>> experience.
>>>>
>>>> I sometimes make a Carto map of the results, e.g.:
>>>> https://tekim.carto.com/viz/baf5c3a2-2aa8-11e7-88b2-
>>>> 0ef24382571b/embed_map
>>>>
>>>> I have more detailed instructions should someone be interested.
>>>>
>>>> Mike
>>>>
>>>>
>>>> On Fri, Jun 16, 2017 at 9:15 AM, Donal Hunt <donal.hunt at gmail.com>
>>>> wrote:
>>>>
>>>>> *[I don't speak for the team (I'm just a volunteer contributor to
>>>>> missing maps) so someone with more context may chime in with better
>>>>> info...]*
>>>>>
>>>>> I'm not familiar with the implementation of the leaderboard at all.
>>>>> Someone from the HOTOSM web team probably knows. Or you can file a feature
>>>>> request here I think: https://github.com/MissingMaps/missingmaps.
>>>>> github.io/issues
>>>>>
>>>>> For the features specific to the tasking manager, I suspect that
>>>>> filing them here
>>>>> <https://github.com/hotosm/osm-tasking-manager2/issues> (
>>>>> https://github.com/hotosm/osm-tasking-manager2/issues) is the right
>>>>> thing to do. That way one of the regular maintainers or a volunteer
>>>>> developer can see the priority of the feature being developed and
>>>>> contribute to it being implemented.
>>>>>
>>>>> For the stats related requests, I suspect something probably exists
>>>>> already within the OSM community / ecosystem that could be tweaked for your
>>>>> needs. I don't have enough context right now to suggest something (maybe in
>>>>> the future).
>>>>>
>>>>> Regards
>>>>>
>>>>> Donal
>>>>>
>>>>> On Fri, Jun 16, 2017 at 3:38 PM, Bjoern Hassler <bjohas+mw at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Dear Jan, dear Donal, dear friends,
>>>>>>
>>>>>> Is there an instance of https://github.com/tgrippa/
>>>>>> Mapathon_HOT_OSM_WhatWeMapped running somewhere?
>>>>>>
>>>>>> This is great: http://www.missingmaps.org/leaderboards/#/HASHTAG,
>>>>>> though there's some discrepancy between the data it shows and what I can
>>>>>> see from the overpass api directly and from
>>>>>> http://resultmaps.neis-one.org/. How does the leaderboards tool
>>>>>> collect the information?
>>>>>>
>>>>>> Ideally, what I would like is:
>>>>>> - all users who contributed to tasking managers under id #1234 (from
>>>>>> date/time-date/time), including incomplete tasks
>>>>>> - all users who contributed to the bbox of project #1234  (from
>>>>>> date/time-date/time)
>>>>>> - all users who used hashtag in a variety of spellings (#hashtag |
>>>>>> #Hashtag | etc)  (from date/time-date/time)
>>>>>> (- all users who attended a mapathon.... see below)
>>>>>>
>>>>>> Ideally as GET parameters or similar, i.e.
>>>>>> http://someservice/?project=1234|1235|1236&
>>>>>> includeincompletetasks=yes&includeprojectareas=yes&
>>>>>> hastag=hashtag|hashtag2&hastagcasesensitive=no&start=...&end=...
>>>>>>
>>>>>> which would return a list of users, with projects contributed to and
>>>>>> how that fact was determined (via project id1/2/3, project area for project
>>>>>> id1/2/3 or hashtag1, hashtag2, etc), plus number of contributions per user
>>>>>> in different categories (nodes, ways, way[building], way[highway]), plus
>>>>>> (one can but dream!) the age of their OSM account and total changesets.
>>>>>>
>>>>>> Clearly there is redundancy in the query - but that's on purpose. It
>>>>>> would e.g. find people who are mapping in an area, but not working through
>>>>>> the task manager.
>>>>>>
>>>>>> It would also be amazing if on the tasking manager, people can
>>>>>> register their attendance at a mapathon. E.g. as you go to project page it
>>>>>> says: "A mapathon for this project is in progress. Click here if you are
>>>>>> participating in this mapathon in person or remotely."
>>>>>>
>>>>>> (E.g. In terms of the query, add includeattendeelist=yes
>>>>>> http://someservice/?....&includeattendeelist=yes )
>>>>>>
>>>>>> If you had such an attendee list tool, maybe you could even put in
>>>>>> your name and table number. Then we'd immediately know who is there, and
>>>>>> could start looking at their edits, and support them. I sometimes wander
>>>>>> round a mapathon trying to find a user who needs a bit of extra support.
>>>>>>
>>>>>> Having said all of this, I'd be very happy to help build such tools,
>>>>>> but would prefer to do this as part of a small team!
>>>>>> Bjoern
>>>>>>
>>>>>>
>>>>>> On 16 June 2017 at 14:09, Jan Martinec <jan at martinec.name> wrote:
>>>>>>
>>>>>>> Dne 16.6.2017 v 14:13 Bjoern Hassler napsal(a):
>>>>>>>
>>>>>>>> Dear friends,
>>>>>>>>
>>>>>>>>
>>>>>>>> What tools do we normally use to get statistics on a mapathon?
>>>>>>>>
>>>>>>>> Clearly the tasking manager provides contributors to the task in
>>>>>>>> the 'stats'
>>>>>>>> section, and also I can run overpass to look for changes made by
>>>>>>>> those users.
>>>>>>>>
>>>>>>>> I don't think it's possible to get changesets by #hashtag? You'd
>>>>>>>> have to use the
>>>>>>>> main API to get all change sets for the period of the event, and
>>>>>>>> then select the
>>>>>>>> ones that have the right hashtag(s)?
>>>>>>>>
>>>>>>>> Does anybody have some tools they could point me to?
>>>>>>>>
>>>>>>>> (Something like this http://resultmaps.neis-one.org/osm-changesets
>>>>>>>> would be
>>>>>>>> great, if it listed the changesets and users...)
>>>>>>>>
>>>>>>>> Many thanks!
>>>>>>>> Bjoern
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> HOT mailing list
>>>>>>>> HOT at openstreetmap.org
>>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>>
>>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> would this be useful? I think it provides a numerical output as well
>>>>>>> as pretty pictures:
>>>>>>> https://github.com/tgrippa/Mapathon_HOT_OSM_WhatWeMapped
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Jan "Piskvor" Martinec
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> HOT mailing list
>>>>>>> HOT at openstreetmap.org
>>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> HOT mailing list
>>>>> HOT at openstreetmap.org
>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>
>>>>>
>>>>
>>>
>> _______________________________________________
>> HOT mailing list
>> HOT at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/hot
>>
> --
>
> /Erwin Olario
>
> e: erwin at ngnuity.xyz | v/m: https://t.me/GOwin | s:
> https://mstdn.io/@GOwin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20170620/56bcd975/attachment.html>


More information about the HOT mailing list