[HOT] Tasking manager unstable

Wed Nov 13 15:52:38 UTC 2019

Hi all,
We have just send out the following email to all mapathon organisers, but
hopefully the HOT Tasking Manager is properly up and running again soon.
Happy mapping everyone! :-)
Cheers,
Jorieke

Hello Mappers,

Happy GIS day! Unfortunately many of you have experienced some issues with
HOT Tasking Manager during mapathons today. We want to apologize for this
inconvenience, and to share some guidance on what to do if this happens
during your mapathon.

*HOT Tasking Manager is experiencing some issues - why?*

HOT Tasking Manager is currently unstable due to a failed maintenance from
last night. We don’t presently know when this problem will be fixed. The
HOT tech team is working as fast as possible to fix the problem. To find
out more about the status, please check:

   - Status.hotosm.org <http://status.hotosm.org/>: this will tell you if
   the tool is down, not if it is experiencing delays (as it is presently)
   - follow @hotosm_tech on Twitter for updates
   - Join the HOT Slack (http://slack.hotosm.org/) and ask questions in the
   #tasking-manager channel

*Contingency plans for your mapathon: what are your options if the HOT
Tasking Manager is not working during your event?*

   1. Map directly in OSM! Check out an area you know well and contribute
   your own knowledge. If you’d like to host a map-walk, please see this guide
   on maps.me <https://osmgeoweek.org/docs/maps_me_general_guide.pdf>.
   2. Host a Mapswipeathon. MapSwipe <https://mapswipe.org/> has several
   tasks up and running on the app. You can help swiping in Indonesia,
   Venezuela, Philippines and Colombia. Please note this is a mobile app only.
   3. There are alternative Tasking manager instances you can use like
   https://tm3.openstreetmap.id/ & tasks.teachosm.org. Please note that
   these will not have the same projects on them, so you’ll need to pick a new
   project.

*Note:* Mapswipe, and the alternative Tasking Managers, are completely
separate systems that let you contribute to OSM, so they should *not *be
affected by the outage of the main HOT Tasking Manager.

Thank you for your understanding, we appreciate your flexibility. We are
working with the HOT Tech Team to get this fixed as soon as possible.  If
you have any questions or need help course correcting please reach out.

Op wo 13 nov. 2019 om 15:20 schreef Marco Minghini <
marco.minghini86 at gmail.com>:

> I also experienced the same problem when running my mapathon today at the
> European Commission - JRC in Ispra (Italy):
> https://tasks.hotosm.org/ was down, but for the purposes of the mapathon
> (introducing humanitarian mapping and introducing participants to mapping)
> the instance at https://tasks.teachosm.org was totally fine.
>
> This solution might be useful for many other people involved in mapathons
> during this week!
>
> Best,
> Marco
>
>
> Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre <
> felix.delattre at hotosm.org> ha scritto:
>
>> Hi all,
>>
>> Yes, this is the worst time. Almost the whole tech team is sole focusing
>> on that emergency.
>>
>> We had earlier today some problems with a maintenance procedure that went
>> out of control. And at the moment we are facing the following issues on the
>> Tasking Manager that has been brought back:
>>
>> - Locking of tasks is not working and throws an error message.
>> - Login is failing for some, not for others
>> - Activities of late yesterday, such as projects that have been created,
>> might be lost in the database, as we had to use a backup from yesterday.
>>
>> We are working on solving it soon. And will report here and in our other
>> channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
>> information.
>>
>> In the meantime we can only offer alternative instances such as:
>>
>> * tm3.openstreetmap.id
>> * tasks-assisted.hotosm.org
>> * tasks.teachosm.org
>>
>> Please excuse the inconvenience and we will post any updates as soon as
>> possible.
>>
>> So far an update from the infrastructure experts on what happened earlier
>> today:
>>
>> > *Summary:*
>> > Tasking Manager is now back and working. Unfortunately, many email
>> addresses were lost and therefore some users who login will be asked for
>> their email addresses again. Projects are intact and unaffected
>> >
>> > *So what happened?*
>> > During the scheduled maintenance to upgrade Tasking Manager instance,
>> one of our automated scripts deleted our database. This is not unexpected.
>> But the backup we made before deletion was deleted along with the database.
>> In a nutshell, we lost the production database completely, including
>> automated backups. We however had a separate offline backup that was taken
>> manually yesterday. But we had stripped email addresses from this database
>> backup for privacy reasons. We had to restore email addresses from another
>> offline backup from 10 days ago to fix it. We are now closely monitoring
>> the DB and the application for any problems.
>> >
>> > *What can you expect in the future?*
>> > We learned many valuable lessons from this including our approach to
>> data backups, performance and stability. All our experiences from the
>> outages of recent past have made us think about how to overhaul our
>> approach to optimize our infrastructure. We will work to implement
>> processes to mitigate performance, deployment and stability problems over
>> the next few weeks.  We are also working on a detailed RCA with technical
>> details of the outage.
>> We understand that this would have caused quite a bit of frustration and
>> we apologize for that. You can expect things to improve in the next few
>> weeks. As always, you can follow us on Twitter  (
>> https://twitter.com/hotosm_tech) or Slack for updates in addition to our
>> status page at (https://status.hotosm.org).
>>
>>
>>
>> On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke <jorieke.vyncke at gmail.com>
>> wrote:
>>
>>> Hi all,
>>> Yes, please HOT, can you do everything to keep the TM online and stable
>>> today / this week?
>>> It's GISday today and there are more than 30 mapathons worldwide and
>>> 100+ this week for OSMgeoweek. Mapathons are happening in Uganda, Nigeria,
>>> Amsterdam, Sydney, Rio, etc.
>>> At the moment I tell people to go to tasks.teachosm.org and MapSwipe,
>>> if the HOT TM is not working.
>>> Best regards,
>>> Jorieke
>>>
>>> Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT <
>>> hot at openstreetmap.org>:
>>>
>>>> I'm sure you are all aware, but just in case... this is geo week and
>>>> there are more than 70 mapathons planned for this week. Maybe updates from
>>>> the tech team need to be posted across channels on this as it is critical
>>>> infrastructure for all of these events and people might not be connected to
>>>> slack / list / facebook etc...
>>>>
>>>> Cheers,
>>>>
>>>> Pete
>>>>
>>>> On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani <michealyani78 at gmail.com>
>>>> wrote:
>>>>
>>>>>   Hello Tech team, we  just want to let you know that the tasking
>>>>> manager isn't stable yet, It has has worked for 2 hrs and now is on and
>>>>> off.
>>>>> _______________________________________________
>>>>> HOT mailing list
>>>>> HOT at openstreetmap.org
>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>
>>>> _______________________________________________
>>>> HOT mailing list
>>>> HOT at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>
>>> _______________________________________________
>>> HOT mailing list
>>> HOT at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>>>
>>
>>
>> --
>>
>> *Felix Delattre*
>> Technical Project Manager
>> felix.delattre at hotosm.org
>>
>> --
>> *Humanitarian OpenStreetMap Team*
>> *Using OpenStreetMap for Humanitarian Response & Economic Development*
>> web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> |
>> facebook <https://www.facebook.com/hotosm> | donate
>> <https://donate.hotosm.org/>
>> _______________________________________________
>> HOT mailing list
>> HOT at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/hot
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20191113/539bbec4/attachment.html>