[HOT] Tasking manager unstable

Wed Nov 13 20:08:05 UTC 2019

Thanks for the follow up Felix. It seems to work fine now! I'm seeing a lot
of happy mapping people faces on social media. :-)

On Wed, 13 Nov 2019, 18:11 Felix Delattre, <felix.delattre at hotosm.org>
wrote:

> Hi all,
>
> we are back with the HOT Tasking Manager. We are continuing to monitor the
> application servers, database, and network, and are investigating further
> what went wrong. Again, we apologize for the delays and frustration this
> has caused, especially during GeoWeek.
>
> Felix
>
> On 11/13/19 3:52 PM, Jorieke Vyncke wrote:
>
> Hi all,
> We have just send out the following email to all mapathon organisers, but
> hopefully the HOT Tasking Manager is properly up and running again soon.
> Happy mapping everyone! :-)
> Cheers,
> Jorieke
>
>
>
> Hello Mappers,
>
>
>
> Happy GIS day! Unfortunately many of you have experienced some issues with
> HOT Tasking Manager during mapathons today. We want to apologize for this
> inconvenience, and to share some guidance on what to do if this happens
> during your mapathon.
>
>
>
> *HOT Tasking Manager is experiencing some issues - why?*
>
> HOT Tasking Manager is currently unstable due to a failed maintenance from
> last night. We don’t presently know when this problem will be fixed. The
> HOT tech team is working as fast as possible to fix the problem. To find
> out more about the status, please check:
>
>    - Status.hotosm.org <http://status.hotosm.org/>: this will tell you if
>    the tool is down, not if it is experiencing delays (as it is presently)
>    - follow @hotosm_tech on Twitter for updates
>    - Join the HOT Slack (http://slack.hotosm.org/) and ask questions in
>    the #tasking-manager channel
>
>
>
> *Contingency plans for your mapathon: what are your options if the HOT
> Tasking Manager is not working during your event?*
>
>    1. Map directly in OSM! Check out an area you know well and contribute
>    your own knowledge. If you’d like to host a map-walk, please see this guide
>    on maps.me <https://osmgeoweek.org/docs/maps_me_general_guide.pdf>.
>    2. Host a Mapswipeathon. MapSwipe <https://mapswipe.org/> has several
>    tasks up and running on the app. You can help swiping in Indonesia,
>    Venezuela, Philippines and Colombia. Please note this is a mobile app only.
>    3. There are alternative Tasking manager instances you can use like
>    https://tm3.openstreetmap.id/ & tasks.teachosm.org. Please note that
>    these will not have the same projects on them, so you’ll need to pick a new
>    project.
>
> *Note:* Mapswipe, and the alternative Tasking Managers, are completely
> separate systems that let you contribute to OSM, so they should *not *be
> affected by the outage of the main HOT Tasking Manager.
>
>
>
> Thank you for your understanding, we appreciate your flexibility. We are
> working with the HOT Tech Team to get this fixed as soon as possible.  If
> you have any questions or need help course correcting please reach out.
>
>
>
> Op wo 13 nov. 2019 om 15:20 schreef Marco Minghini <
> marco.minghini86 at gmail.com>:
>
>> I also experienced the same problem when running my mapathon today at the
>> European Commission - JRC in Ispra (Italy):
>> https://tasks.hotosm.org/ was down, but for the purposes of the mapathon
>> (introducing humanitarian mapping and introducing participants to mapping)
>> the instance at https://tasks.teachosm.org was totally fine.
>>
>> This solution might be useful for many other people involved in mapathons
>> during this week!
>>
>> Best,
>> Marco
>>
>>
>> Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre <
>> felix.delattre at hotosm.org> ha scritto:
>>
>>> Hi all,
>>>
>>> Yes, this is the worst time. Almost the whole tech team is sole focusing
>>> on that emergency.
>>>
>>> We had earlier today some problems with a maintenance procedure that
>>> went out of control. And at the moment we are facing the following issues
>>> on the Tasking Manager that has been brought back:
>>>
>>> - Locking of tasks is not working and throws an error message.
>>> - Login is failing for some, not for others
>>> - Activities of late yesterday, such as projects that have been created,
>>> might be lost in the database, as we had to use a backup from yesterday.
>>>
>>> We are working on solving it soon. And will report here and in our other
>>> channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
>>> information.
>>>
>>> In the meantime we can only offer alternative instances such as:
>>>
>>> * tm3.openstreetmap.id
>>> * tasks-assisted.hotosm.org
>>> * tasks.teachosm.org
>>>
>>> Please excuse the inconvenience and we will post any updates as soon as
>>> possible.
>>>
>>> So far an update from the infrastructure experts on what happened
>>> earlier today:
>>>
>>> > *Summary:*
>>> > Tasking Manager is now back and working. Unfortunately, many email
>>> addresses were lost and therefore some users who login will be asked for
>>> their email addresses again. Projects are intact and unaffected
>>> >
>>> > *So what happened?*
>>> > During the scheduled maintenance to upgrade Tasking Manager instance,
>>> one of our automated scripts deleted our database. This is not unexpected.
>>> But the backup we made before deletion was deleted along with the database.
>>> In a nutshell, we lost the production database completely, including
>>> automated backups. We however had a separate offline backup that was taken
>>> manually yesterday. But we had stripped email addresses from this database
>>> backup for privacy reasons. We had to restore email addresses from another
>>> offline backup from 10 days ago to fix it. We are now closely monitoring
>>> the DB and the application for any problems.
>>> >
>>> > *What can you expect in the future?*
>>> > We learned many valuable lessons from this including our approach to
>>> data backups, performance and stability. All our experiences from the
>>> outages of recent past have made us think about how to overhaul our
>>> approach to optimize our infrastructure. We will work to implement
>>> processes to mitigate performance, deployment and stability problems over
>>> the next few weeks.  We are also working on a detailed RCA with technical
>>> details of the outage.
>>> We understand that this would have caused quite a bit of frustration and
>>> we apologize for that. You can expect things to improve in the next few
>>> weeks. As always, you can follow us on Twitter  (
>>> https://twitter.com/hotosm_tech) or Slack for updates in addition to
>>> our status page at (https://status.hotosm.org).
>>>
>>>
>>>
>>> On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke <
>>> jorieke.vyncke at gmail.com> wrote:
>>>
>>>> Hi all,
>>>> Yes, please HOT, can you do everything to keep the TM online and stable
>>>> today / this week?
>>>> It's GISday today and there are more than 30 mapathons worldwide and
>>>> 100+ this week for OSMgeoweek. Mapathons are happening in Uganda, Nigeria,
>>>> Amsterdam, Sydney, Rio, etc.
>>>> At the moment I tell people to go to tasks.teachosm.org and MapSwipe,
>>>> if the HOT TM is not working.
>>>> Best regards,
>>>> Jorieke
>>>>
>>>> Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT <
>>>> hot at openstreetmap.org>:
>>>>
>>>>> I'm sure you are all aware, but just in case... this is geo week and
>>>>> there are more than 70 mapathons planned for this week. Maybe updates from
>>>>> the tech team need to be posted across channels on this as it is critical
>>>>> infrastructure for all of these events and people might not be connected to
>>>>> slack / list / facebook etc...
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Pete
>>>>>
>>>>> On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani <michealyani78 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>   Hello Tech team, we  just want to let you know that the tasking
>>>>>> manager isn't stable yet, It has has worked for 2 hrs and now is on and
>>>>>> off.
>>>>>> _______________________________________________
>>>>>> HOT mailing list
>>>>>> HOT at openstreetmap.org
>>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>>
>>>>> _______________________________________________
>>>>> HOT mailing list
>>>>> HOT at openstreetmap.org
>>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>>
>>>> _______________________________________________
>>>> HOT mailing list
>>>> HOT at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/hot
>>>>
>>>
>>>
>>> --
>>>
>>> *Felix Delattre *
>>> Technical Project Manager
>>> felix.delattre at hotosm.org
>>>
>>> --
>>> *Humanitarian OpenStreetMap Team*
>>> *Using OpenStreetMap for Humanitarian Response & Economic Development*
>>> web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> |
>>> facebook <https://www.facebook.com/hotosm> | donate
>>> <https://donate.hotosm.org/>
>>> _______________________________________________
>>> HOT mailing list
>>> HOT at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20191113/a707e2d3/attachment.html>