[HOT] Tasking manager unstable

Wed Nov 13 12:30:46 UTC 2019

Hi all,

Yes, this is the worst time. Almost the whole tech team is sole focusing
on that emergency.

We had earlier today some problems with a maintenance procedure that went
out of control. And at the moment we are facing the following issues on the
Tasking Manager that has been brought back:

- Locking of tasks is not working and throws an error message.
- Login is failing for some, not for others
- Activities of late yesterday, such as projects that have been created,
might be lost in the database, as we had to use a backup from yesterday.

We are working on solving it soon. And will report here and in our other
channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
information.

In the meantime we can only offer alternative instances such as:

* tm3.openstreetmap.id
* tasks-assisted.hotosm.org
* tasks.teachosm.org

Please excuse the inconvenience and we will post any updates as soon as
possible.

So far an update from the infrastructure experts on what happened earlier
today:

> *Summary:*
> Tasking Manager is now back and working. Unfortunately, many email
addresses were lost and therefore some users who login will be asked for
their email addresses again. Projects are intact and unaffected
>
> *So what happened?*
> During the scheduled maintenance to upgrade Tasking Manager instance, one
of our automated scripts deleted our database. This is not unexpected. But
the backup we made before deletion was deleted along with the database. In
a nutshell, we lost the production database completely, including automated
backups. We however had a separate offline backup that was taken manually
yesterday. But we had stripped email addresses from this database backup
for privacy reasons. We had to restore email addresses from another offline
backup from 10 days ago to fix it. We are now closely monitoring the DB and
the application for any problems.
>
> *What can you expect in the future?*
> We learned many valuable lessons from this including our approach to data
backups, performance and stability. All our experiences from the outages of
recent past have made us think about how to overhaul our approach to
optimize our infrastructure. We will work to implement processes to
mitigate performance, deployment and stability problems over the next few
weeks.  We are also working on a detailed RCA with technical details of the
outage.
We understand that this would have caused quite a bit of frustration and we
apologize for that. You can expect things to improve in the next few weeks.
As always, you can follow us on Twitter  (https://twitter.com/hotosm_tech)
or Slack for updates in addition to our status page at (
https://status.hotosm.org).

On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke <jorieke.vyncke at gmail.com>
wrote:

> Hi all,
> Yes, please HOT, can you do everything to keep the TM online and stable
> today / this week?
> It's GISday today and there are more than 30 mapathons worldwide and 100+
> this week for OSMgeoweek. Mapathons are happening in Uganda, Nigeria,
> Amsterdam, Sydney, Rio, etc.
> At the moment I tell people to go to tasks.teachosm.org and MapSwipe, if
> the HOT TM is not working.
> Best regards,
> Jorieke
>
> Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT <
> hot at openstreetmap.org>:
>
>> I'm sure you are all aware, but just in case... this is geo week and
>> there are more than 70 mapathons planned for this week. Maybe updates from
>> the tech team need to be posted across channels on this as it is critical
>> infrastructure for all of these events and people might not be connected to
>> slack / list / facebook etc...
>>
>> Cheers,
>>
>> Pete
>>
>> On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani <michealyani78 at gmail.com>
>> wrote:
>>
>>>   Hello Tech team, we  just want to let you know that the tasking
>>> manager isn't stable yet, It has has worked for 2 hrs and now is on and
>>> off.
>>> _______________________________________________
>>> HOT mailing list
>>> HOT at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>>>
>> _______________________________________________
>> HOT mailing list
>> HOT at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/hot
>>
> _______________________________________________
> HOT mailing list
> HOT at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot
>

-- 

*Felix Delattre*
Technical Project Manager
felix.delattre at hotosm.org

--
*Humanitarian OpenStreetMap Team*
*Using OpenStreetMap for Humanitarian Response & Economic Development*
web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> | facebook
<https://www.facebook.com/hotosm> | donate <https://donate.hotosm.org/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20191113/b60b690d/attachment.html>