[HOT] Tasking manager unstable

Marco Minghini marco.minghini86 at gmail.com
Wed Nov 13 15:20:22 UTC 2019

I also experienced the same problem when running my mapathon today at the
European Commission - JRC in Ispra (Italy):
https://tasks.hotosm.org/ was down, but for the purposes of the mapathon
(introducing humanitarian mapping and introducing participants to mapping)
the instance at https://tasks.teachosm.org was totally fine.

This solution might be useful for many other people involved in mapathons
during this week!


Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre <
felix.delattre at hotosm.org> ha scritto:

> Hi all,
> Yes, this is the worst time. Almost the whole tech team is sole focusing
> on that emergency.
> We had earlier today some problems with a maintenance procedure that went
> out of control. And at the moment we are facing the following issues on the
> Tasking Manager that has been brought back:
> - Locking of tasks is not working and throws an error message.
> - Login is failing for some, not for others
> - Activities of late yesterday, such as projects that have been created,
> might be lost in the database, as we had to use a backup from yesterday.
> We are working on solving it soon. And will report here and in our other
> channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
> information.
> In the meantime we can only offer alternative instances such as:
> * tm3.openstreetmap.id
> * tasks-assisted.hotosm.org
> * tasks.teachosm.org
> Please excuse the inconvenience and we will post any updates as soon as
> possible.
> So far an update from the infrastructure experts on what happened earlier
> today:
> > *Summary:*
> > Tasking Manager is now back and working. Unfortunately, many email
> addresses were lost and therefore some users who login will be asked for
> their email addresses again. Projects are intact and unaffected
> >
> > *So what happened?*
> > During the scheduled maintenance to upgrade Tasking Manager instance,
> one of our automated scripts deleted our database. This is not unexpected.
> But the backup we made before deletion was deleted along with the database.
> In a nutshell, we lost the production database completely, including
> automated backups. We however had a separate offline backup that was taken
> manually yesterday. But we had stripped email addresses from this database
> backup for privacy reasons. We had to restore email addresses from another
> offline backup from 10 days ago to fix it. We are now closely monitoring
> the DB and the application for any problems.
> >
> > *What can you expect in the future?*
> > We learned many valuable lessons from this including our approach to
> data backups, performance and stability. All our experiences from the
> outages of recent past have made us think about how to overhaul our
> approach to optimize our infrastructure. We will work to implement
> processes to mitigate performance, deployment and stability problems over
> the next few weeks.  We are also working on a detailed RCA with technical
> details of the outage.
> We understand that this would have caused quite a bit of frustration and
> we apologize for that. You can expect things to improve in the next few
> weeks. As always, you can follow us on Twitter  (
> https://twitter.com/hotosm_tech) or Slack for updates in addition to our
> status page at (https://status.hotosm.org).
> On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke <jorieke.vyncke at gmail.com>
> wrote:
>> Hi all,
>> Yes, please HOT, can you do everything to keep the TM online and stable
>> today / this week?
>> It's GISday today and there are more than 30 mapathons worldwide and 100+
>> this week for OSMgeoweek. Mapathons are happening in Uganda, Nigeria,
>> Amsterdam, Sydney, Rio, etc.
>> At the moment I tell people to go to tasks.teachosm.org and MapSwipe, if
>> the HOT TM is not working.
>> Best regards,
>> Jorieke
>> Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT <
>> hot at openstreetmap.org>:
>>> I'm sure you are all aware, but just in case... this is geo week and
>>> there are more than 70 mapathons planned for this week. Maybe updates from
>>> the tech team need to be posted across channels on this as it is critical
>>> infrastructure for all of these events and people might not be connected to
>>> slack / list / facebook etc...
>>> Cheers,
>>> Pete
>>> On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani <michealyani78 at gmail.com>
>>> wrote:
>>>>   Hello Tech team, we  just want to let you know that the tasking
>>>> manager isn't stable yet, It has has worked for 2 hrs and now is on and
>>>> off.
>>>> _______________________________________________
>>>> HOT mailing list
>>>> HOT at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/hot
>>> _______________________________________________
>>> HOT mailing list
>>> HOT at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>> _______________________________________________
>> HOT mailing list
>> HOT at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/hot
> --
> *Felix Delattre*
> Technical Project Manager
> felix.delattre at hotosm.org
> --
> *Humanitarian OpenStreetMap Team*
> *Using OpenStreetMap for Humanitarian Response & Economic Development*
> web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> | facebook
> <https://www.facebook.com/hotosm> | donate <https://donate.hotosm.org/>
> _______________________________________________
> HOT mailing list
> HOT at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20191113/415ee0a5/attachment.html>

More information about the HOT mailing list