[HOT] Tasking manager unstable

Felix Delattre felix.delattre at hotosm.org
Wed Nov 13 18:11:32 UTC 2019

Hi all,

we are back with the HOT Tasking Manager. We are continuing to monitor
the application servers, database, and network, and are investigating
further what went wrong. Again, we apologize for the delays and
frustration this has caused, especially during GeoWeek.


On 11/13/19 3:52 PM, Jorieke Vyncke wrote:
> Hi all, 
> We have just send out the following email to all mapathon organisers,
> but hopefully the HOT Tasking Manager is properly up and running again
> soon. 
> Happy mapping everyone! :-) 
> Cheers, 
> Jorieke
> Hello Mappers,
> Happy GIS day! Unfortunately many of you have experienced some issues
> with HOT Tasking Manager during mapathons today. We want to apologize
> for this inconvenience, and to share some guidance on what to do if
> this happens during your mapathon.
> *HOT Tasking Manager is experiencing some issues - why?*
> HOT Tasking Manager is currently unstable due to a failed maintenance
> from last night. We don’t presently know when this problem will be
> fixed. The HOT tech team is working as fast as possible to fix the
> problem. To find out more about the status, please check:
>   * Status.hotosm.org <http://status.hotosm.org/>: this will tell you
>     if the tool is down, not if it is experiencing delays (as it is
>     presently)
>   * follow @hotosm_tech on Twitter for updates
>   * Join the HOT Slack (http://slack.hotosm.org/) and ask questions in
>     the #tasking-manager channel
> *Contingency plans for your mapathon: what are your options if the HOT
> Tasking Manager is not working during your event?*
>  1. Map directly in OSM! Check out an area you know well and
>     contribute your own knowledge. If you’d like to host a map-walk,
>     please see this guide on maps.me
>     <https://osmgeoweek.org/docs/maps_me_general_guide.pdf>.
>  2. Host a Mapswipeathon. MapSwipe <https://mapswipe.org/>has several
>     tasks up and running on the app. You can help swiping in
>     Indonesia, Venezuela, Philippines and Colombia. Please note this
>     is a mobile app only.
>  3. There are alternative Tasking manager instances you can use like
>     https://tm3.openstreetmap.id/&tasks.teachosm.org
>     <http://tasks.teachosm.org/>. Please note that these will not have
>     the same projects on them, so you’ll need to pick a new project.
> *Note:*Mapswipe, and the alternative Tasking Managers, are completely
> separate systems that let you contribute to OSM, so they should *not
> *be affected by the outage of the main HOT Tasking Manager.
> Thank you for your understanding, we appreciate your flexibility. We
> are working with the HOT Tech Team to get this fixed as soon as
> possible.  If you have any questions or need help course correcting
> please reach out.
> Op wo 13 nov. 2019 om 15:20 schreef Marco Minghini
> <marco.minghini86 at gmail.com <mailto:marco.minghini86 at gmail.com>>:
>     I also experienced the same problem when running my mapathon today
>     at the European Commission - JRC in Ispra (Italy):
>     https://tasks.hotosm.org/ was down, but for the purposes of the
>     mapathon (introducing humanitarian mapping and introducing
>     participants to mapping) the instance at
>     https://tasks.teachosm.org was totally fine.
>     This solution might be useful for many other people involved in
>     mapathons during this week!
>     Best,
>     Marco
>     Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre
>     <felix.delattre at hotosm.org <mailto:felix.delattre at hotosm.org>> ha
>     scritto:
>         Hi all,
>         Yes, this is the worst time. Almost the whole tech team is
>         sole focusing on that emergency.
>         We had earlier today some problems with a maintenance
>         procedure that went out of control. And at the moment we are
>         facing the following issues on the Tasking Manager that has
>         been brought back:
>         - Locking of tasks is not working and throws an error message. 
>         - Login is failing for some, not for others
>         - Activities of late yesterday, such as projects that have
>         been created, might be lost in the database, as we had to use
>         a backup from yesterday.
>         We are working on solving it soon. And will report here and in
>         our other channels (slack, twitter.com/hotosm_tech
>         <http://twitter.com/hotosm_tech> and status.hotosm.org
>         <http://status.hotosm.org>) more information.
>         In the meantime we can only offer alternative instances such as:
>         * tm3.openstreetmap.id <http://tm3.openstreetmap.id>
>         * tasks-assisted.hotosm.org <http://tasks-assisted.hotosm.org>
>         * tasks.teachosm.org <http://tasks.teachosm.org>
>         Please excuse the inconvenience and we will post any
>         updates as soon as possible.
>         So far an update from the infrastructure experts on what
>         happened earlier today:
>         > *Summary:*
>         > Tasking Manager is now back and working. Unfortunately, many
>         email addresses were lost and therefore some users who login
>         will be asked for their email addresses again. Projects are
>         intact and unaffected
>         >
>         > *So what happened?*
>         > During the scheduled maintenance to upgrade Tasking Manager
>         instance, one of our automated scripts deleted our database.
>         This is not unexpected. But the backup we made before deletion
>         was deleted along with the database. In a nutshell, we lost
>         the production database completely, including automated
>         backups. We however had a separate offline backup that was
>         taken manually yesterday. But we had stripped email addresses
>         from this database backup for privacy reasons. We had to
>         restore email addresses from another offline backup from 10
>         days ago to fix it. We are now closely monitoring the DB and
>         the application for any problems.
>         >
>         > *What can you expect in the future?*
>         > We learned many valuable lessons from this including our
>         approach to data backups, performance and stability. All our
>         experiences from the outages of recent past have made us think
>         about how to overhaul our approach to optimize our
>         infrastructure. We will work to implement processes to
>         mitigate performance, deployment and stability problems over
>         the next few weeks.  We are also working on a detailed RCA
>         with technical details of the outage.
>         We understand that this would have caused quite a bit of
>         frustration and we apologize for that. You can expect things
>         to improve in the next few weeks. As always, you can follow us
>         on Twitter  (https://twitter.com/hotosm_tech) or Slack for
>         updates in addition to our status page at
>         (https://status.hotosm.org).
>         On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke
>         <jorieke.vyncke at gmail.com <mailto:jorieke.vyncke at gmail.com>>
>         wrote:
>             Hi all, 
>             Yes, please HOT, can you do everything to keep the TM
>             online and stable today / this week? 
>             It's GISday today and there are more than 30 mapathons
>             worldwide and 100+ this week for OSMgeoweek. Mapathons are
>             happening in Uganda, Nigeria, Amsterdam, Sydney, Rio, etc.
>             At the moment I tell people to go to tasks.teachosm.org
>             <http://tasks.teachosm.org> and MapSwipe, if the HOT TM is
>             not working. 
>             Best regards, 
>             Jorieke
>             Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT
>             <hot at openstreetmap.org <mailto:hot at openstreetmap.org>>:
>                 I'm sure you are all aware, but just in case... this
>                 is geo week and there are more than 70 mapathons
>                 planned for this week. Maybe updates from the tech
>                 team need to be posted across channels on this as it
>                 is critical infrastructure for all of these events and
>                 people might not be connected to slack / list /
>                 facebook etc...
>                 Cheers,
>                 Pete
>                 On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani
>                 <michealyani78 at gmail.com
>                 <mailto:michealyani78 at gmail.com>> wrote:
>                       Hello Tech team, we  just want to let you know
>                     that the tasking manager isn't stable yet, It has
>                     has worked for 2 hrs and now is on and off.   
>                     _______________________________________________
>                     HOT mailing list
>                     HOT at openstreetmap.org <mailto:HOT at openstreetmap.org>
>                     https://lists.openstreetmap.org/listinfo/hot
>                 _______________________________________________
>                 HOT mailing list
>                 HOT at openstreetmap.org <mailto:HOT at openstreetmap.org>
>                 https://lists.openstreetmap.org/listinfo/hot
>             _______________________________________________
>             HOT mailing list
>             HOT at openstreetmap.org <mailto:HOT at openstreetmap.org>
>             https://lists.openstreetmap.org/listinfo/hot
>         -- 
>         *Felix Delattre
>         *
>         Technical Project Manager
>         felix.delattre at hotosm.org <mailto:felix.delattre at hotosm.org>
>         --
>         *Humanitarian OpenStreetMap Team*
>         *Using OpenStreetMap for Humanitarian Response & Economic
>         Development*
>         web <http://hotosm.org/> | twitter
>         <https://twitter.com/hotosm> | facebook
>         <https://www.facebook.com/hotosm> | donate
>         <https://donate.hotosm.org/>
>         _______________________________________________
>         HOT mailing list
>         HOT at openstreetmap.org <mailto:HOT at openstreetmap.org>
>         https://lists.openstreetmap.org/listinfo/hot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/hot/attachments/20191113/ccab9b5c/attachment.html>

More information about the HOT mailing list