[HOT] Fwd: OSM Upload slowness - what's going on?

Blake Girardot bgirardot at gmail.com
Fri May 13 13:29:12 UTC 2016

I thought this might be of interest to everyone.


---------- Forwarded message ----------
From: Grant Slater <openstreetmap at firefishy.com>
Date: Fri, May 13, 2016 at 2:59 PM
Subject: Re: [OSM-talk] Upload slowness - what's going on?
To: Tim Waters <chippy2005 at gmail.com>
Cc: "osmf-talk at openstreetmap.org" <talk at openstreetmap.org>

Hi All,

On Monday 9th May 2016 the master OSM database server was moved to
York (Bytemark) from London (Imperial).
This was to avoid multiple upcoming weekends of planned power testing
& maintenance at the Imperial data centre. For the last few years
Imperial has housed all our main critical systems including master &
slave DB servers and frontend & backend web/api servers. We also added
4 new frontend/backend web/api server to York on Monday.

We now have the master database server in York and the secondary
database server in Imperial. We also have a warm standby slave db in
AWS Ireland. A fourth SSD (NVMe) based DB server was delivered
yesterday (Thursday), but it needs testing (burn-in, reliability,
performance etc) before we can start using it. Slave DB servers can be
promoted to master if required.

The slave db servers serve Web/API read traffic and writes go to the
master. When the frontend + backend servers were in the same data
centre as the master db server the latency was <1ms. We now run a VPN
to connect the servers up and the latency is ~8ms Imperial to
Bytemark. Currently we are using the frontend & backends server at
Imperial (closest to slave db read server) and sending writes over the
VPN to Bytemark. The extra 8ms roundtrip is triggered multiple times
based on the size of the upload changeset, this is the root cause for
the slower uploads. The link between Imperial & Bytemark can handle
gigabit speeds. Over the last few days we've been tweaking the VPN
settings to get optimal latency & throughput over the links.

Over today (for at least the weekend) we are switching to the new
frontend & backend servers in York (Bytemark). London Imperial will be
offline from approximately 5pm (GMT+1) for the first weekend of power

In summary: The slow uploads are a known issue and we'll fix as soon
as practical. Our main concern has been setting up multiple data
center redundancy to avoid extended downtime.

Here is the list of all core hardware and hosting locations:

Hope that answers the questions. ;-)

Photos or it didn't happen:
* Syncing & powering down before we start London -> York DB move:
* Staged photo of racking up the master DB server at Bytemark:
* Testing the new Frontend / Backend servers a week ago:

Bytemark are a fantastic hosting company and their ongoing support of
the OpenStreetMap project is highly commendable. Please support them
;-) https://twitter.com/bytemark/status/729698435339853824

Kind regards,

Part of the OSM Ops team.

More information about the HOT mailing list