[OSM-dev] planet.osm.org is verry slow

Grant Slater openstreetmap at firefishy.com
Sat Jan 25 15:45:20 UTC 2020


Hi Wolfram and others,

On Sat, 25 Jan 2020 at 10:32, Wolfram Schneider <wosch at freebsd.org> wrote:
>
> I wanted to download the latest planet from planet.osm.org and the
> download rate is  less than 450KByte/s. In the past I got
> 30-60Mbyte/s. It takes now more than 36 hours to download the planet
> (was 15min). All mirrors are outdated, you have to wait until Sat
> morning to see a new planet.osm.pbf file.
>

I am part of the OpenStreetMap Operations team and Operations Working Group.

I am the person who temporarily instigated the download rate limit.
https://github.com/openstreetmap/chef/commit/1647548590878bfd153b8b8c74e73eda0b8a2cd3

Some background...

www.openstreetmap.org + API, planet and many other services primarily
run in our Amsterdam data center (Equinix AM6).
See servers here: https://hardware.openstreetmap.org/#equinix-am6
Our uplink from these servers to the internet is a 1Gbps link to Cogent.
Bandwidth usage:
https://munin.osm.org/openstreetmap.org/switch1.openstreetmap.org/if_bytes/if_49.html

Due to recent increases in outbound bandwidth usage (partially due to
switch to BBR congestion control
https://github.com/openstreetmap/chef/commit/67c05c6c4b484a6cdac9473b4e6951e5c76c42a3),
 we started to exceed 1Gbps... Once we exceed the limit, packets get
dropped.
We initially did not understand what was causing the packets to be
dropped. We suspected a flaw with switches and/or GBIC modules and/or
fibre link.

The dropped packet caused DNS failures and reliability issues with the
API, which were reported in many channels: email, Slack osm-us, irc
etc.

The planet downloads are the main reason the upstream link is being
swamped. 60Mbyte/s alone is nearly 50% of our available upstream
capacity. Normally around 30 to 60 simultaneous requests.

I've started to prep for upgrading our upstream link to 10Gbps. But
we've had ongoings with Cogent peering and would prefer to switch to
using a new provider upstream.
https://twitter.com/OSM_Tech/status/1220276733359919104

The 20GB+ planet .osm.bz2 files are automatically redirected to the
https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/
mirror
But unfortunately due to flaw in my  mirror update script the redirect
was now working for PBF files. Pull requests welcome:
https://github.com/openstreetmap/chef/blob/master/cookbooks/planet/templates/default/planet-mirror-redirect-update.erb
I have manually fixed the PBF file mirroring now (for current files).

Once we have fixed the upstream link issue, I will remove the download
rate limits.
For the moment, please use a mirror for full speed downloads.

The minutely, hourly and daily diff files will only take a second or 2
extra to download with the current late limits.

Kind regards,

Grant



More information about the dev mailing list