[OSM-dev] planet.osm.org is verry slow
openstreetmap at firefishy.com
Sat Jan 25 15:45:20 UTC 2020
Hi Wolfram and others,
On Sat, 25 Jan 2020 at 10:32, Wolfram Schneider <wosch at freebsd.org> wrote:
> I wanted to download the latest planet from planet.osm.org and the
> download rate is less than 450KByte/s. In the past I got
> 30-60Mbyte/s. It takes now more than 36 hours to download the planet
> (was 15min). All mirrors are outdated, you have to wait until Sat
> morning to see a new planet.osm.pbf file.
I am part of the OpenStreetMap Operations team and Operations Working Group.
I am the person who temporarily instigated the download rate limit.
www.openstreetmap.org + API, planet and many other services primarily
run in our Amsterdam data center (Equinix AM6).
See servers here: https://hardware.openstreetmap.org/#equinix-am6
Our uplink from these servers to the internet is a 1Gbps link to Cogent.
Due to recent increases in outbound bandwidth usage (partially due to
switch to BBR congestion control
we started to exceed 1Gbps... Once we exceed the limit, packets get
We initially did not understand what was causing the packets to be
dropped. We suspected a flaw with switches and/or GBIC modules and/or
The dropped packet caused DNS failures and reliability issues with the
API, which were reported in many channels: email, Slack osm-us, irc
The planet downloads are the main reason the upstream link is being
swamped. 60Mbyte/s alone is nearly 50% of our available upstream
capacity. Normally around 30 to 60 simultaneous requests.
I've started to prep for upgrading our upstream link to 10Gbps. But
we've had ongoings with Cogent peering and would prefer to switch to
using a new provider upstream.
The 20GB+ planet .osm.bz2 files are automatically redirected to the
But unfortunately due to flaw in my mirror update script the redirect
was now working for PBF files. Pull requests welcome:
I have manually fixed the PBF file mirroring now (for current files).
Once we have fixed the upstream link issue, I will remove the download
For the moment, please use a mirror for full speed downloads.
The minutely, hourly and daily diff files will only take a second or 2
extra to download with the current late limits.
More information about the dev