[OSM-dev] Changefile Downloads Stalling

Sat Nov 15 00:55:40 UTC 2014

Thanks very much for looking into this Toby.  I’ve been attempting to run a trace from my company’s server to planet.osm.org using tracepath and traceroute, but I’m getting “no reply” or similar at every hop.  I’m thinking this may have to do with our network settings as I’m getting the same results with any address that I try, so I’ve inquired about this and will post the trace path if I’m able to get addresses for the hops.

Beyond that do you (or others) have recommendations for next steps?  Should I look to determine which hop is slowing things down and attempt to contact them?

From: Toby Murray [mailto:toby.murray at gmail.com]
Sent: Thursday, November 13, 2014 8:10 AM
Cc: dev at openstreetmap.org
Subject: Re: [OSM-dev] Changefile Downloads Stalling

I was one of the people on IRC yesterday and can confirm that I am seeing similar problems when downloading files from planet.osm.org<http://planet.osm.org> while at work (university campus) however when downloading to a VM I have with cloudatcost (I think they are in Canada), I am not having any problems.

I just tried to download the latest note dump file. The first time it stalled out at 2.6 MB. A second attempt stalled at 1.something then picked back up to 3 MB. Now it is trickling in at less than 70 KB/sec.

Since this looks like something happening along the way and not at either end, here is a trace from me to planet.osm.org<http://planet.osm.org>:

 1?: [LOCALHOST]                                         pmtu 1500
 1:  no reply
 2:  no reply
 3:  kanren-ksu-pplant-border.peer.net.kanren.net<http://kanren-ksu-pplant-border.peer.net.kanren.net>          2.479ms
 4:  bb-ksu-hale-e2-1.bb.net.kanren.net<http://bb-ksu-hale-e2-1.bb.net.kanren.net>                    3.196ms
 5:  bb-kc-walnut-e2-4.bb.net.kanren.net<http://bb-kc-walnut-e2-4.bb.net.kanren.net>                   6.768ms
 6:  et-10-0-0.1050.rtr.kans.net.internet2.edu<http://et-10-0-0.1050.rtr.kans.net.internet2.edu>             5.911ms
 7:  et-9-0-0.106.rtr.chic.net.internet2.edu<http://et-9-0-0.106.rtr.chic.net.internet2.edu>              17.584ms
 8:  et-9-0-0.115.rtr.wash.net.internet2.edu<http://et-9-0-0.115.rtr.wash.net.internet2.edu>              35.037ms
 9:  abilene-wash.mx1.fra.de.geant.net<http://abilene-wash.mx1.fra.de.geant.net>                   127.550ms
10:  ae1.mx1.ams.nl.geant.net<http://ae1.mx1.ams.nl.geant.net>                            133.560ms
11:  ae2.mx1.lon.uk.geant.net<http://ae2.mx1.lon.uk.geant.net>                            155.513ms
12:  janet-gw.mx1.lon.uk.geant.net<http://janet-gw.mx1.lon.uk.geant.net>                       158.483ms
13:  ae29.londpg-sbr1.ja.net<http://ae29.londpg-sbr1.ja.net>                             169.894ms
14:  be24.londic-rbr1.ja.net<http://be24.londic-rbr1.ja.net>                             157.260ms
15:  imperial-college.ja.net<http://imperial-college.ja.net>                             158.031ms
16:  me-rach.net.ic.ac.uk<http://me-rach.net.ic.ac.uk>                                142.874ms
17:  no reply
18:  no reply

Toby

On Tue, Nov 11, 2014 at 7:31 PM, Humphries, Grant <HumphriG at trimet.org<mailto:HumphriG at trimet.org>> wrote:
Hi All,

I have been trying intermittently for a few weeks now to use osmupdate to refresh a local OSM extract.  The problem that I am having relates to the download of changefiles from planet.openstreetmap.org<http://planet.openstreetmap.org>.  Ultimately I want to update my extract once per day, so most of the changefiles that need to be downloaded and applied are of the hourly variety, such as this one: http://planet.openstreetmap.org/replication/hour/000/018/973.osc.gz

These files are generally 2 to 5 MB in size and the problem that I’m having is that my download will stall, sometimes for 20 to 30 minutes, sometimes up to a couple of hours, on one or more of these small changefiles virtually every time I attempt to use osmupdate.  The odd thing is that changefiles that are downloaded back-to-back, and that are roughly the same size can take vastly different amounts of time to complete.  For instance today osmupdate started with changefile 18975 and 18974 which took seconds, but then 18973 took around 20+ minutes.  I’m running this tool on a linux cent os server, and osmupdate uses wget to download the files.  I’ve tried using wget and curl on the same files outside of osmupdate and the problem persists.  I’ve also tried downloading the files through Firefox on a windows PC and had similar problems.  I’ve asked others on IRC to attempt the same downloads with varying results, some experience a similar lag, and some are able to download the same files I’m having trouble with in seconds.

I was just going to say that this file in particular has given me trouble https://planet.openstreetmap.org/replication/hour/000/018/823.osc.gz as I have tried to download it many times and it has always stalled and taken an extremely long time to complete, but I just tried it one more time and it took 8.8 seconds.  So is this kind of variance to be expected?

Thanks in advance for any help

Best,

-Grant

_______________________________________________
dev mailing list
dev at openstreetmap.org<mailto:dev at openstreetmap.org>
https://lists.openstreetmap.org/listinfo/dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20141115/365669b1/attachment.html>