[OSM-dev] planet.osm.org is verry slow

[marc marc]

Le 25.01.20 à 21:43, Simon Poole a écrit :
> 
> Am 25.01.2020 um 21:12 schrieb marc marc:
>> is there a convenient way to update a pbf to exactly the same diff as
>> osm.org? and/or add the state.txt of the pbf produced on osm.org?
>> this would allow to have the same pbf mirrored on other sites without
>> having to download and/or encode the same thing several times
> 
> The answer is likely no, because I don't believe there is a guarantee
> that the planet dump contains exactly the commits of a certain set of
> diffs,

the planet dump is done from the db itselft and not using by a cascade
of diffs (min->hourly>daily>weekly) ?
or maybe a special diff between planet dump.

> But that doesn't matter in a practical sense

it avoids having different files between mirrors, which is quite
annoying if a bug occurs on one variant and not on another depending
on the "mirror".

> In any case there is no argument to be made for the ~1'000 downloads 
> that start shortly every week after the plant dump has been produced, 

I agree.
maybe a rate limit "1st full speed" or "full speed for mirror"

> and of which for weird reasons 25% are actually
> downloading the compressed XML file (which is 30% larger).

there are probably some tools that don't support pbf (for example
osmfilter (I don't mind too much, I do an osmconvert before, but I guess
this extra step is not done and/or not available and/or not known by
other people).

> 
> Simon
> 
>>
>> Le 25.01.20 à 18:34, marc tobias a écrit :
>>> Wolfram, who asked about the bandwidth, is already running a mirror
>>> https://download.bbbike.org/osm/planet/
>>> (listed on https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading)
>>> so downloading it every week is likely trying to update his mirror.
>>>
>>> marc tobias
>>>
>>>> Message: 2
>>>> Date: Sat, 25 Jan 2020 11:46:20 +0100
>>>> From: Simon Poole <simon at poole.ch>
>>>> To: dev at openstreetmap.org
>>>> Subject: Re: [OSM-dev] planet.osm.org is verry slow
>>>> Message-ID: <4f5c3f2c-f3c9-0545-0dd2-98df390e4948 at poole.ch>
>>>> Content-Type: text/plain; charset="utf-8"
>>>>
>>>> Yes, there are now bandwidth restrictions on connections to
>>>> planet.openstreetmap.org because the usage had skyrocketed and outgoing
>>>> bandwidth was completely saturated.
>>>>
>>>> But the obvious question is: why are you even trying to download the
>>>> full planet twice within a fortnight? Particularly given the file can
>>>> easily be updated in situ.
>>>>
>>>> Simon
>>> _______________________________________________
>>> dev mailing list
>>> dev at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/dev
>>>
>> _______________________________________________
>> dev mailing list
>> dev at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
>