[OSM-dev] planet.osm.org is verry slow

Simon Poole simon at poole.ch
Sat Jan 25 21:47:52 UTC 2020


Am 25.01.2020 um 22:19 schrieb marc marc:
> Le 25.01.20 à 21:43, Simon Poole a écrit :
>> Am 25.01.2020 um 21:12 schrieb marc marc:
>>> is there a convenient way to update a pbf to exactly the same diff as
>>> osm.org? and/or add the state.txt of the pbf produced on osm.org?
>>> this would allow to have the same pbf mirrored on other sites without
>>> having to download and/or encode the same thing several times
>> The answer is likely no, because I don't believe there is a guarantee
>> that the planet dump contains exactly the commits of a certain set of
>> diffs,
> the planet dump is done from the db itselft and not using by a cascade
> of diffs (min->hourly>daily>weekly) ?
> or maybe a special diff between planet dump.

Yes the dump works via a completely different mechanism.

The problem is actually fixable (not expecting a lot of enthusiasm for
it though), we could post creating the dump run an update from a
specific diff prior to when the dump started on the dump before
publishing. Doing this wouldn't guarantee that the binaries of a
parallel updated planet file are the same, but they would have the same
contents.

>> But that doesn't matter in a practical sense
> it avoids having different files between mirrors, which is quite
> annoying if a bug occurs on one variant and not on another depending
> on the "mirror".
See above.
>
>> In any case there is no argument to be made for the ~1'000 downloads 
>> that start shortly every week after the plant dump has been produced, 
> I agree.
> maybe a rate limit "1st full speed" or "full speed for mirror"

AFAIK that is already the case.

Simon

>> and of which for weird reasons 25% are actually
>> downloading the compressed XML file (which is 30% larger).
> there are probably some tools that don't support pbf (for example
> osmfilter (I don't mind too much, I do an osmconvert before, but I guess
> this extra step is not done and/or not available and/or not known by
> other people).
>
>> Simon
>>
>>> Le 25.01.20 à 18:34, marc tobias a écrit :
>>>> Wolfram, who asked about the bandwidth, is already running a mirror
>>>> https://download.bbbike.org/osm/planet/
>>>> (listed on https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading)
>>>> so downloading it every week is likely trying to update his mirror.
>>>>
>>>> marc tobias
>>>>
>>>>> Message: 2
>>>>> Date: Sat, 25 Jan 2020 11:46:20 +0100
>>>>> From: Simon Poole <simon at poole.ch>
>>>>> To: dev at openstreetmap.org
>>>>> Subject: Re: [OSM-dev] planet.osm.org is verry slow
>>>>> Message-ID: <4f5c3f2c-f3c9-0545-0dd2-98df390e4948 at poole.ch>
>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>
>>>>> Yes, there are now bandwidth restrictions on connections to
>>>>> planet.openstreetmap.org because the usage had skyrocketed and outgoing
>>>>> bandwidth was completely saturated.
>>>>>
>>>>> But the obvious question is: why are you even trying to download the
>>>>> full planet twice within a fortnight? Particularly given the file can
>>>>> easily be updated in situ.
>>>>>
>>>>> Simon
>>>> _______________________________________________
>>>> dev mailing list
>>>> dev at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/dev
>>>>
>>> _______________________________________________
>>> dev mailing list
>>> dev at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/dev
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20200125/c0b52680/attachment.sig>


More information about the dev mailing list