[OSM-dev] XML planets (was Re: New proposed directory layout for planet.openstreetmap.org)

[Brett Henderson]

On 7 September 2012 17:28, Roland Olbricht <roland.olbricht at gmx.de> wrote:

> The planet file is necessary for the first startup. Afterwards, it can
> work forever solely on minute diffs. And new instances can be cloned from
> the exisiting instance over a public interface, without a planet file.
>
> This initial startup has usually the following timing:
>
> 2 hours to download planet.osm.bz2
>
> 12-24 hours to import planet.osm.bz2 into the database. This contains
> various optimizations that are fine tuned on the properties of the XML
> planet structure and maybe gets worse with a differently organised file
> format.
>
> 4-30 hours to catch up by applying the minute diffs. This depends on how
> many days the planet file is old (always at least two days, one for the
> planet file itself, a second because we have done the above import
> procedure).
>
> Altogether, there is not much point in "just improving the download time",
> because it has a diminishing share of the total time needed. It would be by
> far more important to get the import step again fast.
>
> I doubt that converting the PBF into a XML is done in the saved half an
> hour on the download, so the simplest solution to convert after download is
> surely a slowdown.
>

If overall duration is an issue, is it worth patching a planet using
hourly/minutely diffs before importing the planet itself?  That way you can
begin importing a planet file that is no more than a few hours old rather
than using a freshly downloaded file that is already old due to the time
taken to create a planet.  If you do this you'll have less diffs to apply
to your database to catch up after completing the import.

You could convert a PBF planet to XML file as part of the same patching
step.  You could stream the XML output directly into your import job saving
more time.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20120908/67bb764e/attachment.html>