[OSM-dev] XML planets (was Re: New proposed directory layout for planet.openstreetmap.org)

[marqqs at gmx.eu]

Hello Roland,

thanks for your explanations, however, you might be wrong with the assumption a file based update would take much longer than a database based update. I wanted to get sure, therefore I just did some tests with an average 64-bit Linux computer.


Downloading "planet-120822.osm.pbf": about one hour
(using wget)

Updating the planet file on file basis (catching-up ca. 16 days): about one hour (including all downloads)
(using [[osmupdate]] program)


The update process can be accelerated by approximately 10 minutes if you supply the timestamp for the old planet file. Unfortunately, planet PBFs do not have file timestamps whereas XML planet files have (Why is that so??). Therefore you need to extract the XML planet's timestamp and to enter it manually as osmupdate parameter. Extracting the timestamp:
  wget planet.openstreetmap.org/planet-120822.osm.bz2 -O - |bzcat |head -2

Updating will even be faster if you are using .o5m instead of .pbf file format. And it will take much longer if you use .osm (XML) format.

In summary - in my opinion - it makes perfect sense to first update the planet file before you feed it into a database. Furthermore, there are several toolchains which are based on file updates only and running smoothly for years now.

Grüße
Markus


-------- Original-Nachricht --------
> Datum: Sat, 08 Sep 2012 13:35:49 +0200
> Von: Roland Olbricht <roland.olbricht at gmx.de>
> An: Brett Henderson <brett at bretth.com>, dev at openstreetmap.org
> Betreff: Re: [OSM-dev] XML planets (was Re: New proposed directory layout	for planet.openstreetmap.org)

> > If overall duration is an issue, is it worth patching a planet using
> > hourly/minutely diffs before importing the planet itself?
> 
> Ideas are welcome, but I'm quite sure this won't help. The primary issue
> is 
> developing time, not overall duration for this step.
> 
> For the patching itself: Reading 25 GB from disk and then writing it back 
> alone will take about 800 seconds, which equals 13 minutes, for a minute
> diff. 
> Thus, it will take for the assumed 2-16 days delay between a month and 8 
> month, even if Osmosis takes no computation time at all.
> 
> Of course, you can use daily diffs instead for complete days and
> hour/minute 
> diffs for the rest, but so you can do with Overpass API. It is just not
> worth 
> the hassle.
> 
> Ths is what the whole story is about:
> 
> There are projects whose main concern is to process a planet file. In such
> a 
> project, a lot of developing effort and increased system requirements are 
> justified a 25% speedup.
> 
> But there are also projects where a Planet file just needs to be read, in
> the 
> simplest possible way, because it has little impact on the project's 
> performance. For those, having a fancy compression which needs some extra 
> software to decompress it that may or may not work on the target platform,
> is 
> rather a nuisance than a "developers welcome" sign.
> 
> Cheers,
> 
> Roland