[OSM-dev] Merging Changes into Planet

[Andrew Ayre]

On 7/14/2011 9:26 AM, Jochen Topf wrote:
> Its almost certainly CPU. Try
> dd if=planet-latest.osm.pbf of=copy.osm.pbf bs=1M
> and you'll see how fast the files can be copied. Thats the time needed for the
> disk. Everything else is CPU.
> 
>> Does this merge time seem right/reasonable? Are there any approaches or
>> tricks I am missing that can speed it up?
> 
> Try larger buffers:
> $OSMOSIS --read-xml-change $OSCFILE --read-bin $BASEDIR/var/current-planet.osm.pbf --buffer bufferCapacity=12000 --apply-change --buffer bufferCapacity=12000 --write-pbf $NEWPLANET
> 
> This takes about an hour every day on a 800MHz machine.
> 
> Jochen

Thanks everyone for the suggestions. I am using osmosis to generate a
single changes.osc file already. The server is in a server farm so I
don't have control over the disks.

The dd command took 6 minutes 40 seconds so that proves Jochen right.

With my 766MB change file the original merge command that I posted takes
106 minutes.

By using Jochen's merge command with buffers and the same change file it
now takes 58 minutes!

I knew about the possibility of buffers but I never tried them because
[1] appeared quite negative regarding them. Maybe it is out of date?

Thanks, Andy

[1] http://wiki.openstreetmap.org/wiki/Osmosis/Benchmarking

-- 
Andy
PGP Key ID: 0xDC1B5864