[OSM-dev] Optimal free compression algorithm for OSM XML data

[Jon Burgess]

On Thu, 2007-05-10 at 15:53 +0100, Nick Hill wrote:
> Hello Shaun
> 
> Thank you for the pointers for Mac users and 7-zip.
> 
> I have uploaded a copy of the current planet.osm as 7z, where I have further 
> increased compression using bigger dictionary etc.
> 
> planet files are at:
> http://planet.openstreetmap.org/
> 
> The URL for the current planet.osm in 7z format is:
> http://planet.openstreetmap.org/planet-070509.osm.7z
> 
> The new file is 183Mb vs Bzip2 235Mb.
> 
> You should find both downloading and unpacking the new file faster. Please try 
> it and give feedback.
> 

Works OK for me using osm2pgsql like this:

$ 7za -so x planet-070509.osm.7z | osm2pgsql /dev/stdin

I may be able to integrate a 7zip reader into the osm2pgsql input
routines so that the file can be read directly.

The CPU time to decompress .7z vs .bz2 is slightly better, though not by
nearly as much as your measurements. Perhaps the 4MB cache on my Core 2
Duo E6600 helps bzip2.

                 Time
               Unpack

bzip2              86
7-zip              72
gzip               25

I think the increased compression ratio and faster performance are
definitely good reasons to choose .7z

	Jon