[OSM-dev] New OSM binary fileformat implementation.

Scott Crosby scrosby at cs.rice.edu
Fri Sep 17 14:58:10 BST 2010


On Fri, Sep 17, 2010 at 1:33 AM, Frederik Ramm <frederik at remote.org> wrote:
> Mike,
>
> jamesmikedupont at googlemail.com wrote:
>>
>> Hi, ist there any documentation of the binary format changes?
>> I have implemented a c++ reader using protobuf, would update that if
>> there is a new format spec.

The core of the design is the same, but there are some subtle
differences (lat_offset, lon_offset, date_granularity,
required_features) that, if not supported, *will* silently corrupt
data, and more enhancements  (DenseInfo, keys_vals) for better
compression that will very be obvious if missing.

Also, the protobuf spec file has completely different field numbers;
any generated code must be regenerated, and will be unable to read the
older files.

>
> It would be great if you could check whether your reader still works with
> the current implementation, and then I'd be extremely grateful for some sort
> of minimal package that contains only your reader and the stuff absolutely
> necessary to build it - I've checked out your
> http://github.com/h4ck3rm1k3/OSM-Osmosis but ended up with a tree that
> contained half (or al?) of Osmosis and lots of autoconf cruft but wasn't
> buildable for me because it expected Google protobuf stuff to be downloaded
> and installed separately and I didn't know what to get and where to install
> it!

That code pre-dates when I split the binary format into a separate
package. Common, application-independent code for reading the format
belongs in http://github.com/scrosby/OSM-binary. This also contains
the canonical protocol buffer spec files (*.proto) for the format.

My suggestion is to extract the C++ code sitting in
http://github.com/h4ck3rm1k3/OSM-Osmosis and add the
application-independent code (such as for reading/decompressing
fileblocks) to http://github.com/scrosby/OSM-binary. If you want to
commit to a branch in that repo, I'll add you as a collaborator,

>
> Background is, I would like to add binary format support to osm2pgsql and
> was hoping to be able to use your code for that.
>
> Scott, it would be great if apart from a name for the new binary format
> you'd also recommend a default file extension since some of our tools try to
> auto-detect the file format from the name (.osm.bz2, .osm.gz, .osm - maybe
> .osm.bin for the new stuff?).

I've followed advice and chosen 'Protocolbuffer Binary Format' under
the extension  *.osm.pbf

Scott



More information about the dev mailing list