[OSM-dev] Storing way/relation start offset in PBF file

[Jochen Topf]

On Tue, Nov 30, 2010 at 09:48:37PM -0600, Scott Crosby wrote:
> > It has
> > more to do with the file serialization than with the file contents.  Its just
> > something that concerns the tasks reading and writing the pbf.  It has to
> > remember those offsets and add them to the end of the file. Or, as you suggest,
> > add those counts when writing out the data. Thats independent of everything
> > else happening in Osmosis.
> 
> I considered this option, but I couldn't come up with a workable
> design. First, PBF files are designed to be concatenable, a property
> that I want to retain. The problem is if I write some sort of trailing
> block to the file containing extra indexing information, such as
> offsets, How do I determine the length of that block? A reader needs
> to know that in order to seek to the right place to read it. If you
> could find a non-gross way to make this work, I'd appreciate it.

The non-grossest thing I can think of is putting the length of the last block
in some place in the header of the file. Well, actually thats still quite
gross. :-) But it would work and still be flexible.

I think its probably best we'll let this rest for the moment. We should give
the PBF format some time to be implemented in more places and see where all
the problems are and what ideas we have for improvements. Than later when
we have more practical experience with the format we can think about a version 2.

Jochen
-- 
Jochen Topf  jochen at remote.org  http://www.remote.org/jochen/  +49-721-388298