[osmosis-dev] Using PBF for temp files

[Brett Henderson]

Hi Frederik,

On Mon, Jul 4, 2011 at 6:44 AM, Frederik Ramm <frederik at remote.org> wrote:

> Hi,
>
>   when Osmosis writes temporary files, using the SimpleObjectStore, it uses
> a relatively primitive serialisation format and runs gzip on that. Given
> that we now have rather fast and effective OSM data storage with the PBF
> reader and writer, I wonder what it would take to switch from writing
> classic temp files to using (probably uncompressed) PBF as the temporary
> storage format.
>
> I've rummaged around in the code for a bit but at least it wasn't
> straightforward to do since the temporary storage code uses "containers" for
> everything whereas the PBF code doesn't.
>
> Do you reckon it would be easy to do, and can somebody (Brett? Scott?)
> outline what you think would have to be done? Then I'll have a go at
> implementing that.
>

I'm not going to add much value here :-)

I suspect it will be challenging because the serialisation logic currently
used is implemented within the classes actually being persisted.  For
example, classes like SimpleObjectStore have no knowledge of the data types
being persisted and simply get passed instances of Storeable.  You'll have
to create some Entity stream specific storage classes.  You'll be limited to
persisting Entity streams, but perhaps that is good enough to improve
performance in the majority of cases.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20110704/2f1a38a8/attachment.html>