[osmosis-dev] Wrapper scripts for streaming replication

[Brett Henderson]

Hi Paul,

On 18 April 2013 17:36, Paul Norman <penorman at mac.com> wrote:

> I'm considering making use of streaming replication to feed change data to
> a
> process, but I'm not quite sure how to script it to handle interruptions
> like power outages and such correctly without getting in advance of itself.
> Does anyone have any wrapper scripts that would help with this?
>
> I'd rather not reinvent the wheel so I can focus on the logic of
> interpreting the osc data and putting it into my database.
>
> Alternately, I could re-implement a streaming replication reader in python,
> which is what I intend to be writing my osc to DB code in.
>

It'd be great to see the replication streaming getting some usage.  It
doesn't seem to be used at the moment.  In fact it's been down for a couple
of weeks and it's trying to catch up again now.

I assume you have a custom database schema that you're writing to.  In
other words, I assume you can't use existing Osmosis tasks to write to your
database.

There are a couple of example Osmosis command lines on the following wiki
page in case you haven't already seen them.
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming

If you're writing your own database writer, you'll probably have to use
Osmosis to write to standard out, and then pipe data into standard input in
your program.  The problem you'll face with this approach is that Osmosis
will assume that written to standard output has been successfully processed
and will update its local state.txt accordingly.  It has no way of knowing
when the downstream process has finished processing the data and has
committed it.  If a crash occurs between Osmosis updating the state.txt
file and your application committing changes to the database, those changes
will be lost.  I don't think it's possible to solve this using standard
piping between processes, it would require some two way communication.  It
may be simpler to implement a new Osmosis plugin for your database, or to
re-implement the client streaming functionality.

Let me know if the above doesn't make sense :-)

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20130419/62ecdce3/attachment.html>