[osmosis-dev] Invalid actions during replication

[Brett Henderson]

Hi Paweł,

On 11 November 2012 21:24, Paweł Paprota <ppawel at fastmail.fm> wrote:

> Hi all,
>
> On Friday I have implemented a simple validation in the OWL plugin that
> checks whether given entity action (create/modify/delete) can be applied to
> the current database state. See implementation: [1].
>
> By "invalid action" I mean a situation when there is a "delete" action in
> the pipeline but there is no entity in the database to delete. Or "modify"
> action with version X when there is no version X - 1 in the database but
> some other version.
>
> I've been running replication for over a day now (in LOG mode, on a
> database with planet imported) and results are quite interesting. There is
> more invalid actions reported than I thought there would be. Some random
> examples:
>
> INFO: Invalid action: Node 1342627235 - Trying to modify entity that does
> not exist!
> INFO: Invalid action: Node 964649444 - Trying to modify entity that does
> not exist!
> INFO: Invalid action: Way 11133255 - Trying to modify entity that does not
> exist!
> INFO: Invalid action: Way 9105159 - Trying to modify entity that does not
> exist!
>
> I have not yet analyzed closely those entities that are reported but in
> general there should be no invalid actions at all on a full planet database
> with correct replication setup, am I correct in this assumption?
>

It's possible that there could be errors during initial replication, but I
would have thought they'd disappear eventually.  It seems a bit odd.

Unfortunately you will need to look into some examples one at a time to try
to trace the source of the issue.  It can be tedious to get to the bottom
of replication problems.  The pgsnapshot replication functionality has been
around for a long time now and has been fairly reliable, but perhaps
there's an issue in there somewhere.


>
> Also, if anyone's interested, this functionality would be trivial to port
> over to the pgsnapshot Osmosis plugin (--write-pgsql-change) since the OWL
> plugin is built from that plugin. I can do this if you think it could be
> useful.
>
> [1] https://github.com/ppawel/**osmosis/blob/owldb-plugin/**
> owldb/src/main/java/org/**openstreetmap/osmosis/owldb/**
> v0_6/impl/ChangeWriter.java#**L256<https://github.com/ppawel/osmosis/blob/owldb-plugin/owldb/src/main/java/org/openstreetmap/osmosis/owldb/v0_6/impl/ChangeWriter.java#L256>
>
> Implementation notes: basically there is a "invalidActionsMode" paramater
> for the --write-owldb-change task (which is derived from
> --write-pgsql-change but with OWL-specific additions) that can be IGNORE
> (does nothing), LOG (logs information about the action) or BREAK (stops
> processing when invalid action occurs).
>

I don't mind something like this being added, but I'd like to understand
the source of your current issues first.  If it turns out that it's normal
behaviour and can't be fixed then there's not much point raising warnings.
On the other hand, if there is a real issue here then it will be valuable.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20121114/db73ae79/attachment.html>