[OSM-dev] 0.6 Replication (was: Number of Edits)

[Brett Henderson]

Andy Allan wrote:
> On Wed, Sep 24, 2008 at 11:26 AM, Andy Allan <gravitystorm at gmail.com> wrote:
>   
>> On Wed, Sep 24, 2008 at 9:27 AM, Richard Fairhurst <richard at systemed.net> wrote:
>>     
>>> Brett Henderson wrote:
>>>
>>>       
>>>> Would this mean you'd update the same way multiple times within a
>>>> single revision?  I'm nervous about what this means for
>>>> replication.  How does osmosis know when the way is complete and
>>>> ready to be replicated?  Sorry, probably off topic here but just
>>>> keep it in mind.
>>>>         
>>> Not quite sure what you mean. 0.6 transactions will just be (AIUI)
>>> "groups" of edits, nothing more.
>>>       
Perhaps I completely misunderstood you.  I was worried that you might do 
something like this:
1. way 12345 is currently at version 1.
2. Via potlatch you edit way 12345 adding a new tag and create version 2.
3a. Via potlatch you immediately add a second tag and update version 2 
to include the extra tag.
3b. Via potlatch you immediately add a second tag and create version 3 
including the extra tag.

I'm not sure whether 3a or 3b is the expected Potlatch behaviour.  3b 
will allow replication, 3a will cause problems because I may replicate 
version 2 prior to the second tag being added.

As an aside, currently I'm not expecting osmosis to even look at the 
changeset table although I haven't looked at this in enough detail yet.  
Originally I was expecting changeset to be atomic or at least 
optimistically locked which would have let me replicate at the changeset 
level.  Given that the api will allow gradual editing within a changeset 
I don't think that's possible.
>> Just a clarification for me - should we keep referring to these as
>> "changesets" and perhaps start the mantra of "a changeset is not
>> transactional"? I think otherwise the database guys amongst us will
>> get confused...
>>     
I'm happy with whatever label is applied, so long as I understand how it 
works :-)
>
> On reading the wiki, maybe "a changeset is not atomic" is a better
> description? Is there a difference in db-esque parlance between
> "atomic" and "transactional"?
>   
Dunno, they can mean many different things.  We could get into an ACID 
(Atomic Consistent Idompotent Durable ... I think) discussion but many 
of those can be achieved without relying on database transaction support.

It might make more sense to discuss the changesets implementation.  Here 
are several ways I could see them being implemented:
1. A "changeset" is applied within a single database transaction.  
Timestamps don't need to be written to entities (ie. 
nodes/ways/relations) because the changeset timestamp is identical due 
to everything being committed at the same time.  Requires all clients to 
upload in batches or requires a changeset to be created for every 
individual entity update.  Very simple to support replication at the 
changeset level.
2. All edits within a "changeset" are created individually but using 
optimistic locking.  If the changeset remains open, all impacted 
entities remain locked until the changeset is closed.  Timestamps could 
still be located at the changeset level.  Replication can be
 still be supported at the changeset level.  This schema provides 
changeset atomicity but not using db transaction support and at the risk 
of locking entities for a long time during edits.
3. All edits within a "changeset" are created individually without any 
form of locking other than version checking.  This means that two 
changesets being performed at the same time can perform overlapping 
edits on entities.  This means that replication of changesets cannot be 
performed at the changeset level.  If changeset information is required 
in slave databases the changeset records themselves will have to be 
replicated separately to the entities within them.

I suspect we're using scheme 3.  It's the easiest to implement for 
editors and avoids all editor locking issues.  But I'm not sure if 
changeset replication will be possible.

Apologies if none of the above makes sense ... I need to get the 0.6 
database schema and API up and running locally and have a play.

Brett