[OSM-dev] Minute Diffs Broken

[Steve Singer]

On Tue, 5 May 2009, Brett Henderson wrote:

> That does look interesting.  I'd hope to use that outside the main database 
> though.  My thoughts were to use triggers to populate short term flag tables 
> which a single threaded process would read, use as keys to select modified 
> data into an offline database, then clear.  This offline database could then 
> use a queueing system such as PgQ (I haven't seen it before, will have to 
> check it out) to send events to the various consumers of the data.  I'd like 
> to minimise access to the central database if possible because 1. it will 
> scale better, and 2. it adds less burden to existing DBAs.

I agree you'd only want one process pulling data from the central database 
and then let other clients pull from another machine.  You'd have to examine 
how different your trigger + scanning process code will be from using PgQ 
with 1 consumer that then stores the data in another db for publishing.  You 
should at least look to see what problems they solved.

One concern with trigger based systems is that for each real INSERT your 
doing a second insert into a queue or journal table, but there might not be 
a way around that.

>
> Brett
>
>

Steve