[OSM-dev] Minute Diffs Broken

Brett Henderson brett at bretth.com
Tue May 5 05:01:20 BST 2009


Steve Singer wrote:
> On Tue, 5 May 2009, Brett Henderson wrote:
>
>> That does look interesting.  I'd hope to use that outside the main 
>> database though.  My thoughts were to use triggers to populate short 
>> term flag tables which a single threaded process would read, use as 
>> keys to select modified data into an offline database, then clear.  
>> This offline database could then use a queueing system such as PgQ (I 
>> haven't seen it before, will have to check it out) to send events to 
>> the various consumers of the data.  I'd like to minimise access to 
>> the central database if possible because 1. it will scale better, and 
>> 2. it adds less burden to existing DBAs.
>
> I agree you'd only want one process pulling data from the central 
> database and then let other clients pull from another machine.  You'd 
> have to examine how different your trigger + scanning process code 
> will be from using PgQ with 1 consumer that then stores the data in 
> another db for publishing.  You should at least look to see what 
> problems they solved.
I'll take a look.  You're right, I should avoid poorly inventing 
something that others have already done a better job of :-)  I'd hate to 
impose a bottleneck on the entire app.
>
> One concern with trigger based systems is that for each real INSERT 
> your doing a second insert into a queue or journal table, but there 
> might not be a way around that.
I suspect not.  So would the main limitation be IO?  It should be 
possible to separate the synchronisation data (whatever that is) into a 
separate tablespace on other disk(s) if it becomes a problem.  It's 
access patterns should be sequential, with regular reads the dataset 
size should remain comparatively small, and with fixed size records it 
shouldn't fragment.  Hopefully it's not a major issue.

Brett





More information about the dev mailing list