[OSM-dev] Minute Diffs Broken
brett at bretth.com
Tue May 5 05:01:20 BST 2009
Steve Singer wrote:
> On Tue, 5 May 2009, Brett Henderson wrote:
>> That does look interesting. I'd hope to use that outside the main
>> database though. My thoughts were to use triggers to populate short
>> term flag tables which a single threaded process would read, use as
>> keys to select modified data into an offline database, then clear.
>> This offline database could then use a queueing system such as PgQ (I
>> haven't seen it before, will have to check it out) to send events to
>> the various consumers of the data. I'd like to minimise access to
>> the central database if possible because 1. it will scale better, and
>> 2. it adds less burden to existing DBAs.
> I agree you'd only want one process pulling data from the central
> database and then let other clients pull from another machine. You'd
> have to examine how different your trigger + scanning process code
> will be from using PgQ with 1 consumer that then stores the data in
> another db for publishing. You should at least look to see what
> problems they solved.
I'll take a look. You're right, I should avoid poorly inventing
something that others have already done a better job of :-) I'd hate to
impose a bottleneck on the entire app.
> One concern with trigger based systems is that for each real INSERT
> your doing a second insert into a queue or journal table, but there
> might not be a way around that.
I suspect not. So would the main limitation be IO? It should be
possible to separate the synchronisation data (whatever that is) into a
separate tablespace on other disk(s) if it becomes a problem. It's
access patterns should be sequential, with regular reads the dataset
size should remain comparatively small, and with fixed size records it
shouldn't fragment. Hopefully it's not a major issue.
More information about the dev