[OSM-dev] Minute Diffs Broken

Tom Hughes tom at compton.nu
Tue May 5 10:42:42 BST 2009


Brett Henderson wrote:

> If you're referring to multi-mastered clustered databases then that is a 
> whole different problem that shouldn't be confused with what I'm trying 
> to achieve.  I'm simply trying to provide a way for people to access 
> regular updates in a read-only fashion where data integrity is the 
> highest priority.  By allowing delays in delivery (I'd like to get it 
> down to a couple of minutes but I'm not aiming for anything like 
> real-time) it becomes a simpler problem with hopefully a simpler 
> solution.  Any multiple database system is likely to be a long way off 
> so I can't wait for that.

I made my comments because people seemed to be suggesting using one or 
other of the clustering solutions to provide some sort of change stream 
to you.

> 1. Are there any known issues with the current API that could cause 
> delays in excess of 5 minutes?  Or is it just a fact of life with large 
> changesets?  I guess what I'm asking is longer than 5 minutes a regular 
> occurrence with a large changeset or is something strange going on in 
> rare cases?

I have to say I don't fully understand what the issue you're seeing is 
as I only skimmed the vast amount of discussion that went on overnight.

Are you really seeing a five minute delay? Or just a delay that happens 
to be over a five minute boundary?

In other words are you saying that an edit timestamped at 17:50:00 is 
still not visible in the database at 17:55:00? Or just that you look at 
the database at 17:55:00 and don't see an edit which is timestamped at 
17:54:50 which then appears when you look again at 18:00:00?

It's certainly possible that a very large changeset might take some 
minutes to process I guess - if you can give me a specific changeset 
number from the last few days then I can have a look and see how long it 
took.

> 2. What appear to be the current system bottlenecks?  Is the database 
> already approaching processing capacity or is rails the limiting factor?

Nothing specific in terms of the database - that largely seems to be 
fine. The issues are, as always, mostly about rails and the amount of 
memory it uses and getting the daemons to restart cleanly when they get 
too big.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/




More information about the dev mailing list