[OSM-dev] Minute Diffs Broken

Brett Henderson brett at bretth.com
Tue May 5 23:43:12 BST 2009


Matt Amos wrote:
> the XML document is parsed incrementally to save memory, rather than
> for its behaviour, but it appears that rails, lighty and fastcgi all
> support streaming input. i am unsure if they all work together, but
> the rails docs suggest that it does.
>
> streaming input isn't nearly as difficult as streaming output. since
> rails treats (most) uncaught exceptions as 500 errors, but the error
> status is in the header, rails takes the safe option of having the
> full response in a buffer before it returns the status header.
>   
Is this a good idea?  I'm not comfortable with the idea of streaming 
input.  At least with streaming output you are performing a read-only 
operation (I think) and therefore shouldn't be holding any important 
locks.  Unless you update the user table during downloads in which case 
the damage is limited to the user record with the cr at ppy connection.  
Even osmosis which is running physically near the database server always 
dumps each query to disk to minimise read time before assembling results 
and writing "diff" files.

But input is triggering writes and could be obtaining various locks all 
across the database.  If we're waiting on an unknown quality network 
connection to send the data then our transaction durations are totally 
unbounded.  We're exposing our nice database to the whims of the outside 
world.  Not to mention making my life more difficult than it otherwise 
would be ;-)

Brett





More information about the dev mailing list