[OSM-dev] Minute Diffs Broken
brett at bretth.com
Tue May 5 23:43:12 BST 2009
Matt Amos wrote:
> the XML document is parsed incrementally to save memory, rather than
> for its behaviour, but it appears that rails, lighty and fastcgi all
> support streaming input. i am unsure if they all work together, but
> the rails docs suggest that it does.
> streaming input isn't nearly as difficult as streaming output. since
> rails treats (most) uncaught exceptions as 500 errors, but the error
> status is in the header, rails takes the safe option of having the
> full response in a buffer before it returns the status header.
Is this a good idea? I'm not comfortable with the idea of streaming
input. At least with streaming output you are performing a read-only
operation (I think) and therefore shouldn't be holding any important
locks. Unless you update the user table during downloads in which case
the damage is limited to the user record with the cr at ppy connection.
Even osmosis which is running physically near the database server always
dumps each query to disk to minimise read time before assembling results
and writing "diff" files.
But input is triggering writes and could be obtaining various locks all
across the database. If we're waiting on an unknown quality network
connection to send the data then our transaction durations are totally
unbounded. We're exposing our nice database to the whims of the outside
world. Not to mention making my life more difficult than it otherwise
would be ;-)
More information about the dev