[OSM-dev] 0.6 bulk uploader

[Stefan de Konink]

Frederik Ramm wrote:
> Stefan de Konink wrote:
>> Where do we need a limit? 
> 
> I assume that while doing all the inserts, the Ruby code has to keep 
> track of all the Ids involved in order to be able to adjust the 
> references in other objects. 

If this is done within a transaction, the objects ids can be fetched 
*after* the insert is realised. This will just give you the last N rows 
with ids.

> I'd rather impose an arbitrary limit which is much smaller than what the 
> servers can theoretically handle and publish this so that whoever 
> uploads something has a chance to be relatively sure that his query will 
> work - instead of letting anyone upload an arbitrarily big diff that 
> will give a Rails "out of memory" exception somewhere down the line.

Please think along the line of other fcgi applications that do not have 
the same problems as you point out here. There is no a priori rule that 
tells you the complete stack should be in Ruby, and already parts have 
been migrated.


Stefan