[OSM-dev] Mass imports (TIGER and AND)

[Ray Booysen]

Dave Hansen wrote:
> On Tue, 2007-08-28 at 09:34 +0200, Martijn van Oosterhout wrote:
>   
>> On 8/28/07, Dave Hansen <dave at sr71.net> wrote:
>>     
>>> I've been very painfully uploading the TIGER-generated data through
>>> JOSM.  At the rate I'm going it will probably take 5 or 10 years to
>>> upload the entire US.  Literally.  I'm uploading one or two counties a
>>> day, and there are 3,234 counties in the country.
>>>       
>> Ah, do you have any idea on the number of requests per second that
>> approximatly equates to?
>>     
>
> I just did 100 requests in 30 seconds, so ~3.3.  
>
>   
>>> So, I installed the rails port on my laptop, and sicked JOSM on it.  The
>>> uploads are maybe twice as fast as they are to the main OSM server.  So,
>>> the round-trip-time actually isn't that _huge_ of a performance
>>> bottleneck.
>>>       
>> Over the last view days I've written a bulk_import.pl which takes a
>> JOSM or osmChange file and uploads it to the server. It should have
>> lower overhead than JOSM, but from what you're saying it that it won't
>> really help. :(
>>
>> The AND import has > 8 million objects to create, so speed is of concern.
>>     
>
> Actually, at 3.3/second, that's only 28 days.  That's no so bad.  
>
> I'm estimating that the average TIGER county in the US is about 3MB, in
> compressed OSM format.  Using a random 3MB county, that means ~192,000
> objects.  There are 3200 counties in the US, so 192,000*3200 /
> 3.3*86,400 (seconds in a day) = 2154.88 days.  Counting leap days,
> that's ~5.9 years. :)
>
> I'm tempted to start chanting *U* *S* *A* over and over, but I think
> I'll hold myself back on this one. ;)
>
> -- Dave
>
>   

Will not moving some of the work out to seperate clients not solve this 
problem?  With testing you're doing requests from one machine whereas 
the server should be able to handle multiple requests from multiple 
clients better.

Regards
Ray

--
Ray Booysen
raybooysen at rjb.za.net