[OSM-dev] Mass imports (TIGER and AND)

[Dave Hansen]

On Tue, 2007-08-28 at 09:34 +0200, Martijn van Oosterhout wrote:
> On 8/28/07, Dave Hansen <dave at sr71.net> wrote:
> > I've been very painfully uploading the TIGER-generated data through
> > JOSM.  At the rate I'm going it will probably take 5 or 10 years to
> > upload the entire US.  Literally.  I'm uploading one or two counties a
> > day, and there are 3,234 counties in the country.
> 
> Ah, do you have any idea on the number of requests per second that
> approximatly equates to?

I just did 100 requests in 30 seconds, so ~3.3.  

> > So, I installed the rails port on my laptop, and sicked JOSM on it.  The
> > uploads are maybe twice as fast as they are to the main OSM server.  So,
> > the round-trip-time actually isn't that _huge_ of a performance
> > bottleneck.
> 
> Over the last view days I've written a bulk_import.pl which takes a
> JOSM or osmChange file and uploads it to the server. It should have
> lower overhead than JOSM, but from what you're saying it that it won't
> really help. :(
> 
> The AND import has > 8 million objects to create, so speed is of concern.

Actually, at 3.3/second, that's only 28 days.  That's no so bad.  

I'm estimating that the average TIGER county in the US is about 3MB, in
compressed OSM format.  Using a random 3MB county, that means ~192,000
objects.  There are 3200 counties in the US, so 192,000*3200 /
3.3*86,400 (seconds in a day) = 2154.88 days.  Counting leap days,
that's ~5.9 years. :)

I'm tempted to start chanting *U* *S* *A* over and over, but I think
I'll hold myself back on this one. ;)

-- Dave