[OSM-dev] Mass imports (TIGER and AND)

[Dave Hansen]

I've been very painfully uploading the TIGER-generated data through
JOSM.  At the rate I'm going it will probably take 5 or 10 years to
upload the entire US.  Literally.  I'm uploading one or two counties a
day, and there are 3,234 counties in the country.

So, I installed the rails port on my laptop, and sicked JOSM on it.  The
uploads are maybe twice as fast as they are to the main OSM server.  So,
the round-trip-time actually isn't that _huge_ of a performance
bottleneck.

The thing that *IS* on my laptop is the ruby code.  It is responsible
for 90% of the CPU time, and the CPUs are maxed out.  mysql, on the
other hand, is responsible for ~3% of total cpu time.  Even with my
piddly notebook hard drive, the I/O wait time is under 1%.

People have been saying that we should write the import code in ruby to
run on the server and use the existing rails code.  If the ruby code
itself is the bottleneck and not the round-trip time or the disk, is
doing the import through the ruby code going to even help?

-- Dave