[OSM-dev] Server Timeouts - JOSM and the forthcoming big uploads via the API

[Andy Robinson]

We had a discussion at the meetup on Saturday about server timeouts when
working through the API.

While uploading TIGER data last week it became apparent that when the API is
very quiet (like I am just about the only person uploading data) then using
JOSM the server times out as regularly as clockwork at around 250 node
uploads. At busier times the upload might be slower but you may get past the
250 point because someone else suffers the timeout on the specific call
rather than you.

>From the discussion it appears the original memory leak process shutdown was
removed when the rails port went live so therefore the timeouts are perhaps
limits on the query queue to stop it getting into problems?

It would be useful to have this issue confirmed/properly documented so that
solutions can be discussed. Its very painful in JOSM having to attend major
uploads and I'm specifically thinking about any users wishing to use JOSM
for TIGER county imports. Upload of all the AND data soon will also run into
a significant delay each time the timeout response is awaited and the upload
restarted, whether manually or automatically.

A couple of things that were mentioned at the meet were perhaps running API
requests through a proxy at the client end or running squid at the server
end, however these were only brief comments since the exact reason is was
not understood by those discussing.

All this leads on to what impact a lot of AND and TIGER data is going to do
to the platform. We know from last time around that the TIGER data swamped
everything very quickly if left to run even at a 1 second insertion cycle
and hence it was generally running at a 3 second insertion cycle. This time
we have also the AND data to consider and also the fact that once the TIGER
import starts its likely to be done by lots of JOSM uploaders rather than a
single import script.

Thoughts?

Cheers

Andy

Andy Robinson
Andy_J_Robinson at blueyonder.co.uk