[Tile-serving] Problems benchmarking osm.xml

[Paul Norman]

I've been trying to benchmark the new carto style, so I decided to start
with a baseline from osm.xml.

I took a list of 2000 meta-tile requests from a log from yevaud, and put the
first half into one list and the second half into another.

I then rendered the first list, the second, disregarding the times. This was
to get the cache into the a consistent state.

I then alternated between the first list and the second five times,
recording the time to render the second.

The first time I did this I noticed that the 5 benchmark times were
monotonically decreasing. Worried that I was having caching issues, I reran
the entire test sequence twice with the same tile lists.

I got a result of an average of 4.9k seconds (0.35 meta-tiles per second)
with a standard deviation of 4k seconds.

If I throw out the two most extreme results, I still get a standard
deviation over 50% of the average.

Including both render sets, I did about 24 hours of rendering. I'm doing
this on errol which has other processes running, but I would expect them to
average out.

Does anyone have any suggestions on how to get times which I can use to
compare various options, I would welcome them. Right now it seems like I'd
have to do too much benchmarking to get a meaningful result.