[Tile-serving] openstreetmap-carto benchmark results

[Paul Norman]

See
http://lists.openstreetmap.org/pipermail/tile-serving/2013-May/000216.html
for methods.

== Summary ==

In brief, tests are on an m2.2xlarge EC2 instance with 34.2GB of RAM, 4
cores and on-instance storage that gets about 500-1500 iops, with a database
consisting only of rendering tables. Rendering consists of two 1k set of
requests which only uses 24GB of cache with osm.xml and two 9k sets of
requests which do hit the disk. PostgreSQL 9.1 + PostGIS 1.5 are used and
renderd+mapnik 2.1 are from apmon's osm-unstable PPA. All values in meta
tiles per second. +/- values are 1 standard deviation, but as they are all
under 1% I am omitting them. For reference, yevaud handles about 5.0
meta/second at peak and 3.75 meta/second average.

Unfortunately thanks to an error in my methods I don't have meaningful cpu%
figures.

== Tile speed results ==

For the from-ram set of tiles I achieved 3.734 with osm.xml and 2.930 with
openstreetmap-carto. This is a decrease of 22%.

For the larger set I achieved 2.400 with osm.xml and 2.104 with
openstreetmap-carto. This is a decrease of 12%.

== Disk utilization ==

Disk utilization does not count writing meta-tiles which was done to a
different volume.

1k osm.xml and osm-carto disk utilization was close to 0.

9k osm.xml disk utilization varied from 66-75% or 484-672 iops. 

9k osm-carto disk utilization varied from 55-68% or 375-651 iops (10 minute
measurement intervals)

I don't have detailed disk utilization numbers, but it appears higher for
osm.xml. Of course it's quite possible that this is entirely because it's
rendering faster and it actually needs the same io per tile.

== Impact on yevaud ==

The 9k workload should be more representative of the load on yevaud.

These tests have a few limitations for determining what will happen on
yevaud:

- They isolate the effect of the carto stylesheet. If yevaud is upgraded to
a different version of postgres+postgis at the same time that would have a
positive effect on speed. 
	
- yevaud has been running on its current import for a long time.
Re-importing will re-cluster the data and create new indices which will not
of had the page splits the current ones do, increasing index speed.

- yevaud has non-standard indices to optimize some queries

- flatnodes should increase update speed, leaving more capacity for 

If I were forced to guess I would guess that when combined with a reimport
and software upgrade on yevaud, openstreetmap-carto would be no slower than
osm.xml and at worst 10% slower.

== By zoom ==

No tiles below z7 were rendered. Of those rendered, the total time spent and
number of requests were (tab-delimited)

	osm.xml	osm-carto	requests
7	1002.1	1172.8	3
8	0	0	0
9	0	0	0
10	28.2	28.2	3
11	241.9	241.8	2
12	122	144.8	80
13	11092.2	12010	849
14	5379.7	5709.9	1303
15	3780.6	4133.3	1954
16	3421	3625	3029
17	4011.8	4258.6	4758
18	4706.1	5033.3	6021

Graphed: http://www.openstreetmap.org/user/pnorman/diary/19289

z13 stands out as a big spike, with z13 taking as long as z18, z17 and z16
combined. This is pretty clearly the obvious target for optimization.

Additionally, z12 is quicker than z13, z14 or 15 on a per-meta basis. I
wonder if z12 dirtying behavior could be changed to be the same as higher
zooms.