[Tile-serving] openstreetmap-carto benchmark results

Kai Krueger kakrueger at gmail.com
Thu May 16 16:03:13 UTC 2013


Hello,

nice work!

Interesting comparison between the smaller (cached) and the larger (non 
cached) render sets. From those numbers, it appears that the differences 
in db requests between the carto and osm.xml styles are not as 
important, or non existent. Instead the effects appear to be CPU bound, 
although one can't say purely from the numbers, if it might be in 
postgresql or mapnik itself.

I am not entirely sure I buy your conclusion that once put on yevaud or 
orm, the difference would be more towards 12% than 22%. You cite numbers 
of 500 - 700 IOPS that are done in osm.xml. From just those numbers it 
is hard to say if the 700 IOPs are limited due to CPU or IO bandwidth 
(after all, it only has 4 cores compared to yevauds 8 - 16 (with 
hyperthreading) cores, using something like 14 render threads), but 
yevaud likely has higher IO performance (currently the numbers are ~3k/s 
average and a max of ~6k/s and it is mostly CPU bound, so probably could 
do more, even on the single old "slow" 320 SSD). Therefore my guess 
would be (although it needs closer analysis) that on yevaud or orm, you 
would see differences closer to your fully cached regime of 22%.

As you say, there are other factors at play, such as upgrading postgres 
/ postgis and mapnik or a reimport / reclustering when moving to orm 
(which are hopefully positive), so it is hard to judge the absolute 
change in performance when doing the switch. Furthermore, if the setup 
is moved over to multi-system rendering, then that should give a 
substantial performance gain, which initially wouldn't be needed. That 
could give the head room to absorb the 10 - 30% slow down and allow to 
switch very soon and then optimize the process until growth demands 
those extra resources back.

Kai


On 05/15/2013 07:26 PM, Paul Norman wrote:
> See
> http://lists.openstreetmap.org/pipermail/tile-serving/2013-May/000216.html
> for methods.
>
> == Summary ==
>
> In brief, tests are on an m2.2xlarge EC2 instance with 34.2GB of RAM, 4
> cores and on-instance storage that gets about 500-1500 iops, with a database
> consisting only of rendering tables. Rendering consists of two 1k set of
> requests which only uses 24GB of cache with osm.xml and two 9k sets of
> requests which do hit the disk. PostgreSQL 9.1 + PostGIS 1.5 are used and
> renderd+mapnik 2.1 are from apmon's osm-unstable PPA. All values in meta
> tiles per second. +/- values are 1 standard deviation, but as they are all
> under 1% I am omitting them. For reference, yevaud handles about 5.0
> meta/second at peak and 3.75 meta/second average.
>
> Unfortunately thanks to an error in my methods I don't have meaningful cpu%
> figures.
>
> == Tile speed results ==
>
> For the from-ram set of tiles I achieved 3.734 with osm.xml and 2.930 with
> openstreetmap-carto. This is a decrease of 22%.
>
> For the larger set I achieved 2.400 with osm.xml and 2.104 with
> openstreetmap-carto. This is a decrease of 12%.
>
> == Disk utilization ==
>
> Disk utilization does not count writing meta-tiles which was done to a
> different volume.
>
> 1k osm.xml and osm-carto disk utilization was close to 0.
>
> 9k osm.xml disk utilization varied from 66-75% or 484-672 iops.
>
> 9k osm-carto disk utilization varied from 55-68% or 375-651 iops (10 minute
> measurement intervals)
>
> I don't have detailed disk utilization numbers, but it appears higher for
> osm.xml. Of course it's quite possible that this is entirely because it's
> rendering faster and it actually needs the same io per tile.
>
> == Impact on yevaud ==
>
> The 9k workload should be more representative of the load on yevaud.
>
> These tests have a few limitations for determining what will happen on
> yevaud:
>
> - They isolate the effect of the carto stylesheet. If yevaud is upgraded to
> a different version of postgres+postgis at the same time that would have a
> positive effect on speed.
> 	
> - yevaud has been running on its current import for a long time.
> Re-importing will re-cluster the data and create new indices which will not
> of had the page splits the current ones do, increasing index speed.
>
> - yevaud has non-standard indices to optimize some queries
>
> - flatnodes should increase update speed, leaving more capacity for
>
> If I were forced to guess I would guess that when combined with a reimport
> and software upgrade on yevaud, openstreetmap-carto would be no slower than
> osm.xml and at worst 10% slower.
>
> == By zoom ==
>
> No tiles below z7 were rendered. Of those rendered, the total time spent and
> number of requests were (tab-delimited)
>
> 	osm.xml	osm-carto	requests
> 7	1002.1	1172.8	3
> 8	0	0	0
> 9	0	0	0
> 10	28.2	28.2	3
> 11	241.9	241.8	2
> 12	122	144.8	80
> 13	11092.2	12010	849
> 14	5379.7	5709.9	1303
> 15	3780.6	4133.3	1954
> 16	3421	3625	3029
> 17	4011.8	4258.6	4758
> 18	4706.1	5033.3	6021
>
> Graphed: http://www.openstreetmap.org/user/pnorman/diary/19289
>
> z13 stands out as a big spike, with z13 taking as long as z18, z17 and z16
> combined. This is pretty clearly the obvious target for optimization.
>
> Additionally, z12 is quicker than z13, z14 or 15 on a per-meta basis. I
> wonder if z12 dirtying behavior could be changed to be the same as higher
> zooms.
>
>
> _______________________________________________
> Tile-serving mailing list
> Tile-serving at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/tile-serving




More information about the Tile-serving mailing list