[OSM-talk] Slow TileMill rendering - Postgres using 1 core?

Mon Aug 26 03:35:08 UTC 2013

Hi Christian,
  Thanks for the reply. After further investigation the actual problem was
a missing index on planet_osm_polygon. (I'm really not sure why.)

Steve

On Sun, Aug 25, 2013 at 9:00 PM, Christian Quest <cquest at openstreetmap.fr>wrote:

> As far as I know, Tilemill is using mapnik which is querying postgres.
> Plain vanilla Mapnik is not doing more than one postgres query at a time
> (not multithreading queries).
> A patch made by mappy allows mapnik to multithread its pg queries. Are you
> using the exact same version of Mapnik as before ?
>
>
> 2013/8/25 Steve Bennett <stevagewp at gmail.com>
>
>> Hi all,
>>
>> I'm running TileMill on an 8 core Ubuntu VM with 32GB of memory, on an
>> OpenStack cloud. Recently, my VM was destroyed, and I rebuilt it
>> (identically, I thought) on slightly different hardware (same cloud, but
>> different physical infrastructure).
>>
>> The new build is much slower at rendering - a screen worth of tiles at
>> zoom 13 can take around a minute. That is, with virtually the same setup,
>> same data, same styles. You can see some slow tiles here:
>>
>> http://emscycletours.site44.com/mel.html
>>
>> While panning around, the 'top' command shows mostly Postgres processes
>> (different from last time I had performance problems[1], when the
>> bottleneck was in Mapnik). Total CPU usage hangs around 12%: ie, exactly 1
>> out of 8 cores is being used.
>>
>>
>> https://dl.dropboxusercontent.com/u/767553/GIS/Screen%20shot%202013-08-25%20at%2011.15.01%20AM.png
>>
>> top - 11:10:32 up 3 days, 36 min,  1 user,  load average: 0.06, 0.17, 0.22
>> Tasks: 133 total,   4 running, 129 sleeping,   0 stopped,   0 zombie
>> %Cpu(s): 11.5 us,  0.1 sy,  0.0 ni, 88.4 id,  0.0 wa,  0.0 hi,  0.0 si,
>>  0.0 st
>> KiB Mem:  32950396 total,  7150132 used, 25800264 free,   117864 buffers
>> KiB Swap:        0 total,        0 used,        0 free,  5221356 cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
>>  2353 postgres  20   0 8510m 640m 635m S  41.2  2.0  23:36.57 postgres
>>  2354 postgres  20   0 8510m 644m 639m S  40.2  2.0  23:24.26 postgres
>>  2350 postgres  20   0 8510m 642m 638m S  14.0  2.0  23:19.19 postgres
>>  2375 postgres  20   0 8510m 643m 639m S  14.0  2.0  23:17.80 postgres
>> 13102 postgres  20   0 8508m 531m 527m S  13.6  1.7  13:03.21 postgres
>>  2355 postgres  20   0 8508m 531m 526m S  13.3  1.7  13:45.15 postgres
>>  2352 postgres  20   0 8510m 640m 636m S  10.0  2.0  23:31.17 postgres
>>  2348 postgres  20   0 8510m 644m 639m S   9.3  2.0  23:41.88 postgres
>> 12420 mapbox    20   0 3818m 1.0g 755m S   9.3  3.2  36:48.39 nodejs
>>  2357 postgres  20   0 8508m 530m 526m S   7.3  1.7  13:38.57 postgres
>>  2356 postgres  20   0 8508m 531m 526m R   6.3  1.7  13:42.52 postgres
>>  2376 postgres  20   0 8508m 531m 527m S   6.0  1.7  13:35.51 postgres
>> 13195 postgres  20   0 8508m 531m 527m S   5.3  1.7  12:33.65 postgres
>>  3027 postgres  20   0 8508m 531m 527m R   3.3  1.7  13:29.06 postgres
>>  2349 postgres  20   0 8508m 530m 526m S   3.0  1.6  13:38.19 postgres
>>  2358 postgres  20   0 8508m 531m 527m S   3.0  1.7  13:44.59 postgres
>>    26 root      20   0     0    0    0 S   0.3  0.0   0:08.64 ksoftirqd/5
>>  2335 postgres  20   0 8489m 2732 1340 S   0.3  0.0   1:00.48 postgres
>>
>> So, wondering if anyone has any suggestions what the problem is, or how
>> to fix it? Why is Postgres apparently using only one core, even though it
>> has many processes? What tools could I use to further diagnose?
>>
>> My changed Postgres settings are as follows:
>>
>> shared_buffers = 8GB
>> autovacuum = on
>> effective_cache_size = 8GB
>> work_mem = 128MB
>> maintenance_work_mem = 64MB
>> wal_buffers = 1MB
>> checkpoint_segments = 10
>>
>> The server is set up as described here:
>> http://steveko.wordpress.com/2013/05/08/tilemill-server/
>>
>> I'm not yet using any tile cache. I will do that next, but the problem
>> I'm trying to solve at the moment is very slow tile generation, not slow
>> serving of rendered tiles.
>>
>> Many thanks in advance,
>> Steve
>>
>>
>> [1]
>> http://gis.19327.n5.nabble.com/TileMill-performance-td5751158.html
>>
>>
>> _______________________________________________
>> talk mailing list
>> talk at openstreetmap.org
>> http://lists.openstreetmap.org/listinfo/talk
>>
>>
>
>
> --
> Christian Quest - OpenStreetMap France
> Un nouveau serveur pour OSM... http://donate.osm.org/server2013/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20130826/b2482964/attachment-0001.html>