[Tile-serving] [openstreetmap/osm2pgsql] Tile Expiry Performance (#946)

[intergalacticdata]

This issue was posted to the osm-dev mailing list, per Nakaner's request, on May 29 2019. I am documenting it here as well so that it doesn't become lost.

While implementing a tile expiration solution I have run into some issues with the new tile expiry expansion. I'm afraid I do not have precise timings, but I'm seeing what appears to be at least an order of magnitude performance penalty, probably due to memory exhaustion.

Test machine is Debian stretch 4CPUs, 26GB RAM, SSD Array. Initial osm import was performed with osm2pgsql 0.92 and finished in under 48 hours.

Under 0.92 I was running multiple render chains while importing changesets. Tests w/ 0.96 have been run against an otherwise idle machine.

Database & flatnodes file were restored to initial state between each round of testing.

WEEKLY changeset using 0.92:
-=-=-=-
time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U \
$PGUSER -r xml changes.osc \
--flat-nodes /database/postgresql/OSM-FLATNODES --slim \
--number-processes 4 --style openstreetmap-carto.style \
--tag-transform-script openstreetmap-carto.lua -e19 -o \
$WORKDIR_OSM/expired-tiles

With -e19, I was able to import a WEEKLY changeset in roughly 24 hours.


Using git repository (5/27? pull):
-=-=-=-
time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U
$PGUSER -r xml changes.osc \
--flat-nodes /database/postgresql/OSM-FLATNODES --slim \
--number-processes 4 --style openstreetmap-carto.style \
--tag-transform-script openstreetmap-carto.lua -e10-16 -o \
$WORKDIR_OSM/expired-tiles

Consistently crashed w/ a bad alloc(). I didn't note any unusual output in the compile. Crashes even with a 24 hour changeset.


DAILY changeset using Debian backport to stretch (0.96):
-=-=-=-
As above, using -e10-16. 

22 hours spent processing a 24 hour changeset and still importing new relations. It's 35GB into swap, with osm2pgsql claiming 89% of the memory usage.

-------

I realize there is a large difference between zoom level 19 and 10-16, but I assume it should take significantly less RAM/CPU for 10-16.

Same test as above WITHOUT expiring tiles:
-=-=-=-
After 3 hours, the update has reached the same state of progress. osm2pgsql (debian backport) 0.96 maintains < 1GB of memory usage without expiring tiles. 

While expiring tiles, it ballooned to >50GB before being killed.

WORK AROUND / FINAL THOUGHTS
-=-=-=-
This is pretty clearly a memory leak. The initial test results reported by Nakaner for this patch show the same high levels of RAM usage (28GB for Europe alone).

My workaround uses 0.92 output and an external program to emulate what the newer tile expiration code is supposed to do:
https://intergalacticdata.com/public_domain/expandExpire.c

I've spoken w/ Nakaner and get the impression that he considers this memory usage to be a) normal and b) not due to tile expiration.

I submit that using osm2pgsql 0.92 plus my (admittedly simplistic) program does the same job in 2% of the RAM required by 0.96.

0.96 will also complete the job in 2% of the RAM if not asked to expire tiles.

I am not using tile expiration at this time due to business reasons and am therefore unable to dedicate any more time to this issue. I am merely documenting here in hopes of saving the next 'me' a week of their lives.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/946
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20190814/7b1f14c3/attachment.html>