[Tile-serving] [openstreetmap/osm2pgsql] Rewrite tile expiry (management of expired tiles) (#709)

[Michael Reichert]

Here are the performance measurements. I used the same machine as for the performance tests of my first pull request.

The database was imported using `./osm2pgsql -d expirytest --merc --style ../default.style --flat-nodes /ssd/michael/flatnodes/flat.nodes --slim --unlogged --number-processes 2 -C 40000 /home/michael/jobs/tileexpirytest/europe-180204.osm.pbf` (took 10 hours and 44 minutes and 28.9 GiB RAM)

The commands in the order I performed them. There was a break of more than a day between the second and the third test. After each test, I shut down the PostgreSQL daemon using Systemd, restored a backup of the tablespace and restarted the daemon again. I also restored the flatnodes file.

```
My branch (commit 2e612f5a00bb93eb8fb13d8b3ab7eed0908b589e):
Osm2pgsql took 2539s overall
        Command being timed: "./osm2pgsql -d expirytest --merc --style ../default.style --flat-nodes /ssd/michael/flatnodes/flat.nodes --append --slim --number-processes 2 -C 25000 -o /home/michael/jobs/tileexpirytest/expirylog-branch-785.txt --expire-bbox-size 20000 -e 10-16 /home/michael/jobs/tileexpirytest/785.osc.gz"
        User time (seconds): 442.10
        System time (seconds): 181.51
        Percent of CPU this job got: 24%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 42:20.39
        Maximum resident set size (kbytes): 34669536

Master branch (commit e5869274659ada90208927c54cde51b2b35d6bbf):
Osm2pgsql took 2150s overall
        Command being timed: "./osm2pgsql -d expirytest --merc --style ../default.style --flat-nodes /ssd/michael/flatnodes/flat.nodes --append --slim --number-processes 2 -C 25000 -o /home/michael/jobs/tileexpirytest/expirylog-master-785.txt --expire-bbox-size 20000 -e 10-16 /home/michael/jobs/tileexpirytest/785.osc.gz"
        User time (seconds): 395.18
        System time (seconds): 163.13
        Percent of CPU this job got: 25%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 35:51.19
        Maximum resident set size (kbytes): 36647904


Second attempt after ensuring that no other processes disturb the measurements any more:
Master branch (commit e5869274659ada90208927c54cde51b2b35d6bbf):
Osm2pgsql took 2553s overall
        Command being timed: "./osm2pgsql -d expirytest --merc --style ../default.style --flat-nodes /ssd/michael/flatnodes/flat.nodes --append --slim --unlogged --number-processes 2 -C 25000 -o /home/michael/jobs/tileexpirytest/expirylog-master-785-v2.txt --expire-bbox-size 20000 -e 10-16 /home/michael/jobs/tileexpirytest/785.osc.gz"
        User time (seconds): 411.40
        System time (seconds): 194.65
        Percent of CPU this job got: 23%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 42:34.11
        Maximum resident set size (kbytes): 34685544

My branch (commit 2e612f5a00bb93eb8fb13d8b3ab7eed0908b589e):
Osm2pgsql took 2440s overall
        Command being timed: "./osm2pgsql -d expirytest --merc --style ../default.style --flat-nodes /ssd/michael/flatnodes/flat.nodes --append --slim --number-processes 2 -C 25000 -o /home/michael/jobs/tileexpirytest/expirylog-branch-785-v2.txt --expire-bbox-size 20000 -e 10-16 /home/michael/jobs/tileexpirytest/785.osc.gz"
        User time (seconds): 420.88
        System time (seconds): 174.78
        Percent of CPU this job got: 24%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 40:40.53
        Maximum resident set size (kbytes): 36634188

My branch  (commit 2e612f5a00bb93eb8fb13d8b3ab7eed0908b589e) without tile expiry:
Osm2pgsql took 2493s overall
        Command being timed: "./osm2pgsql -d expirytest --merc --style ../default.style --flat-nodes /ssd/michael/flatnodes/flat.nodes --append --slim --number-processes 2 -C 25000 --expire-bbox-size 20000 /home/michael/jobs/tileexpirytest/785.osc.gz"
        User time (seconds): 336.77
        System time (seconds): 170.91
        Percent of CPU this job got: 20%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 41:33.65
        Maximum resident set size (kbytes): 36221356
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/709#issuecomment-365647678
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20180214/1047b686/attachment.html>