[Tile-serving] [openstreetmap/osm2pgsql] Why so slow, even after upgrading to full SSD system and doubling RAM to 32Gb? (#883)

[mdadduzio]

Hi lonvia,

thanks for your reply.

We are both right :-)
I mean, at the time of writing no swap memory was used (yet).
When ways parsing has finished and relation parsing commenced swap has been used heavily, as much as 28.5 - 29 Gb.
As I thought, being the swap memory on the SSD too, swapping didn't slow the process down as much as if it was on a HDD.
In fact this time parsing hasn't lasted around 6 days but only around 12 hours (I'll write down the right numbers when I have them).
Overall I'm experiencing numbers a bit better than the ones you wrote down.

It looks the HW upgrade has paid in the end, but not in the first two stages of parsing which have rates almost equal to the not upgraded HW.

Relation parsing was greatly sped up and, till what I can see now also the parallel processing of the pending ways (not their initial parse step).

When I have the import completed I'll write here the final results.
Unfortunately I'm experiencing another problem as of now: Ubuntu 18.04 rebooted just after ending the pending ways.
I could not determine the cause of the reboot: it may be because of an unattended update or more probably because of overheating.
I disabled all kinds of updates I could, switched the network interface off and put a big big fan cooler on the side of the notebook :-)
I'm just crossing my fingers hoping this time it can make it.
I don't think it ran out of memory, disk wasn't full, swap was 32Gb + a 16Gb swapfile I added on the fly (but swap memory peaked at 28.7Gb, so it wasn't ever used).
Later, after the reboot, I deleted the swapfile and grew the swap partition to 48Gb, just to play on the safe side. 

Let's see how it goes.

I'll test the SSD speed when I finish, I thought Samsung 860 Evo was a good one.

I only have one question: why ways parsing (from PBF) hasn't been affected by the HW upgrade? 
About nodes it was obvious because the flat nodes file was already on SSD, but ways should already be using postgres, so why that step didn't get a speed up as it was for relations and pending ways too (tables and indexes I don't know yet)?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/883#issuecomment-443548607
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20181202/5b6804b3/attachment-0001.html>