[Tile-serving] [openstreetmap/osm2pgsql] Slow --append node processing with osm2pgsql 1.8.1 (7-8* slower than older setup) (Discussion #1971)

[wgds60]

Hi,

I'm currently testing out the new osm2pgsql version 1.8.1 but for some reason the data updating via --append is way slower than on an older dev setup.

The new setup uses:
osm2pgsql version 1.8.1
postgresql version 15.3
postgis version 3.3

as well as better hardware.

But for some reason the --append node processing speed is about 7 to 8 times slower than on an older setup that used:

osm2pgsql 1.2.0
postgresql 10.23
postgis 2.4

The command used is:
`osm2pgsql --append --slim -d <dbname> --number-processes 10 -C 64000 --hstore --flat-nodes=<flatnodefile> --style=<stylefile> --tag-transform-script=<luatransformfile> <changes.osc.gz-file>`

One difference:

On the new setup I compiled osm2pgsql 1.8.1 with LuaJIT enabled - that actually results in an older Lua version used (5.1.4 through LuaJIT 2.1.0-beta3, which seems pretty old...) instead of Lua 5.2.4 without LuaJIT in osm2pgsql 1.2.0.

But the README states that LuaJIT should fasten the import - maybe it is fastening the import but not updates? Or that info is outdated as LuaJIT seems to be stale?

I'm also seeing not many resources being used by osm2pgsql while node processing, cpu, i/o and ram are practically not used at all.

Currently the update on the new osm2pgsql 1.8.1 runs with ~ 0.7k/s node processing, while the old setup runs with about 5k/s or more...

Postgresql tuning settings are about the same.





-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/discussions/1971
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/osm2pgsql/repo-discussions/1971 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20230602/5d500a1a/attachment.htm>