[Tile-serving] [osm2pgsql-dev/osm2pgsql] osm2pgsql --append to existing imported slim database takes ages (Issue #2266)

[Jochen Topf]

It is doing the right query now (`SELECT osm2pgsql_find_changed_ways()`), so it is using the index. That's good. There is still the open COPY which should not be there, I need to investigate that.

Yes, the whole thing is not very fast, but it is not hanging any more as it was before, it is making progress. It has been a long time since I have seen a system with spinning disk. So I am not sure any remaining problem you are seeing is not due to disk I/O. Before you saw the postgres using 100% CPU in state R. Is that still the case?

The processing log lines are interesting. You quote these:

Processing: Node(1320k 4.1k/s
Processing: Node(1325k 0.9k/s)

So the last 5k nodes were really really slow? Could well be that a vacuum or checkpoint or whatever came in there. That's why those numbers are only marginally useful, there can always be something else running there.

In general: There might be some slowdown in newer versions compared to older version due to the different way we are finding the ways/relations we have to reprocess because of changed nodes. We are doing somewhat more work than before. I have not seen such slowdowns on SSDs, but it might well be that they show up on HDDs. But there isn't anything we can do about that.

You might want to look into using a flat node file. For your use case it might be better or worse, hard to say.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/issues/2266#issuecomment-2435557402
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/issues/2266/2435557402 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20241024/36536297/attachment.htm>