[Tile-serving] [osm2pgsql] Extremely slow database update (#128)

[apmon]

The diff import will always be much slower (orders of magnitude) than the initial import. There is just more that needs doing and it also depends on the data in the database. i.e. instead of just writing data linearly into the database, for each object, you first need to query it, then if it already exists i.e. is a modified object delete it and only then write it out. Perhaps eventually someone can find out how to boost this speed, but for the moment it is normal.

However, for diff processing, there is a lot less to do than for the initial import and so the reduced speed is usually acceptable. E.g. a typical minutely diff file will have somewhere between 1k - 4k nodes in it. So if you can process the diff file at 3k/s you should easily be able to complete the diff file in less than the minute.

On fast servers, I have seen the diff process typically be somewhere between 4 - 10 times realtime. I.e. an hours worth of diffs can be imported in ~ 5 - 15 minutes (very rough numbers as they differ greatly depending on hardware.

So overall on first sight, your numbers don't look too terrible. How long does it take you to import e.g. an hourly diff? Is your setup sufficiently fast?

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/128#issuecomment-39381053
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20140402/28b9a4d7/attachment.html>