[Tile-serving] [openstreetmap/osm2pgsql] objections to renumber the ids in the process osm2pgsql / mapnik / renderd (#824)

[sav123]

The pre process builds the flat nodes files and a small patch tells to osm2pgsql to not delete the file and to not set the data but to use it as usual. After that the main process doesn't need more the not-dense nodes and this saves a lot of time. Renumbering makes the o5m and flat input files a little smaller and there are no holes in memory when tracking. The external flat nodes saves 10%, the renumbering 5 to 10%, it depends of the final strategy of aggregation. This covers the time of the prebuild. Take in account that the 1st stage allows to discard tags and relations with the rules of the main process and you grab another percents. As today, after other savings, on a 64 Gb old server ( 2 gHz x2, 16 threads, DDR3, Sata allmost full ) the pre-build and the patched mid process take 15300 s on average of 4 tries and they never consume more than 32 Gb. Another thousand seconds is possible to save. The bottleneck is postgress with 53000 s for xxxx_polygon. Recent documentation on optimization is missing for big tables while tries in real conditions take a lot of time.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/824#issuecomment-368335956
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20180225/eb9529e7/attachment-0001.html>