[OSM-dev] Superslow nominatim index creation with Postgres 9.0
fatzopilot
fatzopilot at gmx.net
Sat Mar 26 16:17:19 GMT 2011
Note: This is a copy of a former post that did not make it on the
mailinglist.
Hi,
On http://wiki.openstreetmap.org/wiki/Nominatim/Installation it says index
creation "will take a very long time - up to 10 days for a full planet on a
small machine.". This is probably information added at 10th September 2010
and in the meantime, the plantefile has grown a bit, but:
Currently, I am indexing a planet file (planet-110126.osm.pbf) for use with
Nominatim and indexing has now nearly run for 800hs (that's more than a
month).
It’s a 8 core machine
> cat /proc/cpuinfo
...
model name : Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
stepping : 5
cpu MHz : 2668.000
cache size : 8192 KB
...
The 8 cores are completely busy all the time (predominantly usr, so CPU is
the bottleneck here) and there is plenty of free RAM. There are huge reads
(from 1,5 - 62 MB/s with an average of ~16MB/s) and from time to time small
writes (20 - 8000 KB with an average of ~100 KB, but most of the time there
are no writes. Here is an example:
> dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
65 0 31 3 0 0|6108k 939k| 0 0 | 14k 5372B|1510 4578
87 0 11 2 0 0| 22M 0 | 60B 842B| 0 0 |2206 7190
94 0 5 0 0 0| 33M 0 | 106B 400B| 0 0 |2398 7069
93 1 5 1 0 0| 32M 0 | 60B 346B| 0 0 |2229 20k
64 0 31 5 0 0| 11M 56k| 60B 346B| 0 0 |2235 6518
92 0 7 2 0 0| 744k 0 | 60B 362B| 0 0 |1923 3812
87 0 11 2 0 0|3244k 0 | 60B 346B| 0 0 |2085 3997
86 0 12 2 0 0|2236k 0 | 60B 346B| 0 0 |2056 3983
81 1 16 2 0 0| 21M 0 | 60B 346B| 0 0 |2244 5375
83 0 15 2 0 0| 31M 0 | 60B 346B| 0 0 |2454 7021
79 1 19 2 0 0| 32M 0 | 60B 346B| 0 0 |2478 11k
83 1 14 1 0 0| 49M 0 | 60B 346B| 0 0 |2607 13k
76 1 21 2 0 0| 45M 0 | 60B 346B| 0 0 |2487 15k
77 1 20 2 0 0| 30M 0 | 60B 346B| 0 0 |2356 20k
84 1 14 1 0 0| 28M 0 | 60B 346B| 0 0 |2332 11k
81 1 17 1 0 0| 37M 0 | 60B 346B| 0 0 |2314 11k
88 0 10 1 0 0| 27M 0 | 60B 346B| 0 0 |2304 5928
95 1 4 1 0 0| 31M 0 | 60B 346B| 0 0 |2362 5464
86 0 12 1 0 0| 34M 0 | 60B 346B| 0 0 |2346 6887
70 0 26 4 0 0| 968k 32k| 60B 346B| 0 0 |1957 4238
67 0 29 4 0 0|7892k 0 | 60B 362B| 0 0 |2177 10k
58 1 36 5 0 0|4836k 0 | 60B 346B| 0 0 |2317 12k
34 0 57 9 0 0|8752k 256k| 60B 346B| 0 0 |2225 9076
82 1 16 1 0 0| 39M 0 | 60B 362B| 0 0 |2366 14k
83 1 14 2 0 0| 42M 0 | 60B 346B| 0 0 |2487 14k
96 0 3 0 0 0| 25M 0 | 60B 346B| 0 0 |2216 4317
91 0 7 2 0 0| 12M 56k| 60B 346B| 0 0 |2003 3668
94 1 4 1 0 0| 17M 0 | 60B 362B| 0 0 |2048 7270
98 0 2 0 0 0| 12M 0 | 60B 346B| 0 0 |1974 3316
95 0 4 0 0 0| 30M 0 | 60B 346B| 0 0 |2104 11k
92 1 7 1 0 0| 18M 0 | 60B 346B| 0 0 |2107 5712
88 1 10 1 0 0| 25M 0 | 60B 346B| 0 0 |2275 8254
91 0 8 1 0 0| 13M 40k| 60B 346B| 0 0 |2090 6409
96 0 3 0 0 0| 28M 0 | 60B 362B| 0 0 |2342 5899
97 1 1 0 0 0| 28M 0 | 60B 346B| 0 0 |2337 3895
92 1 6 1 0 0| 25M 0 | 106B 400B| 0 0 |2200 9122
84 1 14 2 0 0| 29M 0 | 60B 346B| 0 0 |2129 12k
94 1 4 1 0 0| 29M 0 | 60B 346B| 0 0 |2363 10k
96 1 3 0 0 0| 20M 0 | 60B 346B| 0 0 |2117 11k
...
I think, it should run faster and wonder whether this is somehow related to
Postgres 9.0 and the different GIN creation strategy, cf.
http://gis.638310.n2.nabble.com/Patch-to-osm2pgsql-for-faster-updates-tp5952996p5952996.html
So I am interested to hear, what experiences others made in this step and
additionally, whether there is a way to estimate the "ttf" (time to finish)
of that process (maybe index sizes?).
Thanks
fatzopilot
--
View this message in context: http://gis.638310.n2.nabble.com/Superslow-nominatim-index-creation-with-Postgres-9-0-tp6180717p6210768.html
Sent from the Developer Discussion mailing list archive at Nabble.com.
More information about the dev
mailing list