[Tile-serving] [openstreetmap/osm2pgsql] Index only mode (#799)
Roman V. Nikolaev
notifications at github.com
Thu Nov 9 12:09:44 UTC 2017
Please add -- index-only mode to existing --append and --create modes.
I try load osm planet. The data has been uploaded. But at the stage of indexation, there was a сrash. Probably out of memory.
It would be desirable not to load again all the data (a few days), to complete the stage of indexing.
Log:
```
sudo -u tirex osm2pgsql --create \
--keep-coastlines \
--output pgsql --database gis \
--hstore-all --hstore-add-index \
--slim --cache 32110 --number-processes 7 \
--cache-strategy optimized \
--flat-nodes /var/lib/flat_nodes/nodes.cache \
--style "/srv/osm/config/carto/openstreetmap-carto.style" \
"/srv/osm/src/planet-latest.osm.pbf"
osm2pgsql version 0.92.0 (64 bit id space)
Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=32110MB, maxblocks=513760*65536, allocation method=11
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Allocated space for persistent node cache file
Maximum node in persistent node cache: 0
Mid: pgsql, scale=100 cache=32110
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /srv/osm/src/planet-latest.osm.pbf
Using PBF parser.
Processing: Node(4160579k 2999.7k/s) Way(449114k 9.02k/s) Relation(1411380 33.78/s)
Standard exception processing relation id=2509838: TopologyException: side location conflict at 2029381.6000000001 6575291.75
Processing: Node(4160579k 2999.7k/s) Way(449114k 9.02k/s) Relation(3583520 48.28/s)
Standard exception processing relation id=5630039: TopologyException: side location conflict at 1987071.54 6601270.3499999996
Processing: Node(4160579k 2999.7k/s) Way(449114k 9.02k/s) Relation(4980590 55.02/s)
Standard exception processing relation id=7301465: TopologyException: side location conflict at 8275559.8899999997 6486836.4299999997
Processing: Node(4160579k 2999.7k/s) Way(449114k 9.02k/s) Relation(5337580 56.38/s) parse time: 145842s
Node stats: total(4160579385), max(5199168802) in 1387s
Way stats: total(449114431), max(536786282) in 49789s
Relation stats: total(5337586), max(7692433) in 94666s
Maximum node in persistent node cache: 5199888383
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Maximum node in persistent node cache: 5199888383
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Maximum node in persistent node cache: 5199888383
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Maximum node in persistent node cache: 5199888383
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Maximum node in persistent node cache: 5199888383
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Maximum node in persistent node cache: 5199888383
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Maximum node in persistent node cache: 5199888383
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /var/lib/flat_nodes/nodes.cache
Maximum node in persistent node cache: 5199888383
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Going over pending ways...
316151498 ways are pending
Using 7 helper-processes
Finished processing 316151498 ways in 56237 s
316151498 Pending ways took 56237s at a rate of 5621.77/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending relations...
0 relations are pending
Using 7 helper-processes
Finished processing 0 relations in 0 s
Committing transaction for planet_osm_point
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING: there is no transaction in progress
Sorting data and creating indexes for planet_osm_point
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_roads
Stopping table: planet_osm_ways
Building index on table: planet_osm_ways
Stopping table: planet_osm_nodes
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Killed
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/799
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20171109/e09b3beb/attachment.html>
More information about the Tile-serving
mailing list