[Tile-serving] [openstreetmap/osm2pgsql] Docker imports the planet is really slow (Discussion #1787)
myrainbowandsky
notifications at github.com
Fri Sep 30 04:04:10 UTC 2022
system: Ubuntu 20.2
How to rocket the planet importation speed?
I am using docker to import planet like this:
```
docker volume create openstreetmap-data
time docker run -v /home/renderaccount/zambia-latest.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/12/main overv/openstreetmap-tile-server:1.3.10 import
```
But the importation speed is really slow like this:
```
+ '[' 1 -ne 1 ']'
+ '[' import = import ']'
+ chown postgres:postgres -R /var/lib/postgresql
+ '[' '!' -f /var/lib/postgresql/12/main/PG_VERSION ']'
+ createPostgresConfig
+ cp /etc/postgresql/12/main/postgresql.custom.conf.tmpl /etc/postgresql/12/main/conf.d/postgresql.custom.conf
+ sudo -u postgres echo 'autovacuum = on'
+ cat /etc/postgresql/12/main/conf.d/postgresql.custom.conf
# Suggested minimal settings from
# https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/
shared_buffers = 128MB
min_wal_size = 1GB
max_wal_size = 2GB
maintenance_work_mem = 256MB
# Suggested settings from
# https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45
max_connections = 250
temp_buffers = 32MB
work_mem = 128MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000
# checkpoint_segments = 60 # unrecognized in psql 10.7.1
max_wal_size = 2880MB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02
listen_addresses = '*'
autovacuum = on
+ service postgresql start
* Starting PostgreSQL 12 database server [ OK ]
+ sudo -u postgres createuser renderer
createuser: error: creation of new role failed: ERROR: role "renderer" already exists
+ sudo -u postgres createdb -E UTF8 -O renderer gis
createdb: error: database creation failed: ERROR: database "gis" already exists
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
ERROR: extension "postgis" already exists
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
ERROR: extension "hstore" already exists
+ sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
ALTER TABLE
+ sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
ALTER TABLE
+ setPostgresPassword
+ sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '\''renderer'\'''
ALTER ROLE
+ '[' '!' -f /data.osm.pbf ']'
+ '[' disabled = enabled ']'
+ '[' -f /data.poly ']'
+ sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes 4 -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
osm2pgsql version 1.2.0 (64 bit id space)
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
Mid: pgsql, cache=800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using lua based tag processing pipeline with script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Reading in file: /data.osm.pbf
Using PBF parser.
Processing: Node((base) wentao at white:~$ sudo docker logs 07feb9c51fc8 | grep pattern
(base) wentao at white:~$ sudo docker logs 07feb9c51fc8 | grep pattern
(base) wentao at white:~$ sudo docker logs 07feb9c51fc8 -f
+ '[' 1 -ne 1 ']'
+ '[' import = import ']'
+ chown postgres:postgres -R /var/lib/postgresql
+ '[' '!' -f /var/lib/postgresql/12/main/PG_VERSION ']'
+ createPostgresConfig
+ cp /etc/postgresql/12/main/postgresql.custom.conf.tmpl /etc/postgresql/12/main/conf.d/postgresql.custom.conf
+ sudo -u postgres echo 'autovacuum = on'
+ cat /etc/postgresql/12/main/conf.d/postgresql.custom.conf
# Suggested minimal settings from
# https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/
shared_buffers = 128MB
min_wal_size = 1GB
max_wal_size = 2GB
maintenance_work_mem = 256MB
# Suggested settings from
# https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45
max_connections = 250
temp_buffers = 32MB
work_mem = 128MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000
# checkpoint_segments = 60 # unrecognized in psql 10.7.1
max_wal_size = 2880MB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02
listen_addresses = '*'
autovacuum = on
+ service postgresql start
* Starting PostgreSQL 12 database server [ OK ]
+ sudo -u postgres createuser renderer
createuser: error: creation of new role failed: ERROR: role "renderer" already exists
+ sudo -u postgres createdb -E UTF8 -O renderer gis
createdb: error: database creation failed: ERROR: database "gis" already exists
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
ERROR: extension "postgis" already exists
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
ERROR: extension "hstore" already exists
+ sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
ALTER TABLE
+ sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
ALTER TABLE
+ setPostgresPassword
+ sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '\''renderer'\'''
ALTER ROLE
+ '[' '!' -f /data.osm.pbf ']'
+ '[' disabled = enabled ']'
+ '[' -f /data.poly ']'
+ sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes 4 -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
osm2pgsql version 1.2.0 (64 bit id space)
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
Mid: pgsql, cache=800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using lua based tag processing pipeline with script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Reading in file: /data.osm.pbf
Using PBF parser.
Processing: Node(7417817k 157.1k/s) Way(19416k 0.05k/s) Relation(0 0.00/s)
```
--
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/discussions/1787
You are receiving this because you are subscribed to this thread.
Message ID: <openstreetmap/osm2pgsql/repo-discussions/1787 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20220929/068b4566/attachment.htm>
More information about the Tile-serving
mailing list