[Tile-serving] [openstreetmap/osm2pgsql] Improve the human readility of the `osm2pgsql-replication` (PR #1975)

Amanda notifications at github.com
Tue Jun 27 15:21:10 UTC 2023 

I've made the changes as requested (& merged all into one commit). Most of the new output is only at `DEBUG` level, and uses fewer lines.

w/o `-v` the output looks like this:

```
2023-06-27 16:59:04 [INFO]: Using replication service 'http://download.geofabrik.de/europe/malta-updates'.
2023-06-27 16:59:05  osm2pgsql version 1.8.0
2023-06-27 16:59:05  Database version: 15.3 (Debian 15.3-1.pgdg120+1)
2023-06-27 16:59:05  PostGIS version: 3.3
2023-06-27 16:59:05  Setting up table 'planet_osm_point'
2023-06-27 16:59:05  Setting up table 'planet_osm_line'
2023-06-27 16:59:05  Setting up table 'planet_osm_polygon'
2023-06-27 16:59:05  Setting up table 'planet_osm_roads'
2023-06-27 16:59:05  Reading input files done in 0s.                                      
2023-06-27 16:59:05    Processed 2050 nodes in 0s - 2k/s
2023-06-27 16:59:05    Processed 482 ways in 0s - 482/s
2023-06-27 16:59:05    Processed 14 relations in 0s - 14/s
2023-06-27 16:59:05  Going over 263 pending ways (using 4 threads)
Left to process: 0.....
2023-06-27 16:59:07  Processing 263 pending ways took 2s at a rate of 131.50/s
2023-06-27 16:59:07  Going over 130 pending relations (using 4 threads)
Left to process: 0.....
2023-06-27 16:59:09  Processing 130 pending relations took 2s at a rate of 65.00/s
2023-06-27 16:59:09  Done postprocessing on table 'planet_osm_nodes' in 0s
2023-06-27 16:59:09  Done postprocessing on table 'planet_osm_ways' in 0s
2023-06-27 16:59:09  Done postprocessing on table 'planet_osm_rels' in 0s
2023-06-27 16:59:09  All postprocessing on table 'planet_osm_point' done in 0s.
2023-06-27 16:59:09  All postprocessing on table 'planet_osm_line' done in 0s.
2023-06-27 16:59:09  All postprocessing on table 'planet_osm_polygon' done in 0s.
2023-06-27 16:59:09  All postprocessing on table 'planet_osm_roads' done in 0s.
2023-06-27 16:59:09  osm2pgsql took 4s overall.
2023-06-27 16:59:10 [INFO]: Data imported until 2023-06-26T20:21Z. Backlog remaining: 18 hour(s) 37 minute(s)
```

w/ `-v`:

```
2023-06-27 17:01:44 [INFO]: Using replication service 'http://download.geofabrik.de/europe/malta-updates'.
2023-06-27 17:01:44 [DEBUG]: Starting new HTTP connection (1): download.geofabrik.de:80
2023-06-27 17:01:44 [DEBUG]: http://download.geofabrik.de:80 "GET /europe/malta-updates/state.txt HTTP/1.1" 200 124
2023-06-27 17:01:44 [DEBUG]: Applying 4 sequence(s) (3737 → 3733), covering 4 day(s) 0 hour(s) 0 minute(s) (345623 sec) of changes (2023-06-22T20:21Z → 2023-06-26T20:21Z)
2023-06-27 17:01:44 [DEBUG]: Calling osm2pgsql with: osm2pgsql --append --slim --prefix planet_osm /tmp/tmpgg77v9dm/osm2pgsql_diff.osc.gz
2023-06-27 17:01:44 [DEBUG]: Importing from sequence 3733
2023-06-27 17:01:44 [DEBUG]: Starting new HTTP connection (1): download.geofabrik.de:80
2023-06-27 17:01:44 [DEBUG]: http://download.geofabrik.de:80 "GET /europe/malta-updates/state.txt HTTP/1.1" 200 124
2023-06-27 17:01:44 [DEBUG]: Starting new HTTP connection (1): download.geofabrik.de:80
2023-06-27 17:01:44 [DEBUG]: http://download.geofabrik.de:80 "GET /europe/malta-updates/000/003/734.osc.gz HTTP/1.1" 200 17009
2023-06-27 17:01:44 [DEBUG]: Downloaded change 3734. (511937 kB available in download buffer)
2023-06-27 17:01:44 [DEBUG]: Starting new HTTP connection (1): download.geofabrik.de:80
2023-06-27 17:01:44 [DEBUG]: http://download.geofabrik.de:80 "GET /europe/malta-updates/000/003/735.osc.gz HTTP/1.1" 200 28481
2023-06-27 17:01:44 [DEBUG]: Downloaded change 3735. (511830 kB available in download buffer)
2023-06-27 17:01:44 [DEBUG]: Starting new HTTP connection (1): download.geofabrik.de:80
2023-06-27 17:01:44 [DEBUG]: http://download.geofabrik.de:80 "GET /europe/malta-updates/000/003/736.osc.gz HTTP/1.1" 200 16914
2023-06-27 17:01:44 [DEBUG]: Downloaded change 3736. (511766 kB available in download buffer)
2023-06-27 17:01:44 [DEBUG]: Starting new HTTP connection (1): download.geofabrik.de:80
2023-06-27 17:01:44 [DEBUG]: http://download.geofabrik.de:80 "GET /europe/malta-updates/000/003/737.osc.gz HTTP/1.1" 200 13382
2023-06-27 17:01:44 [DEBUG]: Downloaded change 3737. (511722 kB available in download buffer)
2023-06-27 17:01:44  osm2pgsql version 1.8.0
2023-06-27 17:01:44  Database version: 15.3 (Debian 15.3-1.pgdg120+1)
2023-06-27 17:01:44  PostGIS version: 3.3
2023-06-27 17:01:44  Setting up table 'planet_osm_point'
2023-06-27 17:01:44  Setting up table 'planet_osm_line'
2023-06-27 17:01:44  Setting up table 'planet_osm_polygon'
2023-06-27 17:01:44  Setting up table 'planet_osm_roads'
2023-06-27 17:01:45  Reading input files done in 1s.                                      
2023-06-27 17:01:45    Processed 2050 nodes in 1s - 2k/s
2023-06-27 17:01:45    Processed 482 ways in 0s - 482/s
2023-06-27 17:01:45    Processed 14 relations in 0s - 14/s
2023-06-27 17:01:45  Going over 263 pending ways (using 4 threads)
Left to process: 0.....
2023-06-27 17:01:47  Processing 263 pending ways took 2s at a rate of 131.50/s
2023-06-27 17:01:47  Going over 130 pending relations (using 4 threads)
Left to process: 0.....
2023-06-27 17:01:49  Processing 130 pending relations took 2s at a rate of 65.00/s
2023-06-27 17:01:49  Done postprocessing on table 'planet_osm_nodes' in 0s
2023-06-27 17:01:49  Done postprocessing on table 'planet_osm_ways' in 0s
2023-06-27 17:01:49  Done postprocessing on table 'planet_osm_rels' in 0s
2023-06-27 17:01:49  All postprocessing on table 'planet_osm_point' done in 0s.
2023-06-27 17:01:49  All postprocessing on table 'planet_osm_line' done in 0s.
2023-06-27 17:01:49  All postprocessing on table 'planet_osm_polygon' done in 0s.
2023-06-27 17:01:49  All postprocessing on table 'planet_osm_roads' done in 0s.
2023-06-27 17:01:49  osm2pgsql took 4s overall.
2023-06-27 17:01:49 [DEBUG]: Starting new HTTP connection (1): download.geofabrik.de:80
2023-06-27 17:01:49 [DEBUG]: http://download.geofabrik.de:80 "GET /europe/malta-updates/000/003/737.state.txt HTTP/1.1" 200 124
2023-06-27 17:01:49 [INFO]: Data imported until 2023-06-26T20:21Z. Backlog remaining: 18 hour(s) 39 minute(s)
2023-06-27 17:01:49 [DEBUG]: It took <1 minute (5 sec) to apply 4 day(s) 0 hour(s) 0 minute(s) (345623 sec) of changes. This is a speed of ×65566.7.
```

-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/1975#issuecomment-1609733470
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/osm2pgsql/pull/1975/c1609733470 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20230627/5147bcdd/attachment-0001.htm>

More information about the Tile-serving mailing list