[Tile-serving] [osm2pgsql-dev/osm2pgsql] Update behaviour after disk is full. (Discussion #2355)

[Marcos Dione]

I run out of disk this morning:

```
mdione at ioniq:~/src/projects/elevation/data/osm$ osm2pgsql-replication update --port 5432 --database europe
2025-07-12 13:07:33 [INFO]: Using replication service 'https://download.geofabrik.de/europe-updates'.
2025-07-12 13:11:05  osm2pgsql version 2.1.1
2025-07-12 13:11:05  Database version: 16.4 (Debian 16.4-3)
2025-07-12 13:11:05  PostGIS version: 3.4
2025-07-12 13:11:05  Loading properties from table '"public"."osm2pgsql_properties"'.
2025-07-12 13:11:05  Using flat node file '/home/mdione/src/projects/elevation/data/osm/europe.cache' (same as on import).
2025-07-12 13:11:05  Using output 'flex' (same as on import).
2025-07-12 13:11:05  Using style file '/home/mdione/src/projects/elevation/osm-carto/openstreetmap-carto-flex.lua' (same as on import).
2025-07-12 13:11:05  Storing properties to table '"public"."osm2pgsql_properties"'.
2025-07-12 13:11:05  Storing properties to table '"public"."osm2pgsql_properties"'.
2025-07-12 13:43:47  Reading input files done in 1962s (32m 42s).
2025-07-12 13:43:47    Processed 2953438 nodes in 335s (5m 35s) - 9k/s
2025-07-12 13:43:47    Processed 711976 ways in 291s (4m 51s) - 2k/s
2025-07-12 13:43:47    Processed 18907 relations in 1336s (22m 16s) - 14/s
2025-07-12 13:43:48  Going over 88258 pending ways (using 4 threads)
Left to process: 0.......
2025-07-12 13:45:13  Processing 88258 pending ways took 85s (1m 25s) at a rate of 1038.33/s
2025-07-12 13:45:13  Going over 60567 pending relations (using 4 threads)
Left to process: 38687...2025-07-12 14:05:59  ERROR: DB copy thread failed: Ending COPY mode for 'planet_osm_polygon' failed: ERROR:  could not extend file "base/87679982/91569895.60": No space left on device
HINT:  Check free disk space.
Left to process: 38676...2025-07-12 14:06:16 [CRITICAL]: Exception during execution: Command '['/usr/bin/osm2pgsql', '--append', '--slim', '--prefix', 'planet_osm', '-d', 'europe', '-P', '5432', '/tmp/tmpfkov18ov/osm2pgsql_diff.osc.gz']' died with <Signals.SIGSEGV: 11>.
```

This is expected since I am, well, running out of space :)

<img width="1664" height="521" alt="image" src="https://github.com/user-attachments/assets/007f4e14-8b6b-45cf-9e8f-7b7b98699c28" />

In that graph, we can see the correlation with some of the logs up there.

So, I free some space, and I try again. What surprises me is what happens then:

```
2025-07-12 20:00:02 [INFO]: Using replication service 'https://download.geofabrik.de/europe-updates'.
2025-07-12 20:02:17  osm2pgsql version 2.1.1
2025-07-12 20:02:17  Database version: 16.4 (Debian 16.4-3)
2025-07-12 20:02:17  PostGIS version: 3.4
2025-07-12 20:02:17  Loading properties from table '"public"."osm2pgsql_properties"'.
2025-07-12 20:02:17  Using flat node file '/home/mdione/src/projects/elevation/data/osm/europe.cache' (same as on import).
2025-07-12 20:02:17  Using output 'flex' (same as on import).
2025-07-12 20:02:17  Using style file '/home/mdione/src/projects/elevation/osm-carto/openstreetmap-carto-flex.lua' (same as on import).
2025-07-12 20:02:17  Storing properties to table '"public"."osm2pgsql_properties"'.
2025-07-12 20:02:17  Storing properties to table '"public"."osm2pgsql_properties"'.
Processing: Node(2953k 10.5k/s) Way(711k 2.84k/s) Relation(10010 10.9/s)
2025-07-12 20:26:30  ERROR: DB copy thread failed: Ending COPY mode for 'planet_osm_line' failed: ERROR:  could not extend file "base/87679982/91569896.125": No space left on device
HINT:  Check free disk space.
2025-07-12 20:26:31 [CRITICAL]: Exception during execution: Command '['/usr/bin/osm2pgsql', '--append', '--slim', '--prefix', 'planet_osm', '-d', 'europe', '-P', '5432', '/tmp/tmpj6hf90ab/osm2pgsql_diff.osc.gz']' returned non-zero exit status 2.
```

<img width="1664" height="521" alt="image" src="https://github.com/user-attachments/assets/7650b2e6-e60f-422d-8918-e445c6c851e5" />

Yes I run out of space again.

So my question is: how does the update process works? In particular, I would expect a second pass not to use more space until it hits the place where it left before. This clearly is wrong, so I don't understand how the update process. Moreover, this is not the first (and not the last, until I buy a new SSD) that I run out of space, so I wonder if there is some cleanup that can be done before retrying and update.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2355
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2355 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20250712/5e977582/attachment.htm>