[Tile-serving] [osm2pgsql-dev/osm2pgsql] Question: could not extend file "base/361191441/368615606.137": No space left on device (Discussion #2093)

[Klaus Tockloth]

Update 1:

The server used has 128728.9 MiB of RAM. The import without the options '--slim, --drop, --cache' was not successful. All the free disk space was used up by the (parallel) indexing operations:

* Estimated size of database: about 510 GB
* Free and available disk space: 705 GB

```
...
2023-10-18 18:09:27  Analyzing table 'planet_osm_roads'...
2023-10-18 18:37:32  Analyzing table 'planet_osm_point'...
2023-10-18 18:48:51  All postprocessing on table 'planet_osm_point' done in 8625s (2h 23m 45s).
2023-10-18 20:16:14  All postprocessing on table 'planet_osm_roads' done in 6266s (1h 44m 26s).
2023-10-18 20:16:14  ERROR: Database error: ERROR:  could not extend file "base/374190716/380929184.70": No space left on device
...
```

```
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        2.0T  1.2T  705G  63% /               <- Postgres
/dev/md3        5.3T  797G  4.3T  16% /home           <- osm2pgsql
...
```

However, omitting the '--slim, --drop, --cache' options significantly sped up the 'pure' import of raw data (6h 25m 8s versus 55h 47m 4s):

With '--slim, --drop, --cache':
```
...
2023-10-12 04:23:58  Reading input files done in 200824s (55h 47m 4s).                    
2023-10-12 04:23:58    Processed 8658483163 nodes in 3559s (59m 19s) - 2433k/s
2023-10-12 04:23:58    Processed 971692934 ways in 30443s (8h 27m 23s) - 32k/s
2023-10-12 04:23:58    Processed 11340554 relations in 166822s (46h 20m 22s) - 68/s
...
```


Without '--slim, --drop, --cache':
```
...
2023-10-18 16:25:02  Reading input files done in 23108s (6h 25m 8s).                      
2023-10-18 16:25:02    Processed 8672123611 nodes in 3183s (53m 3s) - 2725k/s
2023-10-18 16:25:02    Processed 973116158 ways in 16880s (4h 41m 20s) - 58k/s
2023-10-18 16:25:02    Processed 11372514 relations in 3045s (50m 45s) - 4k/s
...
```

Import command:
```
nohup ./osm2pgsql-1.9.2 --username postgres --multi-geometry --hstore --flat-nodes nodes.bin --create --number-processes 10 --database osmplanet --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua --input-reader=pbf planet-latest.osm.pbf >load_osmplanet.out 2>&1 &
```

Next step:

* Import command (as above) with additional option '--disable-parallel-indexing'.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2093#discussioncomment-7323250
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2093/comments/7323250 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231019/f4abeed7/attachment.htm>