[Tile-serving] [openstreetmap/osm2pgsql] --flat-node option causes >85 GB nodes file to be created with Liechtenstein extract. (Issue #1992)

[mboeringa]

<!-- Please don't use screenshots. Copy and paste the *text* output here if that's needed for context. -->

## What version of osm2pgsql are you using?

1.8.1-101-geaff0924


## What operating system and PostgreSQL/PostGIS version are you using?

2023-07-07 10:38:52  Database version: 15.3 (Ubuntu 15.3-1.pgdg22.04+1)
2023-07-07 10:38:52  PostGIS version: 3.3


## Tell us something about your system

Bare metal 512 GB RAM, 2x22 Intel Xeon E5-2699 v4


## What did you do exactly?

I've been testing some different command line options for osm2pgsql using the Geofabrik 'Liechtenstein' extract. Now I know that some of the options here do not make sense for such a tiny extract, _like specifying a flat-nodes file_, as the flat-nodes file is supposed to be only used on very large extracts or Planet, but they should technically work. 

In fact, _they all do work_ in the sense that the processing will successfully finish for all command lines listed below, but I noticed an unexpected anomaly when specifying the **--flat-nodes** option: Not only is the processing much slower and extended to > 5 minutes instead of under 15 seconds without **--flat-nodes**, but during the node loading stage of the osm2pgsql processing, I see the specified flat nodes file grow to an unexpectedly large and seemingly anomalous size of **87.4 GB** max.

I understand from the documentation and discussions I have seen before that the specific format of the flat nodes file is not really suited for processing small extracts, but this seems anomalous, the 'Liechtenstein' extract PBF is just 2.8 MB. Or is this expected behavior and a mere consequence of the specific flat nodes implementation?

**These command works, import within 15 seconds or less:**

`osm2pgsql -O flex -d micro -U osm --create -W -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim -C 1000 -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim --drop -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim --drop -C 1000 -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

**These commands "fail", import > 5 minutes, flat-nodes file of 87.4 GB(!) created:**

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim --flat-nodes 'data/nodes_micro.bin' -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim --flat-nodes 'data/nodes_micro.bin' -C 1000 -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim --drop --flat-nodes 'data/nodes_micro.bin' -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`

`osm2pgsql -O flex -d micro -U osm --tablespace-slim-data 'osm_e' --tablespace-slim-index 'osm_i' --create -W --slim --drop --flat-nodes 'data/nodes_micro.bin' -C 1000 -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/liechtenstein-latest.osm.pbf'`


## What did you expect to happen?

Using the **--flat-nodes** option may be sub-optimal and slower, but is not expected to result in a flat nodes file massively larger than the corresponding PBF file being imported.

## What did happen instead?

Seemingly anomalous big flat nodes file of 87.4 GB created.

## What did you do to try analyzing the problem?

Tried different command line options like **--slim**, **--drop** and **-C** to exclude these options were involved in the issue. Only the --flat-nodes setting seems to cause the issue.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1992
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/osm2pgsql/issues/1992 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20230707/154124f2/attachment.htm>