[Tile-serving] [openstreetmap/osm2pgsql] Why ways importing speed depends on imported nodes? Is it memory allocation bug? (#1430)

[Valerij Ivashchenko]

I'v tested import on [Chicago extract (51MB) from bbbike in pbf format](https://download.bbbike.org/osm/bbbike/Chicago/Chicago.osm.pbf).

The result is above: 652 - 26 (k/s for nodes and ways).

Then I extract only ways from file (` osmconvert --drop-nodes --drop-relations --out-pbf -o=$RESULT_NAME-nodes.osm.pbf $OSM_SOURCE`) and run import again and I have the results: 0 - 9 .

Why for file with nodes ways imported almost x3 faster then for file without nodes?

To exclude the possibility of influence of the pbf-file format I extracted nodes from source file (`osmconvert --drop-ways --drop-relations --out-pbf -o=$RESULT_NAME-nodes.osm.pbf $OSM_SOURCE`) and then merge nodes with ways (`osmium cat -o $RESULT_FILE -f osm.pbf,add_metadata=false $NODES_FILE $WAYS_FILE
`) and have the same results like for source file (652 - 26).

Also I've tested import source file without cache  (`--cache=0`) and I've got ways import speed 9k/s, like for only ways files.

Its seems like ways buffer depends on nodes buffer and it not used when imported pbf-file without nodes.

At the same time, when I import full planet file, I've got ways import speed just 0.89k/s. But when I extract only ways from planet file I'v got speed 9-12 k/s for ways.

Any ideas why is this happens?

Any suggestion how to debug where is the bottleneck and how to avoid it?

## My environment:
```
# osm2pgsql --version
2021-03-19 20:11:36  osm2pgsql version 1.4.1 (1.4.1-16-gf7993c95)
Compiled using the following library versions:
Libosmium 2.16.0
Proj [API 4] Rel. 6.3.1, February 10th, 2020
Lua 5.2.4
```

```
# sudo --user=postgres osm2pgsql --create --slim --database=gis --merc --hstore-all --style=hstore-only.style --tag-transform-script=openstreetmap-carto.lua --prefix=planet_osm_hstore --number-processes=$NumberProcesses --cache=$CacheSizeMb $OSM_EXTRACT
```
```
CacheSizeMb=16000
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1430
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20210319/24d2d7f8/attachment.htm>