[OSM-dev] osm2pgsql: mixing normal and slim mode

[Frederik Ramm]

Hi,

    currently you can run osm2pqgsl in one of two modes - "normal" or 
"slim". Normal mode puts all temporary data in RAM. Slim mode puts all 
temporary data into the PostGIS database, which, as a "side effect", 
allows applying diffs later:

Mode                   | temp. data     | database ready for diffs
-----------------------+----------------+-------------------------
normal                 | in memory      | no
slim                   | on disk (db)   | yes
-----------------------+----------------+-------------------------

There is no compelling reason why these two features must be mixed; it's 
for historical reasons really. One could just as well have

Mode                   | temp. data     | database ready for diffs
-----------------------+----------------+-------------------------
normal-tmpfiles        | on disk (file) | no
slim-fast              | in memory      | yes
-----------------------+----------------+-------------------------

The "normal-tmpfiles" mode would allow people with less RAM to do a 
"non-slim" import, using some optimised disk storage (e.g. simply a 
mmap()ed file, or a library like stxxl), and thus getting results much 
faster than the current slim mode which they're forced to use. Or is 
there reason to believe that simply using the existing normal mode and 
letting the system eat into swapspace would give similar results?

The "slim-fast" mode would be a way for users with lots of memory to 
utilize this fully when doing an import. Choosing a very high value for 
the node cache in slim mode (-C) already goes a long way in this 
direction but there would have to be a "way cache" as well to make this 
complete.

Has anyone thought about these things (and maybe found them to be 
nonsense), or actually coded anything in that direction?

Bye
Frederik