[Tile-serving] [osm2pgsql] Increase dense block size from 1k to 8k nodes (#441)

[Paul Norman]

The dense cache has a fixed memory cost which is associated with the number of blocks needed. Increase the block size reduces the number of blocks, resulting in reduced memory usage. For this change on modern hardware it is a decrease from 1GB to 128MB. Large blocks are less efficient than small ones, but it turns out that this does not matter for planet imports, where the efficiency change is small and memory usage is actually slightly reduced, or for small imports, where the fixed cost change overwhelms anything else.

On large but not planet sized extracts (e.g. Europe) memory usage is slightly increased, by about 450MB, while the cache in total consumes about 18GB.

Changing the cache size did not have a measurable impact on performance in testing.

Fixes #439

----

I had considered other cache sizes, but

* Going larger is into diminishing returns
* The size chosen is close to minimum planet memory usage, and at a minimum for the Canada test
* Each increase of BLOCK_SHIFT beyond 13 increases memory usage by about 350MB for a Europe extract and about 100MB for the planet.
* 130MB cache overhead is sufficiently low for VMs I'd consider capable of running a rendering server, even on a very small extract

-----

### Data ###
![image](https://cloud.githubusercontent.com/assets/1190866/9321130/2eae9f72-4517-11e5-9f88-e0ca0f1d79d7.png)

![image](https://cloud.githubusercontent.com/assets/1190866/9321057/426d4fc8-4516-11e5-93b4-7eedb22a007f.png)

Memory usage includes queue overhead, which scales with cache size, but is only 3.3 MB for a 27GB cache and 8k node dense blocks, so can be mostly ignored.
You can view, comment on, or merge this pull request online at:

  https://github.com/openstreetmap/osm2pgsql/pull/441

-- Commit Summary --

  * Increase dense block size from 1k to 8k nodes

-- File Changes --

    M node-ram-cache.cpp (2)
    M tests/middle-tests.cpp (2)

-- Patch Links --

https://github.com/openstreetmap/osm2pgsql/pull/441.patch
https://github.com/openstreetmap/osm2pgsql/pull/441.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/441
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150817/a31a332e/attachment-0001.html>