[OSM-talk] Installing your own tileserver on Ubuntu

[Kai Krueger]

Hi,

On 10/19/2011 05:01 AM, Joseph Reeves wrote:
> Hi Kai,
> 
>> The default Ram cache size is 800Mb. You can increase it with the -C
>> parameter of osm2pgsql. But given that your netbook probably doesn't
>> have that much ram, I am not sure increasing that option is such a good
>> idea.
> 
> -C only has an effect in slim mode, unless I'm wrong?

That also changed with the commit a few days ago. As again for smaller
extracts, who have a sparse distribution of node IDs, the cache was very
inefficient, I reused the improved node cache from the --slim mode.
Perhaps unfortunately, that also incorporated the limit and effect of
the -C parameter. On 64 bit machines at least, you can simply set the -C
parameter very high as it only reserves virtual memory. Only the amount
actually used will result in physical ram allocation. It is perhaps a
little bit more problematic on 32 bit Operating Systems though, as
virtual memory is also fairly limited.

Setting a very high -C parameter and --cache-strategy chunked basically
gets you back to the old behavior though.

 As I was hoping
> to append data to my db, I'm running osm2pgsql in full fat mode.
> Thanks for the --cache-strategy tip - I got the import working with
> the sparse option. It seems to be working surprisingly quickly in
> fact.

Fat mode definitely has its advantage in speed, perhaps especially on
slow disk systems like a netbook. This is perhaps why it was a bit
unfortunate that non-slim mode previously was (and for ways and
relations it still is) very wasteful with ram for extracts.

> 
> Of course, as you said, getting the data into the db is one thing, but
> actually using it is another matter.  The netbook is now rendering
> tiles for a large strip across Europe - this often takes a while to
> get tiles created, but I think it should work for what I need it for.
> If the load gets too high I can always empty the db and add extracts
> as I need them. Before that, however, I think I'll try and find any
> database optimisations that might exist.

Playing with indexes, especially partial indexes on the ways columns,
reflecting the where condition of the most expensive queries may help.

If you do come up with good optimizations, it may be good to collect
them somewhere to build up a knowledge base for optimization tips.

Kai

> 
> Thanks again for all this,
> 
> Cheers, Joseph
> 
> 
> 
> On 18 October 2011 18:34, Kai Krueger <kakrueger at gmail.com> wrote:
>> On 10/18/2011 10:48 AM, Joseph Reeves wrote:
>>>> It is possible that one could catch the errors in slim mode and then only do the expensive diff processing for those node / ways that are >duplicate in the extracts.
>>>
>>> Interesting, although I think this is beyond the limits of my OSM
>>> skills.
>>
>> Yes, that comment was more directed at my self that I should look into
>> that, or if any other dev of osm2pgsql gets to it first... ;-)
>>
>>  Unfortunately Austria seems to be beyond the capabilities of
>>> my netbook; an import without --slim gives the error:
>>>
>>> Node cache size is too small to fit all nodes. Please increase cache size
>>>
>>> Presumably a slim import would help, but this would then fail because
>>> of overlapping ways... I can't Google up anyone suffering that error
>>> message before; I guess nobody else is trying to get a number of
>>> European countries into a db on their netbook...
>>
>> You will likely not find that error message on google yet, as if I am
>> not mistaken, I only commited that error message two days ago.
>>
>> The default Ram cache size is 800Mb. You can increase it with the -C
>> parameter of osm2pgsql. But given that your netbook probably doesn't
>> have that much ram, I am not sure increasing that option is such a good
>> idea.
>>
>> Given that you can't fit Austria into 800Mb, I suspect you are using the
>> 32bit version of osm2pgsql. It defaults to the old (for extracts
>> inefficient) cache allocation strategy. You can try using the
>> "--cache-strategy" option and set it to either optimized or sparse. That
>> should be more efficient for extracts than the default method. In the
>> optimized option, you might run out of virtual memory on 32 bit though.
>>
>> But you might have troubles with Austria on a netbook anyway, as it
>> might still be too resource intense.
>>
>> Kai
>>
>>>
>>> Thanks again,
>>>
>>> Joseph
>>>
>>>
>>>
>>>
>>> On 18 October 2011 16:59, Kai Krueger <kakrueger at gmail.com> wrote:
>>>> On 10/18/11 9:31 AM, Joseph Reeves wrote:
>>>>>
>>>>> Hi Kai,
>>>>>
>>>>>> The pre-rendered tiles are stored in /var/lib/mod_tile/default. You can
>>>>>> simply delete those files and they will automatically get rerendered the
>>>>>> next time you view them.
>>>>>
>>>>> Great, thanks, that's working great.
>>>>>
>>>>>> I have seen that you appear to need to restart renderd (sudo
>>>>>> /etc/init.d/renderd restart) after a new import, as it otherwise appears
>>>>>> to
>>>>>> still use old data (It is kind of odd, so I might have the wrong
>>>>>> impression
>>>>>> here).
>>>>>
>>>>> Sorry, I should've said in my previous email - I was assuming this to
>>>>> be the case. Out of interest, is there any log output from renderd?
>>>>
>>>> sudo tail -f /var/log/syslog
>>>>
>>>> should give you some output of what renderd is doing, including which tiles
>>>> it is rendering and when it has completed them
>>>>>
>>>>> I'm running this on a netbook so tiles take a while to be produced;
>>>>
>>>> I'd imagine a netbook might struggle a little with rendering tiles on the
>>>> fly... ;-) But eventually they should be done.
>>>>>
>>>>>  it
>>>>> would be interesting to see some logs so that I know it's all working
>>>>> as I expect.
>>>>>
>>>>>> However, what you are trying to do is as far as I know not supported by
>>>>>> osm2pgsql. Although it seems to be a much requested feature, I don't
>>>>>> think
>>>>>> osm2pgsql currently handles importing of multiple extracts. The --append
>>>>>> option doesn't really do what you would think it does.
>>>>>
>>>>> I tried the append flag and got an error about an already existing way
>>>>> - it would be good if osm2pgsql would simply ignore any ways that
>>>>> already exist in the database. I re-ran osm2pgsql without the --slim
>>>>> option, however, and the import was successful. I currently have
>>>>> Bulgaria and Romania working on my netbook :)
>>>>
>>>> Interesting that the non-slim mode works with appending multiple extracts.
>>>>
>>>> It is possible that one could catch the errors in slim mode and then only do
>>>> the expensive diff processing for those node / ways that are duplicate in
>>>> the extracts.
>>>>>
>>>>> Am trying to re-import Turkey now, then onwards with bits of Europe!
>>>>> If it all works out do you mind if I do a bit of wiki fiddling on your
>>>>> instructions?
>>>>
>>>> No go ahead and improve the instructions
>>>>>
>>>>> Thanks again,
>>>>>
>>>>> Joseph
>>>>>
>>>>
>>>>
>>
>>