[Tilesathome] Two new ROMA...

[Kai Krueger]

Thanks for setting up your ROMA instance. This should get us one big step closer 
to removing the data download as a bottleneck from t at h. At the same time it 
might actually help improve the response times of editing in Potlatch and JOSM 
too, by  keeping the t at h load off the main API server, which has become a 
significant bottleneck even for editing. So the effort of all the people who 
have set up ROMAs is greatly appreciated!

There is no special need for making it work with the load balancer. You just 
need to give Mathieu the go ahead and he will add your server to the load 
balancer configuration. The only option that needs setting and potentially some 
tuning is the maximum number of simultaneous requests under which your server 
achieves its highest throughput. Everything else the load balancer should handle 
by it self including health monitoring, so if you have to take the server 
offline again for a while or other things go wrong, clients going through the 
load balancer should not really notice.

Two things that might be useful to add to your configuration though  I think 
could be supporting compression and setting the character encoding to UTF-8. By 
adding either mod_deflate or mod_gzip to your apache config, the data can 
transparently be compressed which with XML data could save close to 10 fold on 
bandwidth. The t at h clients don't support compression quite yet, but it should be 
easy to enable and I think Patrick wanted to patch t at h to support it soon. The 
other thing is Character encoding. Your server, if I saw it correctly, returns a 
HTTP character-encoding of ISO-8859-1, whereas the actual data sent is correctly 
in utf-8. It is not a problem with the current t at h clients as it just ignores 
the character encoding, but it might become one once compression is enabled as 
the perl HTTP decoding routines also do character decoding/conversion if I 
remember correctly.

Kai


Milenko wrote:
> Thanks for the tips - I had some of them implemented already, but every little bit helps.
> 
> After making these changes along with the indexes that Justin suggested the query time for the bbox below is now down to 2.4 seconds.  That's just a little bit of an improvement over 130+ seconds...  :)
> 
> I have all of my clients running through it overnight - if all is well after a day or so then I think it'll be ready.
> 
> The server address is roma.king-nerd.com.  Feel free to hammer on it and see how it holds up.  Munin graphs are at http://roma.king-nerd.com/munin
> 
> Are there any special changes needed to interact with your load balancer?
> 
> -Jeremy
> 
> -----Original Message-----
> From: tilesathome-bounces at openstreetmap.org [mailto:tilesathome-bounces at openstreetmap.org] On Behalf Of Mathieu Arnold
> Sent: Wednesday, December 03, 2008 4:51 AM
> To: 'TilesAtHome'
> Subject: Re: [Tilesathome] Two new ROMA...
> 
> +--On 2 décembre 2008 19:49:13 -0500 Milenko <milenko at king-nerd.com> wrote:
> | I have been attempting to bring my own ROMA server online as yet another
> | ROMA option (YAROMAO??)
> 
> Once it's up, I'll just add it to the load balancer, and it'll rock.
> 
> | For the record, my server is a 2x 3.0 Ghz Xeon with 8GB RAM and 4x 150GB
> | Velociraptor drives in RAID0 on an LSI MegaRAID 8344ELP controller.
> | Bonnie++ shows block reads at 426MB/s and over 700 seeks/s.
> 
> Hum, mine is a, hum, dual p3 1.2GHz with 3.5GB of RAM, 4x36GB + 2x73GB
> drives in a somewhat raid0 configuration. I think bonnie would have said my
> box is far much less powerful than yours.
> 
> | While fetching bbox 37.221680,11.594560,37.397461,11.709906 I get data in
> | sub-30 seconds from your two servers while mine takes over 130 seconds.
> 
> Just tried on mine, it took 12 seconds.
> 
> | I'll be the first to admit I'm not a pgsql admin, so there hasn't been
> | much tweaking done to the config on my server.
> | 
> | Any help would be greatly appreciated.
> 
> First, what version of postgresql are you running ? What OS ?
> 
> The first thing you have to do, is to run ANALYZE on the database,
> otherwise, postgresql will still think your tables are mostly empty, and
> thus, will certainly do sequential scans instead of index scans. If that's
> enough, you don't have to do much more :-)
> Now, what have I done, in my postgresql.conf file...
> 
> shared_buffers = 128M (that is, you'll certainly have to raise up some SHM
> things in your OS, I can tell you how to do that with FreeBSD, but I have
> no idea for other OS)
> 
> temp_buffers = 128MB (that is not shared memory, so you can raise it
> without problems)
> 
> work_mem = 16MB (for sorting and aggregating, though we shouldn't need much)
> 
> maintenance_work_mem = 64MB
> 
> max_fsm_pages = 1000000
> max_fsm_relations = 1000
> 
> checkpoint_segments = 16 (because 3 just are not enough)
> 
> 
> 
> autovacuum = on
> log_autovacuum_min_duration = 0
> autovacuum_naptime = 60min
> autovacuum_vacuum_threshold = 100000
> autovacuum_analyze_threshold = 50000
> autovacuum_vacuum_scale_factor = 0.005
> autovacuum_analyze_scale_factor = 0.001
> 
> 
> And one last thing, I did a :
> enable_seqscan = off
> 
> which will tell the planer not to, never, ever, do a seq scan if it can do
> something else. (If it has no choice, it will still do a seq scan, but it
> will always prefer an index.)
> 
> All those directives are explained in detail in postgresql's documentation,
> just search google for them, it'l give you the right place.
>