[Tilesathome] How often...

[ben]

80n schrieb:
> On Tue, Feb 10, 2009 at 8:07 AM, Florian Lohoff <flo at rfc822.org 
> <mailto:flo at rfc822.org>> wrote:
> 
>     On Tue, Feb 10, 2009 at 07:47:12AM +0000, 80n wrote:
>      > How well will mapnik scale under load?
>      >
>      > As I understand it, most mapnik tiles are generated on-the-fly.
>      This is
>      > fine when you have a big server and only a few users.  But when
>     you have a
>      > very large number of users (think Google maps) then would the mapnik
>      > approach cope with the load?
>      >
>      > It seems to me that T at H is likely to be more scaleable when there
>     are large
>      > numbers of users. The T at H machine will be mainly concerned with
>     serving
>      > images to the user rather than rendering and then serving.
> 
>     My guess is that mapnik can scale very good with little code added.
>     With the CPU power beeing the limit you can either limit the maximum
>     number of requests or when appropriate cacheing is applied limit the
>     currentness of the map.
> 
>     Pseudo code like this:
> 
>            if (!datachanged() && cachedtileavail) {
>                    send(cached-older-tile)
>            } else {
>                    if (length(renderqueue)*timetorender > 500ms) {
>                            queuererenderrequest(prio 2)
>                            if (cachedtileavail)
>                                    send(cached-older-tile)
>                            else
>                                    send(More OSM Coming)
>                            return
>                    }
>                    rendertile(prio 1)
>                    updatecachewithnewtile
>                    send(new-tile)
>            }
> 
>     Then one could use a squid to proxy to multiple mapnik backends based
>     on URL (e.g. tile xy number - read: geographic region) so one could
>     use a couple smaller machines which do not need to keep the full
>     database which makes the working set smaller and thus the database
>     faster. Once you are at google scale you have a machine per Z12 ;)
> 
>     Annother approach would be to seperate database, render engine and web
>     frontend. Rendering is typically CPU intensive and the database is
>     memory and io bandwidth intensive - So the renderer would have just a
>     "normal" amount of memory like 1-2 GB but tons of cores. The Database
>     would have tons of memory and tons of spindles but not necessarily a lot
>     of CPU.
> 
>     The only problem with mapnik would be a tons of user panning around
>     on previously unrendered territory which obviously cant happen to t at h.
> 
> 
> Well, yes, that is the scenario that I'm describing.  What percentage of 
> the planet's tiles have actually been pre-rendered on the mapnik tile 
> server?
> 

I thought mapnik only renders the hourly diffs, not the tiles being 
requested?