[Openstreetmap-dev] Cache the whole world.

[Tom Carden]

Lars Aronsson wrote:
> Immanuel Scholz wrote:
>   
>> After reading Lars's measurements, I came up to the (probably very old)
>> idea of having the tile server cache the background images fetched from
>> NASA. ;)
>>     
>
> Yes, I think this has been suggested before.  But Steve also plans 
> to download the entire image dataset and generate satellite image 
> tiles locally, instead of calling NASA's server.  

No - the plan is to store a set of pre-tiled images, not generate them 
on the fly.  It will serve static files, and should be as fast as the 
current squid cache.  The OnEarth people are working on their own tiling 
scheme too, which when finished should allow us to synchronise with them 
and allow for easy updates.

> I have no idea 
> what effect that would have on performance.  I guess it depends on 
> how much CPU power is needed to generate the satellite image 
> tiles.  If it adds CPU load, your suggestion to cache satellite 
> image tiles might still make sense.  I also have no idea about 
> Steve's time plan or how we can help him.
>   

Generally, all things are time planned to happen "as soon as possible" 
(yes, this could be better) and as mentioned several times before if you 
want to help you can do so with good hardware, dedicated hosting or 
significant coding time.  That would be in addition to the help you 
already give by using the project, bug hunting, contributing data, and 
participating in the mailing list of course - all valuable!

> But still: Suppose that satellite image tile generation (at NASA) 
> takes 2 seconds and that drawing roads adds 1 second.  Serving a 
> tile from a Squid cache apparently takes 0.15 seconds.  Thus, 
> caching the satellite images could reduce the response time from 3 
> seconds to 1.15 seconds.  Our currently most important problem, 
> however, is that roads that should be drawn are only drawn in 18% 
> of the cases.  What we have is essentially a (more or less slow) 
> viewer for NASA's satellite images, not a viewer for OSM road 
> maps.  The slowness is bad, but it is not our worst problem.
>
>   
>> Lars: Can you rerun your script, this time not accessing the 
>> tileserver from OSM, but directly the URL from the NASA server 
>> where the tile server get its pictures from? And better: can you 
>> do this from a shell from the development server (which should 
>> be the same performance than the tile server ,since they are 
>> placed in the same network, right?)
>>     
>
> I see your point, but I'd like to see the 18% problem (ticket 134) 
> solved before we go further into performance issues.  And perhaps 
> Steve can tell us about the plans to store the satellite images 
> locally?
>
>   
>> Aren't the 48 hours because some people complaining that they 
>> are note able to correctly invalidate the cache and don't see 
>> their map changes drawn in OSM?
>>     
>
> The 48 hours is the worst kind of compromise.  It's too long when 
> updates are made, and it's too short when no updates are made.
> What we need is a way to invalidate the cache on updates.  Then 
> the expiration time can be set to two weeks, months or years.
> I don't see any trac ticket for this need.  I was almost certain 
> it had been suggested already.  Should I add a ticket?  Wasn't it 
> mentioned that Squid has some API for this?
>   

Mikel Maron is working on both these things, last I knew.  His recent 
posts to the dev list have mainly been about a) local landsat cache and 
b) cache refreshing on edit.

Best,

Tom.