[Tilesathome] Proposal: Keeping tileset as one file

[Kai Krueger]

Florian Lohoff wrote:
> On Sat, Jun 14, 2008 at 03:14:34PM +0200, Frederik Ramm wrote:
>> * Will tile download be done with PHP (has this been covered in
>>   the benchmarks - compare executing PHP code to extract data from
>>   a large tile with having Apache load the tile directly), or 
>>   will we have an Apache module for that?
> 
> I thought about this - Basically the apache module could be storage
> agnostic. This means we get a tile name/offset/coordinates and ask a in
> memory index database which returns a filename, offset and length to serve.
> 
> So the only software which needs to be storage aware would be the
> indexer which takes the uploaded tileset and recreates/updates parts of
> the index.

I am not sure I fully understand your proposal, so maybe it is very 
similar or better, but the structure proposed by Jiri nd described at 
http://wiki.openstreetmap.org/index.php/Tiles%40home/Tileset_as_one_file 
seems pretty reasonable. For this there is no need for an index 
database and as such should keep the module very simple.

All the module has to do is check that the zoom level is between 12 and 
17. If so, calculate the z12 tileset coordinates. Check if a full 
tileset file exists. If any of these fail, fall-back to what the server 
does now. If all the checks succeed, then the offset within this file 
can be calculated with reading about 8bytes from the beginning of the 
file (hopefully cached due to common access). So overall I would guess
the overhead should be minimal and be outwayed by the benefits shown in 
the previous artificial benchmarks. I guess how well it really performs 
one can only tell once one has implemented it, but given the current 
status of the tah server which seems to not even keep up with the 
changed tiles let alone the huge priority 3 queue, I think it would be 
worth trying it out.

I cannot guarantee anything yet, but I hope to at least attempt to 
implement this in case no one else gives it a try. It doesn't seem to 
hard. But given I have never written an Apache module, maybe I am simply 
to naive ;-)

> 
> I thought about it and my idea was to let t at h clients upload tilesets as
> a tar.gz. The tileserver then would simply gunzip the tar and let the
> tar exist as the storage. So the indexer would read tars and the apache
> module would serve from the tars.

Is it necessary to zip any of this? The PNGs should be close to 
optimally compressed already anyway and the extra header at the 
beginning is only 5.3kb for a z12-z17 tileset, so even if that could be 
compressed, overall the gain would probably be less than a percent.

In this case, the upload code for a complete tileset would simply be 
putting the complete file at the correct position. No extra processing 
is needed at all. (apart from updating the request queue and user 
stats). Dealing with incomplete tilesets shouldn't be too difficult 
either. If there is no tileset file for the incomplete tileset, then can 
simply use the current upload mechanism and the fall-back code in the 
tile serving will automatically take care of it. In this case you don't 
get the benefits of the single tileset file, but also no additional 
overhead is occurred and hardly any extra code is needed. If there is a 
tileset file for the incomplete tileset, then one can still use the same 
old mechanism, but would have to mark all tiles in the new tileset 
invalid in the index of the old tileset file.

The changes to the client would still be very easy. I have written a 
quick perl script to take the temp directory created by the current tah 
client and create a tileset file from it. So this could be hooked into 
the client before uploading.


So I think all of the problems Frederik mentioned should be solvable, 
but who knows what shit hits the fan once one tries to code things up ;-)


Kai


> 
> If it makes sense to even concatenate tars to even large files needs to
> be tried but the frontend code in the apache module does not need to
> know.
> 
> The module could even fall through to alternative storage in case the
> index does not know about the tile (transition to storage-ng)
> 
> As a starting base one could have a look at the apache mod_ziplook
> which enables apache to serve from zip files directly.
> 
> http://pihl.kumpu.org/ziplook/
> 
> Flo
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Tilesathome mailing list
> Tilesathome at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome