[Tilesathome] Proposal: Keeping tileset as one file

Kai Krueger kakrueger at gmail.com
Mon Jun 30 08:42:31 BST 2008


Dirk-Lüder Kreie wrote:
> spaetz schrieb:
>> On Thu, Jun 26, 2008 at 10:37:41AM +0200, Frederik Ramm wrote:
>>
>>> If client A uploads a tar with tiles 1, 2, 3, and later B uploads a 
>>> tar with tiles 10, 11, 3, then you have two choices:
>>
>> It seems a reasonable policy change to require the uploading of 
>> complete tilesets. The bottleneck is always going to be the server, so 
>> why not push as much work to the clients as possible and make it as 
>> easier for the server.
> 
> There are some problems with tilesets that even now are too big for the 
> current maximum upload size (but can be adapted in php.conf)
> Secondly the clients have a renderFullTileset option already which can 
> be set quite easily via svn.

if I have seen it correctly, the renderFullTileset option, just simply 
keeps all the empty tiles as files, rather than deleting them if the 
emptiness can be inferred recursively? That option would be quite useful 
for the single file per tileset implementation I think, and it should 
also simplify the serving of empty tiles, as one doesn't need to consult 
the database for those. In the tileset structure proposed, this would 
also not add much overhead, as each empty tile would only need 4 bytes, 
so a maximum of 5kb extra per tileset.

In the statistics posted earlier about the ratio of complete to 
incomplete tilesets, is it know why these tilesets are incomplete? What 
fraction is due to the empty tiles rather than not being able to upload 
as large tiles?

> 
>> If we mandate the uploading of full tilesets as tar files and it get's 
>> as easy for the server to process that as to move it to the 
>> appropriate location, then I'm all for it. I'm just not going to code it.
> 
> Same here. So apart from the coding of the serverside stuff we "just" 
> need to adapt the php.conf (or whatever then handles the uploading 
> itself) to allow for larger POST and upload files (I think there were 2 
> config options that had to be modified)
> and the clientside config option for maximum upload size (zipHardLimit 
> IIRC)

Over the weekend I have started writing an Apache module for serving 
tiles from within a tileset file. As I have never written an Apache 
module before, this is not the fastest endeavour though, but at least 
with the help of the mod_tile code, I have now managed to get to a 
stage, where it correctly serves blank tiles from within the tileset and 
is close to serving normal tiles as well. It also nicely falls back to 
the old code, if it doesn't find the tileset file. Unfortunately the 
code is still too ugly to post quite yet, but hopefully after a bit of 
re factoring, I can post an early version soon if people are interested.

Of cause, there is still a fair bit of plumbing needed to integrate this 
all into the upload handlers and tile statistics pages, but that should 
all be doable.

The thing I potentially see as biggest hurdle though is how to deal with 
deleting the old files. Although my current approach should work 
correctly without deleting files, that would then double the overall 
storage requirements, which can't work long term. On the other hand if 
you need to check each of the old tiles and delete them in the upload 
handler, then I could imagine a big portion of the benefits of having a 
single file per tileset is lost. Perhaps it could work to have a 
clean-up daemon periodically go through the list of recently uploaded 
tilesets and clean up duplicate files and recombine incomplete tilesets 
with individual tiles. Do people have other suggestions for this issue?


> 
>> Marking blank tiles could still be done by passing the appropriate 
>> blankness info through a special HTTP call.
> 
> however we will have all that info inside our "full Tileset" uploads.
> 
> we might want to run a simplifier for ocean-tiles from time to time.
> 
> or we need to calculate the overhead of storing the entire sea via such 
> files...

In the case of having only parts of a z12 tileset empty, I think the 
overhead shouldn't be too bad. I would think the big problem would be if 
one ended up storing them for the complete pacific ocean, but if an 
entire z12 tileset is empty, then there shouldn't be a need to store a 
tileset at all and fall back to the current recursive database lookup.

How does the current client deal with entire z12 being empty?

Kai

> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Tilesathome mailing list
> Tilesathome at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome





More information about the Tilesathome mailing list