[Tilesathome] Proposal: Keeping tileset as one file

[Florian Lohoff]

On Thu, Jun 26, 2008 at 12:36:57AM +0100, Kai Krueger wrote:
> I am not sure I fully understand your proposal, so maybe it is very 
> similar or better, but the structure proposed by Jiri nd described at 
> http://wiki.openstreetmap.org/index.php/Tiles%40home/Tileset_as_one_file 
> seems pretty reasonable. For this there is no need for an index 
> database and as such should keep the module very simple.

Basically i propose to not use a special file format with an index at
the very beginning (which has the problem of the incomplete tileset)
but rather use any uncompressed file format which has a contigous
storage format for the data and have an out-of-band index listing all
the tiles in any of the files.

So users could upload tar, cpio, ar or whatever uncompressed archive
they have as long as the indexer scanning the upload is capable of
parsing. The data of the client would not be modified but simply indexed
and moved to the storage.

> can be calculated with reading about 8bytes from the beginning of the 
> file (hopefully cached due to common access). So overall I would guess

Its not 8 byte its more - But in the end it comes down whether you cache
the leading 1-4Kbyte of every tileset file, or a central index file.
My first guess is that in the end the approach of a single file will
again win the race as it does not need that much kernel interaction and
has benefits in on disk storage speed.

My thoughts about this would be an index daemon communicating with the
apache module/cgi script via a unix domain socket. Getting passed the
zoomlevel, x and y and getting filename, offset and length in return.
(Read: its a mod_alias thing)

A nice thing which i saw in the facebook presentation was that they hand
out URLs which contain the filename, offset and length so when the http
client request a tile it does not do this via coordinates but the
storage location. This has some security problems which need to be dealt
with. 

I even thought about translating the request for the apache to a
http/1.1 query with a range: option. In the end HTTP supports reading
only parts of a file.

So a mod_alias which not only translates the URI but also adds a Range:
header ;) The problem though might be that the HTTP return codes for
Range requests would not be expected by the webbrowser.

But probably one could hide this somehow ...

> I cannot guarantee anything yet, but I hope to at least attempt to 
> implement this in case no one else gives it a try. It doesn't seem to 
> hard. But given I have never written an Apache module, maybe I am simply 
> to naive ;-)

I had a look at some apache modules the last days and it does not look
too bad. You need ~5 Days to get a build environment and 4 hours to
write the code ;)

> >I thought about it and my idea was to let t at h clients upload tilesets as
> >a tar.gz. The tileserver then would simply gunzip the tar and let the
> >tar exist as the storage. So the indexer would read tars and the apache
> >module would serve from the tars.
> 
> Is it necessary to zip any of this? The PNGs should be close to 
> optimally compressed already anyway and the extra header at the 
> beginning is only 5.3kb for a z12-z17 tileset, so even if that could be 
> compressed, overall the gain would probably be less than a percent.

I kept the current upload scheme where a compressed archive gets
uploaded. So given that we dont change everything at once the first
thing would be to convert a zip to a tar (non .gz). The next step would
be to let the clients upload ".tar.gz" (if the gzip part is worth the
compression ratio) and just uncompress the tar (not unarchive)

Flo
-- 
Florian Lohoff                  flo at rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.openstreetmap.org/pipermail/tilesathome/attachments/20080626/6595de01/attachment.pgp>