[OSM-talk] osmarender tile server broken?

Dirk-Lüder Kreie osm-list at deelkar.net
Sat Feb 24 20:07:12 GMT 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nick Hill wrote:

> I see the biggest problem with the current T at H set-up is the database records 
> are variable length, and the length of each record does vary regularly. That 
> creates a computational problem of how to store and retrieve such data 
> efficiently and adds workload when a larger record replaces a smaller record.

With the tiles that is the norm, because the maps always get more
complex, almost never less.

> We also need to consider how different file systems behave with large numbers of 
> small files and file size changes/ possibility of fragmentation.

I agree.

> As an example of different file system approaches; some file systems use linked 
> lists for directory look-ups. These are efficient for smaller number of 
> directory entries, inefficient for large numbers of entries. Other file systems 
> index directories using b-trees or b+ trees providing good performance even with 
> very large numbers of files per directory. Some file systems store small files 
> completely in metadata rather than assigning wasteful clusters.

Most of the files are still in the 0-2kb range I'd suspect, the largest
ones in the 100 kB ballpark.

> My tests on file systems show JFS or reiserFS as a much better file system than 
> either ext2 or ext3 when larger numbers of small files are being stored and 
> retrieved.

I agree on that, however I personally haven't tested JFS. ext2 is an
option over ext3 because of the lack of journalling. (why we can skip
journalling, see below).
however ReiserFS performs better than ext2 and has journalling anyway,
so that would not even be an issue.

> In general, both database and file system approaches have benefits and 
> drawbacks. There is a balancing act.
> 
> Currently, T at H tiles database has 23m records. I haven't tried copying/backing 
> up 23m files. I wonder whether most tools will cope.

Backup? when we can regenerate large portions of the data in a
reasonable amount of time? Same goes for journalling. if something goes
wrong with some tiles, you'd simply re-render them instead of trying to
fix the files.

Should a backup be of interest you can partition it up should problems
arise with the amount of files.
you can use lowzoom tilenames (z<<12) for that.

Dirk-Lüder "Deelkar" Kreie


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFF4JrwFUbODdpRVDwRAsGfAKCOZhII9eTfJBVtnvlwNRK92CwuOwCeKhpz
OyND2ZU3MY6HtL/kuZpQh0g=
=Xksq
-----END PGP SIGNATURE-----




More information about the talk mailing list