[OSM-talk] osmarender tile server broken?

[matthew-osm at newtoncomputing.co.uk]

Hi,

On Sat, Feb 24, 2007 at 07:33:42PM +0100, Frederik Ramm wrote:
> Having the tiles in the database is bad, because access 
> (apache->php->mysql->filesystem) is expensive in terms of CPU load. 
> Having the tiles in the filesystem (apache->filesystem) is ideal.

Yes.

> Filesystems are made for this kind of thing, and Apache is made for this 
> kind of thing, so if this is at all possible, do create a file system 
> with enough i-nodes. To Matthew: Let us *please* refrain from adding 
> another layer of indirection via funky C programs and file archives!

1. it is not a "funky" C program. It would be as simple as you can get -
   literally extract a series of bytes from a file.

2. as has been pointed out here and on IRC, an existing program such as ar or
   unzip should work just as well.

3. yes, it may be slightly slower than apache hitting files directly. It will
   also be lots faster than the current apache->php->mysql->filesystem method.

> A plain simple ext3 file system is very capable of providing what we 
<snip stuff about filesystems I already know>
> (/tiles/zoom/x0/x1/y0/y1.png where x0=x/1000 and x1=x%1000), 2^18 files 
> in one directory would work.

When I tried thousands of files in one directory on ext3 a couple of years ago,
directory listing was slower than running your average MySQL query... ;-).
ResiserFS was always lightening fast. That was thousands, not millions. Maybe
the've fixed it now, though. Really - thousands was unusable.

> There must be a way of making a file system that can handle our required 
> number of inodes. It is absurd to build work-arounds that will make the 
> whole thing slow and less manageable just because you cannot afford to 
> do a proper mkfs.ext3! If all else fails, move data from one hard disk 
> partition to another server, create new file system, move data back. I 

We're talking about OSM here, and like many projects, the quickest way of
getting something done is to do it yourself. tiles at home is desperately in need
of something to make it faster, and changing stuff like filesystems needs others
to plan and make it happen - Steve and Nick are busy people, like the rest of
us. Even doing loopback filesystems needs an admin to mount it for you.

If tweaking FS parameters to get things right is the ultimate way to go, great.
However, until things like that happen there are a lot of people out there who
just want to view their maps. Putting files in mini archives is one fairly easy
way to get things going again that does not push work onto other people, and
does not use up all the spare FS inodes (and more) that we currently have.

Thanks,

-- 
Matthew
(running off to do a bit of "tile spotting", although they're
becoming increasingly hard to find these days)