[OSM-dev] Freemap Rendering

[Christopher Schmidt]

It is my intention that over this weekend, the freemap rendering of the
map will be up to snuff to the extent that if it were desired to drop
into place in the place of the current tile renderer, it could be done.

However, this brings up a couple of technical questions on how to
proceed:

  * GD issue: I believe that rendering anti-aliased lines currently
    requires running a patched version of GD to prevent crashes when
    drawing single-pixel anti-aliased lines.
  * Mapserver version: Currently, the london freemap is running against
    a 4.9-CVS powered mapserver which has some improvements to stability
    and rendering. 
  * The Freemap server is running with some non-standard proj.4 codes to
    do reprojection into Mercator.

All of these are possible to replicate in another server setup, but it
may not be worthwhile to do so. Given that the data is being rendered
against a static dataset, and there is no tie in the rendering code to
the OSM database, this might be a candidate for a task which can be
delegated to the freemap server, rather than taxing OSM's resources.

Process:
  * Assuming a nightly dump can be made, it is done. The data is then
    bzipped and shipped across the wire to the freemap server.
  * Freemap server converts planet.osm dump to GML, then uses ogr2ogr to
    convert it to shapefiles.
  * Shapefiles are moved into data directory, moving old data aside and
    archiving it.
  * Cache files for vector data are removed from file-cache directory.
    (Later, this could presumably be changed to be more intelligent on
    detecting changes.)

One of the more resource-intensive parts of this process is the
planet.osm->GML step: it takes about 5 minutes and 600MB of RAM as
currently implemented. (There may be improvements to this to be made,
but I think they're minor if so.)

All of this seems to me like it could be automated. I know Schuyler has
offered the use of the freemap server in the past for rendering or any
other work that needs to be done, and the machine is an AMD64Bit machine
with 2GB of physical RAM, which is probably quite a bit faster than the
existing hardware on this particular task. (Mapserver benefits greatly
from the 64bit aspect of the processor in testing.)

Naturally, all code would be checked into subversion so that if any
aspect of the setup were to fail -- Schuyler and Jo and I all fall off 
the planet, for example -- it could be easily replicated inside of OSM. 

Although I can't speak for Schuyler (and the freemap server is much more
his hardware than mine) this might allow OSM to change to a new
rendering system with the minimum of fuss. The caches may grow quite
large, but the disk on the freemap server has 200GB free, so I think
that's safe for a little while -- and probably if they were to get that
big, OSM would be suffering similar problems. 

It feels to me like this would help alleviate as much load as possible
off the OSM servers with as little effort as possible. I'm assuming here
that some sort of access would be given to Steve and someone else
(NickH?) to ensure that the servers could be maintained by them in a
case as above where Schuyler, Jo and I are unavailable -- if this proves
impossible, I would understand that this scheme might not work too well.

This is just some thoughts I had when discussing how this stuff might
proceed with Jo. Again, Schuyler can offer a firmer offer, but this
seems like a sound technical, if not sound social, solution to change
platforms without worrying about harming OSM's performance.

-- 
Christopher Schmidt
Web Developer