[Openstreetmap-dev] Source overview

Mikel Maron mikel_maron at yahoo.com
Mon Dec 5 18:46:23 GMT 2005

There are potential efficiencies along the whole chain of this process.
It's tricky to profile everything on the live machine, so the dev
machine should help determine where the best potential is.

Starting at the bottom of this stack...

+ ruby may be overkill for generating tiles. just running gpx.rbx or
streets.rbx on its own takes overly long -- even "+" is implemented as
an OO method in ruby. I implemented the same code in perl/GD and there
was a significant speedup, but still not great - there's the initial
compile/startup cost. Perhaps we should be looking at something with as
little overhead as possible - like C.

+ I tried ImageMagick over Cairo. Even with the kludge of saving out to
a file, Cairo is slightly faster than ImageMagick. GD I suspect is
better than both.

+ The GPS and Streets layers are requested by mapserver through a WMS
request -- an additional HTTP request which must go through squid, then
apache. So, these layers are being unnecessarily cached in squid (they
are not usually requested on their own, usually combined with landsat).
We want to at least bypass squid. Better would be to request these
tiles directly from the local scripts, without going through http.

+ There are ways to get Mapserver talking directly to mysql, with
Mapserver rendering the geometries directly. This could be very very
good, but will require some MapScript coding to test.

+ Mapserver requests tiles Landsat from JPL. If some of these could be
cached seperately, that would save a lot of time. We need a lot more
disk space for the cache. To host the entirety of Landsat 7 would
require 4 Tb.

+ Mapserver itself may be bloated. In the current setup it is only
composting 2 or 3 images together .. so that's overkill and could
easily be done with something more lightweight. However, there is a lot
of good stuff there -- especially in styling tiles, applying labels to
streets, POI. Mapserver is run as a CGI -- if it could be run
persistantly that would speed things up.

+ Squid only has 2GB of cache. 

+ The cache is presently invalidated when the tile is 48 hours old.
This can cause a delay in edits appearing on the tile server, and many
tiles are being invalidated when no change has taken place. A smarter
cache invalidation script could make marked improvements here.


--- SteveC <steve at asklater.com> wrote:
> the http request hits squid (caching) and if in the cache stops there
> and an image is served. if not it's passed on to mapserver. mapserver
> makes a call to JPL to get the landsat images and to a ruby script
> (gpx.rb) to get a transparent layer of gps dots. mapserver composites
> these and sends them back to apache, which sends it to squid, which
> sends it to the applet.
> gpx.rb wakes up when called and grabs n gps points from the database
> and
> renders them using the ruby Cairo bindings. unfortunately it needs to
> save the image and read it back in as you can't get the binary out of
> Cairo. Hence a move is in progress to ruby-magick bindings.
> Mikel and Nick also want to work on this stuff to make the tile
> server
> not drop layers and also render prettier streets. You guys want to
> brain
> dump your thoughts?

More information about the dev mailing list