[OSM-dev] Proposal: Database accelerator and dirty tile marker based on simple algorithm.

[Nick Hill]

Hello Anselm

OSM has a lot of potential growth, which seems to usually be limited 
software and hardware technology.

A server upgrade myself and Steve completed a few months ago gave us 
good performance for a short time. The system then became extremely slow 
as demand caught up with supply. I recently isolated an SQL query which 
I posted an improvement for, which was soon improved again. This has led 
to the current state of affairs where the database and API systems are 
easily able to cope with current demands. I have made further database 
optimisations which provide further growth potential.

It is only a matter of time before demand and OSM growth rate bring the 
system back to it's knees.

Looking for method changes which can improve performance possibly by 
another order of magnitude is really what we need. We should do this 
before attempting to deploy massively parallel systems (something our 
budget will not currently stretch to in any case).

OSM is different from any other GIS project. Funding is almost zero, 
potential userbase is massive, and the data set is growing at a very 
fast rate. More than any other GIS project, OSM needs to provide 
cost-effective delivery of geodata.

If there is usable existing GIS code to make OSM more efficient, then we 
need to evaluate it, just like we have been evaluating ideas on this 
list. If you know of such code, please add to the discussion. It will 
require analysis of the technology. If it is promising, we'll need side 
by side comparisons in situations which simulate a working environment, 
considering all aspects of computational load including I/O memory and 
CPU requirements.

Myself and Steve have already evaluated R-tree indexes and independently 
found shortcomings. These tests have to a large extent molded the shape 
of OSM's current storage and delivery strategy.

Myself and Nigel have been arriving at solutions which have not yet been 
tested for use by OSM. They are very promising and provide the potential 
for delivery improvements.

There are also other approaches, such as partitioned databases. A 
partitioned database could itself provide the extra order of magnitude 
improvement we need. That does not stop consideration of yet more models 
which could be better still, and could be integrated with database 
partitioning.

OSM is more than a housekeeping operation of existing GIS technology. 
Anyone who thinks the necessary technology is already out there to make 
OSM more efficient ought to prove it.




> If you look at the discussion: Nick proposing a dirty tile marking
> scheme, Nigel Magnay noting that tiles are only marked dirty at the
> end-points of a line-segment (rather than more correctly dirtying a
> bresenhams or a boundary of the line-segment) and proposing a quadtree.
> Then the discussion moving onto using decimal integer coefficients (I
> personally used hexadecimal coefficients for performance in
> http://thingster.org/client.html which is an adobe svg only demo with
> precomputed tiles)...  Overall you see a lot of possible branching
> decisions that take significant mental computation to evaluate.
>
> I do agree with Lars, although not the tone used, that PostGIS, GRASS and
> the like should be leveraged.  The OpenLayers and the OSGEO community I
> think would be more than happy to help standardize the back end data
> representation and storage of line segments in such a way that all of the
> concerns could be satisfied; such as making sure that tiles are never
> obsolete.  The 'Mapping Hacks' book is worth spending a few days browsing
> over; and it is worth even just trying mapserver, playing with postgresql,
> ka-maps and the like such as OpenLayers is pushing.  There is even still
> possibly a role for squid where it could be used as a tile invalidation
> scheme.  For me, putting together http://maps.civicactions.net was pretty
> trivial aside from the javascript mostly because I just used off the shelf
> solutions.  I feel like there is so much value in what OSM is doing that
> if there were some way that the strengths of GIS approaches could be used
> without having to discard them due to gating considerations such as
> concern over tile invalidation, that it would be best.
> 
> So no technical advice; just basically a 10,000 foot view comment...
> 
>  - a
> 
> 
> 
> 
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>