[OSM-dev] Cartagen - client-side vector based map renderer, dynamic maps

Jeffrey Warren warren at mit.edu
Fri May 8 19:04:48 BST 2009

> * The proxy receives XML from the api or xapi server. Currently it
> requests the full dataset.
> * Then it removes unnec. tags (like note, fixme, attribution and a whole
> bunch of others that are not needed for rendering). Some of them are
> very minor, but 10000 nodes with "attribution=veryvery long string
> here" can make up like 40% of all the data, and just clog the line and
> browser :)

Yes, I'm thinking of trying to cache locally but still request changesets if
the ?live=true tag is set... caching locally is great for more static data
but for the live viewer, I'm trying to not use caching, but increase
efficiency in the requests.

* The data is then pruned into (currently 3) levels and stored in a

> cache:
>  * level 0 - full
>  * level 1 - no POI, no paths, streams, tracks etc. used for zoom 11
>  * level 2 - no tertiary roads etc. used for zoom 10 and below
> * The client is served the level it currently requested as JSON.gz.

Great, this is what I'm working on too. I'm thinking a ruleset about what
features are relevant for what zoom levels could be something to work
together on? I was also thinking of correlating tags with a certain zoom
level. But maybe each tag should be associated with a range of zoom levels,
like "way: { zoom_outer: 3, zoom_inner: 1 }". Thoughts?

> * There are three servers in the list (api.openstreetmap,
> xapi.informationfreeway and tagwatch) and a lot of them do not complete
> the request (internal error, not implemented etc. etc.). It can take a
> lot of retries to finally get the data.
> * Even when you get the data, it takes seconds (10..40 seconds
> is "normal") to minutes - upwards to 360 seconds just to serve one
> request.
> So currently all received data is stored in the cache for 7 days to
> avoid the very very long loading times.
> Ideas of fetching the full dataset and pre-computing the cache simple
> don't work because I don't have a big enough machine and no big enough
> online account to store the resulting JSON :(

> Also, somehow processing 150 Gbyte XML into JSON will prove to be a
> challange :)

So I'm having the same problems with the APIs. The standard 0.6 api has been
pretty good but of course it serves XML, not JSON. The xapi is not very
responsive to me, it seems. I thought parsing XML in JS would be molasses,
so if you're interested, we should put up our own XAPI or custom api off the
planet.osm file, and send JSON?

I have an quad-core Intel Mac Pro with 1.5 TB and a bunch of RAM we can
dedicate to this effort, with plenty of bandwidth. And perhaps when Stefan's
work is published, we could run it as well, since it seems to be a great
solution to requesting fewer nodes for large ways... but for now do you
think you could use an XAPI? i think all my requests fit into that api.

Alternatively, Stefan points out that the dbslayer patch for the Cherokee
server allows direct JSON requests to a database. So some very thin db
wrapper might serve us for now? This isn't my area of expertise, so if you
have better ideas on how to generate JSON direct from the db, like GeoServer
or something, and still have tag-based requests, i'm all ears.

Yes, but reducing the polygons is also a lot of work :) I haven't

> started on this yet, because on zoom 12 or higher you need to render
> almost anything, anyways. Plus, you would then need to cache thepartial
> data somehow (computing it is expensive in JS..)

Seems like Stefan's work may address this, no? Or if we did cache it, seems
like we'd calculate it on the server side.

> > d) oh, and localStorage. I've partially implemented that but haven't
> > had much testing... other work... ugh. So caching on a few levels,
> > basically.
> I fail to see what localstorage actually gains, as the delivered JSON is
> put into the browser cache, anyway and the rest is cached in memory.
> Could you maybe explain what your idea was?

Yes, localStorage persists across sessions so you could build up a permanent
local cache and have more control (in JS) over requesting it and
timestamping when you cached it, not to mention applying only changesets and
not complete cache flushes. This has some advantages over the browser cache,
although that does of course persist across sessions too.

> * There is a talk I proposed for State of the Map and I don't want to
> spoil everything before :)

yes, me too! so if you want to discuss off-list that's fine.

Of course, semi-dynamic rules like "color them according to feature X by
> formula Y" are still useful and fun, and avoid the problems above.
> (Like: "use maxspeed as the color index ranging from red over green to
> yellow" :).

Yes, this is an exciting area to me, for example the color by authorship
stylesheet i posted before:


or this one i threw together yesterday, based on the tags of measured width
instead of on a width rule:


A more fully-rendered screenshot is here:


Anyways, thanks for sharing; one thought I had was that besides sharing
ideas and solutions online, we should try *different* approaches, so that we
try all the possibilities. I think multiple projects working on the same
problem can sometimes be redundant, but more often it's beneficial for all
parties since there's a diversity of approaches to a problem. Let's take
advantage of that by specifically attempting different solutions to the
problems we face, and discussing the results... if you're willing. If one of
us tries a technique and it doesn't work, we can all learn from the attempt.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20090508/097b93c8/attachment.html>

More information about the dev mailing list