[OSM-dev] API/XAPI caching proxy server

Ian Dees ian.dees at gmail.com
Fri Dec 17 00:18:24 GMT 2010


On Thu, Dec 16, 2010 at 5:39 PM, Brett Henderson <brett at bretth.com> wrote:

>
> In my experience the biggest limitation in performance is disk seeking,
> rather than the amount of data returned.  The earlier pgsimple schema used
> to be well indexed, but due to the order in which data is created in OSM
> (ie. light scatterings of changes across the globe every day), the results
> will tend to be scattered across the entire disk which is disastrous for
> performance.  The newer pgsnapshot schema clusters data geographically which
> reduces the disk seeking considerably.  It works quite well for bounding box
> style queries, but I don't know how well hstore GIST indexes will work for
> tag queries.
>

In my limited test with the Minnesota extract from Cloudmade, running
queries on the tags was very quick after indexing it. For example it took
roughly 100ms to tell me that there are 46 ways in the state tagged
leisure=playground. Only problem so far: the database size on disk was
roughly 7GB (the .bz2 I got from Cloudmade was ~300MB). We'll see if the
13GB planet can fit on the 414GB disk that's available to me...


>
> A tile based storage approach resolves much of the disk seeking issue by
> design, but you'll have to do much of the heavy lifting yourself to come up
> with your own storage mechanism.  Note also that any custom storage needs
> the ability to be kept up to date with minute diffs which makes it more
> challenging.  This may well be the best approach, I really don't know.
>

I toyed with this a little bit and got fairly far. Generating z12 data tiles
for the entire world would take roughly 1-2 days on an EC2 m1.xlarge based
on it taking ~12 hours to do the continental US, if my memory serves me
correctly. I stopped working on this when I couldn't come up with a nice way
to figure out which tiles to re-create based on Osmosis-applied minutely
changesets.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20101216/e260919c/attachment.html>


More information about the dev mailing list