[OSM-dev] Tiled Data Extracts

[jharris84 at devisland.net]

There seems to be have been a bit of interest in tiled data recently,  
such as Michael Daines GSoC project. I've been looking at this for a  
while as well, but perhaps in a different (and hopefully  
complementary) manner.

My idea is to:

1. For each osm element, find the highest zoom level quadtree tile  
which fully contains the element
2. Using a quadtree data structure, find groups of tiles, each with  
approximately 8000 elements
3. Write these tiles as a pbf file, with an index file showing the  
tile quadtree value and the location in the file

The two main differences between my approach and existing schemes are that:

1. Each osm element appears in exactly one tile
2. The tiles form a hierarchy, usually between 10 and 16 levels deep;  
a query on a point would return 10-16 tiles. Sparse areas have fewer  
active levels.

It is then straightforward to produce an extract file by simply  
copying the tiles which overlap the area you wish: this takes seconds.  
These can then be trimmed to closely match the specified area, and  
sorted into the usual order, either using osmosis or the tools  
provided in the package below.

My attempt at this is available at

http://github.com/jharris2268/osmquadtrees

To install, and run, see

http://github.com/jharris2268/osmquadtrees/wiki/Getting-started

It is written in python, with the computationally intensive parts in a  
C++ extension module. I have only tested it on Linux, but I don't  
think there would be to many problems in getting it to work in  
windows, or on a mac. To work on a full planet file you need a 64-bit  
version of python, and at least 8gb of ram. The github wiki pages  
contain some further instructions and details of the extensions I have  
made to the pbf format.

Also available is a demo of how extracts could be created on the fly:  
as mentioned above, all it entails is copying parts of one file to the  
HTTP stream. This covers a small area around London.

http://www.jamesharris.me.uk/osmquadtrees/

Please let me know if any of this is of interest or use,

James