[OSM-dev] Splitting the planet into thousands of pieces in one pass.

Nic Roets nroets at gmail.com
Wed Dec 1 13:27:57 GMT 2010


Hello Scott,

How do you keep track of what bboxs each entity belongs to ?

I'm not really asking a question, I'm just saying that I found a way to
reduce the memory requirement for that considerably. Instead of a bit per
bbox per entry, I store only 16 bits or 32 bits per entry. Here is the
source.
http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp?rev=24484

Ask if you have any questions.

On Wed, Dec 1, 2010 at 2:59 PM, Scott Crosby <scrosby at cs.rice.edu> wrote:

> I've been working on making various improvements to the mkgmap
> splitter. It was originally intended to split a map into smaller
> pieces which could then be converted into Garmin format and artifacts
> of that original purpose continue to exist. (Eg, its coordinate system
> for representing the areas.)
>
> However, I've committed several improvements to crosby_integration
> branch in mkgmap SVN that massively increase its scalability and may
> make it more useful to a general audience:
>
> On a 8gb core i7 machine, the new version can:
>   Split the whole planet into 1200 tiles in about an hour (from pbf
> to osm.gz format) in a single pass.
>   Split the whole planet into 20000 tiles in about 4 hours over 3 passes.
>
> There are some limitations, both in the implementation, and design.
>
>    Due to inherent serialization of the internal tracking tables, it
> only uses a few CPU cores.
>
>    The code uses bounding boxes, not bounding polygons. In the
> original purpose of generating mkgmap tiles, tiles are square there
> are no polygon tests. Polygon support is straightforward to add, but a
> fileformat must be defined to specify bounding polygons instead of
> bounding boxes.
>
>    At thousands of tiles being output concurrently in a pass, memory
> consumption in output buffering can be significant.
>
>    The process assumes that the file is sorted with nodes first, then
> ways, then relations. Efficiency degrades if they are not further
> sorted based on ID.
>
>    Ways are not 'completed', A way is included if any node of it is
> within the bounding box. Other nodes in the way are not included
> unless they too lie inside the bounding box. (So, typically, the
> bounding box is expanded to include an overlap region.)
>
>    All metadata is dropped.
>
> The code hasn't been generalized to overcome these limitations, but
> might be useful to those creating lots of excerpts or breaking maps up
> by continent, nation, state, or county. As I don't expect to have much
> time to work on the code, I mention it now.
>
> Scott
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20101201/b51ad456/attachment.html>


More information about the dev mailing list