[osmosis-dev] Cutting PBF file into 1° tiles

[Sylvain Melin]

Hi Peter and thank you for your help.

I had the same kind of approach and I guess the problem was your 4th point.
I'll give a try at osmconvert with "complex-ways" option as a second step.
But as you mentioned, there will be a problem with features that are 
bigger than the margin.
Anyway, this will be a good temporary solution and I will have to move 
Jochen's solution as soon as I have a better system with enough RAM.

Thank you again for this detailed description and your sources, I'll 
definitely give it a look.

Regards,

Sylvain

On 18/04/2016 13:42, Peter Svensson wrote:
>
> Hi
>
> I've spent a lot of time trying to solve this problem for OSMTileMachine.
> OSMTimeMachine is basically a tool that bridges the gap between a full 
> planet file and a data consumer loads all the data into RAM.
>
> This makes it possible to render a map of a very large region using 
> Maperitive (which itself cannot handle very large amounts of data, due 
> to the loads-everything-into-memory-limitation). See example of 
> results here: www.mtbmap.se <http://www.mtbmap.se> . This map is 
> rendered using OSMTileMachine. I don't know about your requirements, 
> but as a working proof of concept, the input do that map is a single 
> planet.pbf and all of this is rendered on a Windows PC with 8 GB of RAM.
>
> So what I have figured out so far is this:
>
> 1. osmconvert is MUCH faster than osmosis for extracting data, at 
> least for me.
>
> 2. For generation of map tiles, complete ways/relations are needed. 
> This means that at the boundary, everything related to each 
> way/relation passing through the boundary of the tile image needs to 
> be complete.
>
> 3. osmconvert with the "complex-ways" option is very suited to 
> generate complete data for visualisation of a geograpically limited 
> region (square). I use 0.04 degrees of margin when i extract data for 
> a tile for zoom level 10.
>
> 4. it's not possible to use the complex-ways cutmethod at the full 
> planet due to time and/or RAM/DISK  usage.
>
> 5. If i extract the data in two steps, it is possible to prepare 
> complete data for a small box:
>
> 6a) First extract a region AROUND the box of interest with cutmethod 
> "clip". I think this is default for osmconvert. I use 2 degrees of 
> extra margin around my box of interest here. This makes it possible to 
> produce anything within reasonable abount of time and RAM / disk usage.
>
> 6b) Since the output of 6a) is too big for my data consumer (in my 
> case maperative) i need to shrink it again and cut away even more 
> data. This time i clip at the exact box of interest + 0.04 degrees. 
> (This is what i need to have enough overlap for place-labels etc that 
> can stick into neighboring map tiles). If i use the complex-ways 
> method i will not loose any data inside my map file and the results 
> looks good.
>
> The drawback is that features that are larger than 2 degrees will be 
> incomplete. This is mainly a problem for very large lakes and of 
> course the coastline.
>
> Some source code, if you have an interest:
>
>
> https://github.com/psvenz/OSMTileMachine/blob/master/osmTileMachine/src/osmTileMachine/SplitAndRenderStrategy.java
>
> Hope this helps.
>
> with regards,
> zvenzzon
>
>
> On Mon, Apr 18, 2016 at 10:10 AM, Sylvain Melin <s.melin at alsim.com 
> <mailto:s.melin at alsim.com>> wrote:
>
>     Hi everyone.
>
>     I'm working on a 3D earth rendering engine and I want to use osm
>     data for forest, cities, roads, etc...
>     Of course it involves a lot preprocessing to filter and convert
>     vector data into OpenGL friendly format.
>
>     My plan is to :
>     - exploit a planet sized pbf file
>     - cut it into 1° tiles using osmosis
>     - filter and extract the data from these tiles as shapefiles using
>     libosmium
>     - rasterize these shapefiles with gdal
>
>     My problem occurs when trying to cut the big PBF file into tiles.
>     I can't find an efficient way to prevent the elements from being
>     clipped on the edges of the tiles.
>
>     I've tried to use completeWays=yes and completeRelations=yes but
>     the first tile was still empty after a few hours when it took 6
>     minutes to process it without these options.
>
>     The only way I can see to limit the damage is to set a wider
>     bounding box than necessary but there will always be a polygon
>     bigger than my margin that will be clipped, and the size of my
>     tiles is also greatly impacted.
>
>     Do you have any suggestion about this ?
>
>     Thanks.
>
>
>
>     _______________________________________________
>     osmosis-dev mailing list
>     osmosis-dev at openstreetmap.org <mailto:osmosis-dev at openstreetmap.org>
>     https://lists.openstreetmap.org/listinfo/osmosis-dev
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20160418/be62c72f/attachment.html>