[OSM-dev] Migrating osm.org to vectors/Kartotherian

[Paul Norman]

On 10/31/2015 6:32 PM, Yuri Astrakhan wrote:
> #1 OSM download -> SQL tables
> We have used osm2pgsql to produce default table structure. Yet, it is 
> not the most efficient way to parse data afterwards, and Paul has been 
> working on the new ClearTables structure [3].
> TBD: We would need to agree on a prefered table structure for this 
> project.

Database schema is not going to be the same for every project. I'll 
probably be announcing something soon, but I expect my work to live 
alongside osm2pgsql C transforms, osm2pgsql pgsql Lua transforms, and 
imposm3, as well as vector tile generation platforms that don't involve 
a database.

> #2 SQL -> Vectors
> This is very similar to the set of SQL statements that osm-carto uses 
> to generate data. Mapbox is now on version 6 of vtile structure, so it 
> is obviously a hard problem to nail down. [4] We could make it 
> compatible, or come up with a different structure.
> TBD: vtile structure

A style designed for showing off OSM data and mapper feedback will put 
different data into vector tile different ways than one designed for 
Wikimedia's purposes. If osm-carto was vector tile based, we'd probably 
be changing the vector tile definitions every 6-12 weeks, and these 
would be backwards-incompatible changes.

> #3 Vectors -> (a) PNG server side and (b) WebGL client side
> If #2 produces Mapbox-compatible vtiles, we can easily reuse all the 
> open source styles Mapbox produced, both MSS and WebGL, or create new 
> ones. The problem here is that MSS & WebGL are different languages, 
> and keeping them in sync with Mapbox Studios (Classic & GL) is hard. 
> Richard Fairhurst is working on Glug[5] to simplify this.  Eventually 
> it could be just the WebGL style and Kartotherian could render WebGL 
> on the server side if needed.

Speaking as someone who has worked with two styles which were supposed 
to look the same, but had a different structure internally, writing the 
same style twice should be avoided at all costs. It's not twice the 
work, it's often more. Some kind of common language that gets processed 
to both Mapnik XML and GL JSON is needed. I don't know what this 
language will end up being, but it's too easy to write styles which have 
too many cascaded rules in CartoCSS so it probably won't be that. On the 
other hand, CartoCSS has multiple implementations which talk to multiple 
rendering engines, so it might win out for that reason.

I'm not sure Richard has any intent of targeting Mapnik XML with Glug.

If it weren't for minutely updates and the need to handle planet-sized 
files, I'd probably go with tilemaker[1] for vector tile generation. 
This would completely replace SQL tables. Keep in mind, I've been a 
PostGIS consultant, am a maintainer of osm2pgsql, and know writing SQL 
for layer definitions like few others do, and I'm prepared to throw that 
away for the advantages of how tilemaker transforms its data. 
Unfortunately, for what I'm interested in, I need minutely updates and 
planet dump handling.

I have a blog post on style complexity coming out when I gather some 
more numbers.