[Talk-us] While we're fixing things in iterations

Thu Sep 24 14:29:18 UTC 2020

On Thu, Sep 24, 2020 at 6:04 AM Minh Nguyen
<minh at nguyen.cincinnati.oh.us> wrote:
> More recently, Kevin Kenny reimplemented the shield renderer using a
> more robust approach [3] and has discussed route relation support with
> the openstreetmap-carto developers. [4]
>
> OSMUS is interested in offering an Americentric renderer replete with
> shields. If anyone would like to help with the server situation, let's
> get in touch. Also I'm sure Kevin would welcome any pull requests to his
> osm-shields project, which would probably be a good starting point for
> this renderer if we go with Mapnik and raster tiles.

Everything Minh said here is true. I'd welcome all the help I can get!

The project has been on hold for quite some time, because I got
tremendously frustrated with it.  There are a lot of moving parts that
all need to come together to make it work. It touches a many
components in the rendering chain.

OSM-Carto is not the main problem.  Its developers are quite
responsive indeed to the idea. (I know that it would eventually bog
down for a while in controversy, but if we're talking about a
US-centric openstreetmap.us server, we could fork the style.) It's not
been my #1 priority, simply because I use the shield rendering for
some of my own maps, and I don't use the OSM style for them.

For OSM-Carto to do the shield rendering, Carto itself would have to
handle it.  That requires support in Carto for the GroupSymbolzer in
Mapnik. [5] I don't imagine this would be all that hard a problem, and
again the Carto developers are interested and would most likely accept
a well-done PR.

Mapnik is a non-issue. For my maps, I was able to use Mapnik itself
'out of the box', except for the fact that I render in two layers, one
for fill colour and one to overlay linear features, icons and labels.
(I work it this way because I interrupt linear features with
transparent cutouts for labels, and I want the fill colour to show in
the transparent areas.) This is easy enough in Python; I have no idea
what the impact would be on Carto because at present I don't use it.

The shield placement in Mapnik depends on quite a complex piece of
SQL. [6] The reason for the complexity is that readable shield
placement is extremely difficult when considering a route as a bucket
of discrete ways. It gets much easier if ways can be grouped into the
longest possible linear sequences that share a set of markers (this
happens at [7]). The SQL in turn depends on a couple of extra tables
in the rendering database: 'planet_shieldroute', which enumerates the
route relations that require shields; and 'planet_shieldway', which
enumerates the ways that are members of the routes and gives their
roles.

Because of the complexity of composing the shielded routes at the time
of rendering, Paul Norman and Sarah Hoffmann are convinced that the
approach I'm using would be far too slow if it were tried on an OSM
tile server, and demand a different one. Neither of them has, so far,
sketched an alternative design, and I have so far not been able to
come up with one myself. Paul, in one message, hinted that developing
a renderer for concurrent routes that would satisfy all his
constraints may not, in fact, be possible. Unfortunately for me, this
discussion spilled over into the changes that would be required for
osm2pgsql; I sketched what osm2pgsql would need for the two auxiliary
tables to be created, and drafted a formal proposal [8] for the idea.
(Note that in that proposal I proposed no changes whatsoever to OSM's
main renderer.) The developers of osm2pgsql unambiguously indicated
that a PR implementing the change would not be considered. As a
result, I decided that the project would have to switch to another
database importer, likely imposm3. [9] Maintaining a fork of osm2pgsql
indefinitely was not an acceptable approach to me! Making the switch
would require me to retool quite a lot of unrelated work, so I
reluctantly decided to put the project on hold.

Since then, Joseph Eisenberg and Jochen Topf have stepped in on the
osm2pgsql project and committed the 'Flex' backend. [10] I have not
had the time to try to use the Flex backend to develop an importer for
the tables that I need, but quickly scanning the documentation
suggests that it has everything that the project would require - and
was found acceptable to the osm2pgsql developers. I think, therefore,
that particular roadblock may have been removed.

So the limiting factor, then, is my time - which is in
catastrophically short supply at the moment. I'm facing clamouring
demands at work because people seem finally to be getting the idea
that I'm retiring - with only five weeks left to go.  Once I've
finally left the workplace and had a little bit of time to rest, I
might take another run at the hill.

In the meantime, the Github for the project has some further notes on
what needs to be done to make the shaped rendering and route relation
handling scalable. [11]

On Wed, Sep 23, 2020 at 6:57 PM Andy Townsend <ajt1047 at gmail.com> wrote:
> 1. You'd need to derive the shields from the ref and the road itself from the way, and you're going to get some edge cases where they "don't seem to match".
> 2. I expect that it would be _really_ difficult to render refs from relations in the one country where that's needed and refs from ways in the other 190-odd.  The OSM style is a global style, and that means that local edge cases (which is what the US is here) can't get the "special-case handling" that might be nice.
> 3. The infrequency with which OSM data is loaded now means that more has to be done "on the fly" rather than "at data load", which somewhat limits the options for how to solve the problem.

Could you give me some more detail about what sort of mismatch you
expect to see?  If the rendering database maintains referential
integrity, it's hard for me to understand what you think is going to
be the issue.

I'm pretty convinced that I have item (2.) under control.  Look, if
you will, at https://kbk.is-a-geek.net/catskills/test4.html?la=42.9170&lo=-72.2736&z=12,
where the area around Keene, and a few trunk roads, have route
relations and get shaped shields, while the area to the south and
east, and the smaller state highways, still have the routes managed
with ref=* tags on the ways and get OSM-like rendering. (If you like,
also zoom into https://kbk.is-a-geek.net/catskills/test4.html?la=42.9368&lo=-72.2969&z=15
and you can see the handling of concurrencies.) The styles can be
intermixed pretty freely.

With item (3.), I did sketch a design for keeping the tables up to
date with minutely updates. You're right that I haven't implemented it
yet; as I said above, I wasn't willing to develop into a fork of
osm2pgsql, and the 'flex' backend is a relatively new development.
Since I haven't yet started doing on-the-fly rendering for any of my
maps (I really don't have the resources to maintain the whole planet
in PostgreSQL and process minutely updates, so I depend on geofabrik
extracts), I didn't take this issue on yet. It's on the list.

Summary:  Help Wanted!

> [1] http://elrond.aperiodic.net/shields/
> [2] https://bugs.launchpad.net/osm-shields/
> [3] https://github.com/kennykb/osm-shields/
> [4] https://github.com/gravitystorm/openstreetmap-carto/issues/596
[5] https://github.com/mapnik/mapnik/wiki/GroupSymbolizer
[6] https://github.com/kennykb/osm-shields/blob/master/queryprocs.sql.in
[7] https://github.com/kennykb/osm-shields/blob/aa3df920d19cc597ebb8385e804fc1b275798922/queryprocs.sql.in#L347
[8] https://github.com/kennykb/osm-shields/wiki/Proposal:-add-route-tables-to-osm2pgsql
[9] https://github.com/omniscale/imposm3
[10] https://github.com/openstreetmap/osm2pgsql/tree/master/flex-config
[11] https://github.com/kennykb/osm-shields/projects/1
-- 
73 de ke9tv/2, Kevin