[OSM-dev] Very long ways have been split (was: Status of Database Server after 0.4 Upgrade: Fragile)

[Jon Burgess]

Sorry this response is a few days late. There were too many emails on
this topic to keep track of them all.

On Mon, 2007-05-14 at 01:37 +0200, Frederik Ramm wrote:
> Hi,
> 
> (me:)
> > I also found the ideas detailed in your Plan B and C quite good. Feel 
> > free to submit patches for the rails server, JOSM, and tiles at home as 
> > soon as you have them implemented ;-)
> 
> Apologies for the tone. That doesn't get us anywhere.
> 
> Maybe I overreacted a bit to Steve's post. I said that the system is 
> still fragile. To someone working primarily from the planet file, this 
> is probably hardly noticeable - but I am very unhappy with the 
> performance of the API at present, and I wanted to help getting it 
> fixed, preferably within a timeframe of a few days, not a few weeks.

It is difficult to comment on your mail since it revisits some topics
several times, but here goes...

Yes a short term solution to the immediate problem is good. I'll refer
to this later on.


> I saw the items that Steve listed in hist post as all contributing to 
> the problem, and picked one that I thought was easy to fix. Frankly, at 
> that moment I thought that even if my actions would break the rendering 
> of those 250 (of 600,000) ways in the database, that would be a small 
> price to pay for more stability, and one could still work out the 
> details later.

I agree. A quick fix which results in them not rendering in some clients
seems acceptable in the short term.


> You are right in saying that some sort of "proper" dealing with complex 
> objects would be good (personally, I am still thinking "superways" which 
> would combine parts of a boundary into one area).

I don't especially like superways since introducing another level of
abstraction add complexity and ambiguity in my mind without buying us
much. I could expand on this, but I think it is off-topic for now.


> The solution - or let us say, the quick fix - that I implemented will 
> probably break the rendering of 19 lakes in the tiles at home layer, but I 
> thought that (a) I can fix this in a reasonable timeframe and (b) it 
> doesn't really matter all that much because most of those lakes were 
> actually never properly rendered pre-0.4 because it is only since then 
> that the ways are returned in its entirety. My fix also breaks those 19 
> lakes in the Mapnik layer unless work is done to osm2pgsql.c.
> 
> Your proposed solution, in turn, requires no changes to osm2pgsql.c, but 
> changes to the API and potentially all its clients.

> Frankly, I cannot see how your solutions - either Plan B or Plan C - 
> could be implemented short-term.

I disagree with most of the latter 2 statements. Plan B in its simplest
form appear to be a case of the following pseudo code:

get_ways()
{
	for each way
		if (count(segments) > limit) then continue  <-- New code
		fetch segments in way
		fetch nodes for segments
}

Trivial in my eyes. I'll code it up and commit it into the rails SVN if
you like.

Since the data would not be included in the OSM output renderers and
clients simply would not see the data. No changes would be needed.

I see this as fulfilling your criteria for quick implementation and not
needing any recoding of clients. It also makes things no worse than the
current situation where the broken tiles can not be rendered at all.

My suggestion to enhance this with an extra tag indicating the way has
been omitted was just a suggestion of a future usefulness of this
approach since many people would surely criticise the fact that the data
was being hidden from clients.

I'll ignore plan C since again this is a longer term suggestion.

> It wasn't ok for me to change things without thinking about Mapnik. But 
> neither is it ok for you to lay back and say "I work from the planet 
> file, I can handle long ways, I don't care if the API chokes on those, 
> you sort that out among yourselves somehow."

We are talking about different things. Actively breaking something can
not be compared against in-action to fix something which is already
broken.

I'm perfectly willing to discuss other solutions and provide fixes for
them. Why haven't I done it already you may ask. I'm trying to make sure
that I get buy-in from the OSM development team before implementing
something like plan B which risks upsetting some people. Please don't
read this as a sign of procrastination or trying to float blue-sky ideas
with no intention of implementing them.


If you read my email thread about adding key/values to assist in
re-joining ways I hope you should also get the idea that i'm not being
completely negative. I'm trying to *improve* the situation for everyone.


> May I suggest the following:
> 
> 1. Leave my split in place for all linear features and coastlines, 
> assuming that Mapnik has some way to deal with split coastlines. (If 
> that assumption is wrong, then I would have to hand-pick those coastline 
> ways that represented whole islands.)

The mapnik plan for those is to render them as a thin blue line , but
this is no different to before since they have never been closed
polygons.

> 2. Revert my split for the 19 "natural=water" features, assuming that 
> they represent lakes.

I think this would help Mapnik, although like I mentioned before I have
another proposal for how we could use virtual ways.

> 3. Manually inspect those lakes with the highest number of segments, and 
> try to simplify them manually on a case-by-case basis (some will 
> probably have more detail than necessary, e.g. many nodes in one line, 
> and if not, they can be split up in several adjacent, closed areas).

It would be great if you did this since it will benefit everyone. It is
a shame there is no automated tool for Josm to do this. I've seen
several roads near me which are direct conversions from GPX and have
roughly 10 times the number of nodes and segments then they should. I'm
not volunteering to write such plugin at the moment, i've got enough to
do right now :-)

> And, of course,
> 
> 4. Further discuss how large areas can and should be handled in the future.

Sounds good to me. I believe we do need a better solution to this.

> Bye
> Frederik
>