[OSM-talk] Something wrong with OSM server?

[Andy Robinson]

Steve Chilton wrote: 
>Sent: 03 February 2007 9:47 AM
>To: Nick Hill; talk at openstreetmap.org
>Subject: Re: [OSM-talk] Something wrong with OSM server?
>
>Nick,
>
>Sorry, but could you say that again in plain English!
>Are you recommending changes in working practice to try to alleviate this
>situation meantime?
>

I think what Nick is saying is that there has always been a limit on the
size of the bounding box that you can request from the database. JOSM for
instance doesn't know what that limit is but if it's too big you get a reset
response from the server and nothing delivered to JOSM. This restriction is
there to stop anyone trying to download the whole planet and therefore
hogging all the resource.

What Nick is pointing out is that the size of the bounding box is no
reflection of the quantity of data within that box. Now that we have near
saturation of map data (and GPX data??) in some areas, parts of London etc
etc for instance, then the time taken to process the necessary query is
simply too long and causes a bottleneck for other users.

We saw a similar problem the other week before the rss node edit feed was
turned off for a similar reason, ie too may requests locking the database
tables causing poor performance for the majority. This is partly a
reflection of more editors now being active. 

This is actually very good news, more people have got involved! What we are
now facing is the sure fact that we have a lot more editors creating map
data, a good chunk of these new contributors are probably using the Y!
imagery since the number of GPX uploaders has remained static. As of
yesterday the number of editors active over the last month has risen now to
310 which is a 30% increase in just 2 months! By contrast the number of GPX
uploaders has not changed over the same period and is just under half at 178
users.

Thus my feeling is that we need to address two issues, the first is the need
to decide on a method and size for reasonable database queries and implement
a method to control it.

If this first point shows that to be reliable the queries will effectively
be quite small then we need a plan and budget for scaling the platform to
the next level and then look for funding to achieve that. I think we should
be discussing this anyway now since functions like the rss node edit feed is
something we should look to turn back on.

In the meantime I think it is beneficial to everyone who is pulling data out
to show restraint and keep those bbox sizes small. That is assuming a
greater number of small queries is better than one large one.

Would be interesting to chart query statistics in some way, as I question if
our clamour to update the Osmarender tiles at home map tiles to the most
current map data is having an impact here.

Cheers

Andy

Andy Robinson
Andy_J_Robinson at blueyonder.co.uk

>Cheers
>STEVE
>
>	-----Original Message-----
>	From: talk-bounces at openstreetmap.org on behalf of Nick Hill
>	Sent: Fri 2/2/2007 11:25 PM
>	To: talk at openstreetmap.org
>	Cc:
>	Subject: Re: [OSM-talk] Something wrong with OSM server?
>
>
>
>	Hi All
>
>
>	It seems to me the 'problem' is that in areas like Central London,
we
>have a
>	very high aerial density of nodes and segments.
>
>	The servers are protected by having a bounding box limitation, which
>has up
>	until now, sanitised huge queries and only returned those the
>hardware can
>	reasonably handle.
>
>	Given there is no inherent limit on aerial density of OSM data, and
>given we
>	have areas of extremely high density, It is no longer the case that
>establishing
>	a maximum bounding box is adequate to preserve system integrity.
>We'll need to
>	consider other mechanisms to maintain system integrity.
>
>	We can, of course, upgrade systems, but that alone gives no
>guarantee. We will
>	therefore need to abort queries where the number of nodes exceeds a
>pre-set
>	threshold.
>
>	This means the client will not know apriori if a query will be
>accepted by the
>	API. However, the API could perhaps respond with a series of
bounding
>boxen
>	which the client may like to try instead.
>
>	Perhaps the API can be accessed by a queued, asynchronous system
>where a server
>	is set aside to handle large queries one by one. Perhaps queries
>mailed to the
>	server then returned by mail as and when they are processed.
>
>
>
>	Adrian Frith wrote:
>	> Hi folks,
>	>
>	> Is anyone else having trouble getting through to the server? I
>can't
>	> open the Slippy map or the Java editing applet; I just get
>connection
>	> timeouts. I can get to the wiki without any trouble
>	>
>	> It's very possible that my ISP is screwing something up, so I
>thought
>	> I'd check and see if anyone else is having trouble.
>	>
>	> Thanks,
>	> Adrian
>	>
>	>
>	>
-------------------------------------------------------------------
>-----
>	>
>	> _______________________________________________
>	> talk mailing list
>	> talk at openstreetmap.org
>	> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
>
>	_______________________________________________
>	talk mailing list
>	talk at openstreetmap.org
>	http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
>
>
>
>_______________________________________________
>talk mailing list
>talk at openstreetmap.org
>http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk