[OSM-dev] Fixing the history tab

Fri May 3 12:55:50 UTC 2013

What exactly do you mean by "slow"? OWL API response is slow or
client-side Javascript is slowing down the browser (likely because of
the number of GeoJSON features displayed at once)?

The OWL API performance issues is indeed the most challenging problem to
solve here. There are couple of things:

1. Database size - as the number of changesets grows, the number of
vector tiles grows as well. So far there is several millions of
changesets in the database and scale factor is not too bad actually -
for high zoom levels it is performing pretty well. For low zoom levels
and especially on frequently modified areas there need to be
optimizations. All in all, I'm not concerned with the amount of data -
even the basic solution as it is right now (using btree index in
Postgres) is acceptable for most use cases - or at least it is not
blocking for the production deployment.

2. Selecting last X changesets on low zoom levels where there is a lot
of changesets present - this is indeed a very big problem because btree
index helps only to narrow down the geographical aspect but the temporal
aspect cannot be indexed and there is a sequential sort/scan - this is
slow when you want to display last 20 changesets at zoom level 8 for
London for instance.

There is a couple of things here to research/try:

* PostgreSQL now supports k nearest neighbors indexing[1][2] - the
distance function could be defined 

* Matt suggested looking into buffer trees - haven't spent too much time
on it myself yet but it looks interesting and tackles exactly the
problem in OWL.

Of course there may be other ways to optimize this and it is exactly my
point - that people could try to solve this problem - I don't have a
ready answer for it because it *is* a hard problem. Once solved, there
would be a great new (and performant) way to look at OWL data.

There are more things in OWL API that I introduced before I went
inactive - now there are generic vector tiles, way "subversions" so
basically you can already build on top of that and create Kothic browser
for OSM data. It is definitely not easy and more work is needed on the
OWL API part to serve proper Kothic tiles and also only the data that
makes sense for given zoom levels etc. but it is all doable - all it
needs is some work = time.

[1]
http://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.1#K-Nearest-Neighbor_Indexing
[2] excoventures.com/talks/knn.pdf‎

Paweł

On Fri, May 3, 2013, at 14:21, Alex Barth wrote:
> Pawel, i do not understand how the New History Tab will scale and how it
> will be fast. Right now it's slow.
> 
> The old owl fell over b/c of scale problems and my understanding is that
> the underlying technical challenge of how to make massive changesets
> browsable in rev chronological order on a map is unsolved.  I must be
> missing something, I don't think you have explained this anywhere.
> 
> I'd realy like to understand, where can I read up?
> 
> On Friday, May 3, 2013, Paweł Paprota wrote:
> 
> > IMHO people really should put their minds to fixing *the* problem of
> > "Querying all changesets that actually modified data in an arbitrary
> > bounding box of the world and displaying them in reverse chronological
> > order is computationally expensive" instead of coming up with yet another
> > half way solution.
> >
> > It looks nice as such tools often do but does not solve the problem
> > properly - why not use the time to help out with OWL / New History Tab
> > instead of starting another project from scratch?
> >
> > Anyway, let me use this opportunity to give people a quick update about
> > OWL and the History Tab Beta.
> >
> > As is painfully obvious from my "Contributions" graphic on Github[1], I've
> > not been active at all in OSM since late February. Coincidentally, I
> > started at my new job at that time...
> >
> > Currently I am on a two-month work trip in Germany with not much free time
> > but I really miss OSM and OWL development so I plan to get back to it some
> > time in June when I'm back at home.
> >
> > When you look at it, there is really not a lot of stuff to be done before
> > OWL and New History Tab can be rolled out to production. Mainly there are
> > UI tweaks to be done so it looks nicer - that's why I'm a bit sad that no
> > one volunteered to help out with JS/CSS - it's the easy part of the
> > project...
> >
> > On the backend side (OWL), there is support for relation to be added which
> > is a gigantic challenge but I think this could be scheduled for 2.0...
> >
> > In any case, please consider helping out with the project, especially the
> > frontend side, as it is not that far from being in
> > potentially-mergable-state.
> >
> > [1] https://github.com/ppawel
> >
> > Paweł
> >
> > On 05/02/2013 05:18 PM, Alex Barth wrote:
> >
> >> At this weekend's Chicago hack weekend Tom and I worked on a prototype
> >> that could be a viable solution for our currently broken history tab. It
> >> is taking a very different approach in comparison to Pawel's history tab
> >> [1] by not showing the entire history up front, but only latest changes
> >> to visible elements. I wrote up the details in a diary entry, would love
> >> to hear peoples thoughts on this. I think from a user story perspective
> >> this would work and it would be much cheaper to implement than a fast
> >> historic changeset browser.
> >>
> >> http://www.openstreetmap.org/**user/lxbarth/diary/19185<http://www.openstreetmap.org/user/lxbarth/diary/19185>
> >>
> >> Alex
> >>
> >> [1] http://lists.openstreetmap.**org/pipermail/talk/2013-**
> >> January/065556.html<http://lists.openstreetmap.org/pipermail/talk/2013-January/065556.html>
> >>
> >>
> >> ______________________________**_________________
> >> dev mailing list
> >> dev at openstreetmap.org
> >> http://lists.openstreetmap.**org/listinfo/dev<http://lists.openstreetmap.org/listinfo/dev>
> >>
> >>
> >
> > ______________________________**_________________
> > dev mailing list
> > dev at openstreetmap.org
> > http://lists.openstreetmap.**org/listinfo/dev<http://lists.openstreetmap.org/listinfo/dev>
> >