[OSM-dev] TRAPI status

[Brett Henderson]

On Fri, Mar 12, 2010 at 11:24 PM, Lars Francke <lars.francke at gmail.com>wrote:

>
> > Most OSM systems tend to have a large number of disorganised and
> > uncontrolled clients.  Does this work well with the AMQP paradigm?  In
> other
> > words, does it take administrative overhead to register new subscriptions
> to
> > a queue?  What happens if large numbers of subscriptions are created then
> > the clients disappear?  Is AMQP targetted at a world where the clients
> are
> > relatively controlled and small in number?  It's important to minimise
> > administration overhead where possible.
>
> I'll try to answer those in order.
> Subscriptions in AMQP are defined client side[1] so there is _no_
> administrative overhead at all. Clients only need to know which
> exchange/queue they should bind to and that's just a string they need
> to know.
>
> Queues can be declared in different ways, one way is to declare it as
> autoDelete[2]: "true if we are declaring an autodelete queue (server
> will delete it when no longer in use)". Again no administrative
> overhead. It just disappears when no longer used so no messages are
> routed there. So depending on the use case there are multiple options.
>
> I have no huge real-world experience with AMQP but I believe it is
> targeted at both worlds: Small and controlled and large and
> uncontrolled. The structure they have found seems to work really well
> in any case.
>
> Since the installation of the RabbitMQ server I did not have to touch
> it again. The only thing one _could_ do is to add some kind of user
> management (currently everyone is allowed to do everything for example
> send fake messages[4]) to allow only certain users to publish to
> certain exchanges but that are two lines in the admin console.
>

Okay, that all sounds promising.  My main queueing experience is with MQ
Series, or WebSphere MQ, or whatever it's called these days.  It's a little
more top heavy :-)


>
> > Clients will experience outages whether that be due to a network problem,
> > server reboot, or just not running a system 24x7.  Presumably they need a
> > way to catch up on missed events.  There are a few options: 1. The server
> > holds events for clients until they become available again, 2. The client
> > catches up using an out of band mechanism (eg. downloads diffs directly),
> or
> > 3. The client can request that the server begin sending data from a
> specific
> > point.  I think that only options 1 and 2 are possible using AMQP.  1 is
> not
> > scalable, and 2 adds additional client complexity.  3 is what I'd like to
> > see, but I don't think it can be done using a typical reliable messaging
> > system such as AMQP.  I hope I'm wrong though.
>
> Options 1) and 3) are kind of interchangeable and it depends on the
> circumstances which one to use. Both are possible.
>
> I'm glad to inform you that you are indeed kind of wrong. AMQP is used
> for this kind of things through private "unnamed" temporary reply
> queues. An example: Program A needs all diffs since time T so it
> creates a private reply queue the server generates a temporary name
> and then it just sends a message to the "Query queue" with the routing
> key "osm.query.osc" and the payload is just the timestamp (or any
> extra options one might think of) with the "replyTo"[3] field set to
> the previously created private queue. Some (might be a different one
> depending on the query kind, could for example decide dynamically to
> use an API or a XAPI reply handler depending on the request) process
> reads the message and begins sending the reply on the private queue
> which is automatically destroyed at the end.
>

Okay, that sounds workable.  Perhaps a little more complicated than a HTTP
or raw socket request, but that's not necessarily bad.


> > Something to note about the current replication mechanism is that it
> doesn't
> > use any transactional capabilities other than creating files in a
> particular
> > order.  All replication state tracking is client side where transactions
> are
> > actually occurring (eg. writing to a database, updating a planet file,
> etc)
> > which keeps the server highly scalable and agnostic of client
> reliability.
>
> I didn't mean to say that what you're doing is wrong or bad in any
> way! I'm sorry if it came across as if I want to disregard everything
> you have done with Osmosis. The replication diffs are a huge step
> forward.
>

It's all fine :-)  I'm more than happy to see alternative ideas out there.
And to be perfectly honest, if a better system comes out of it and Osmosis
replication can be retired then I'm happy to reclaim a bit of my life back
;-)


>
> > I don't know how you'd hook into the Ruby API effectively and reliably.
> You
> > can't just wait for changeset closure events because changesets can
> remain
> > open for large periods of time.  You really want to be replicating data
> as
> > soon as possible after it becomes available to API queries.  This may
> mean
> > receiving notification about every single entity as it is created,
> modified
> > or deleted from the db, but this will result in huge numbers of events
> which
> > will be difficult to process in an efficient manner.
>
> That's exactly what I wanted to do: Send one message for each
> successful API call (create, update, delete). Yes it'll be a lot of
> messages but nothing even remotely (judging by the minutely diffs)
> hard to handle. Those messages should of course not be processed on
> the API server but if the messages are routed to the dev server it
> shouldn't have a huge impact. It would obviously put _some_ burden on
> the API server(s) but we'd gain a lot of flexibility and
> opportunities. We'd certainly be the first big open source project
> that has this kind of (for lack of a better term) fire-hose stream of
> its data. All it takes to make it even more accessible is for someone
> to write the PubSubHubBub/XMPP implementation.
>

Yep, okay.  I'm not 100% convinced this is the right approach, but again I'm
more than happy to see what comes out of it.

In the meantime, if you do need something more regular than minute updates,
the existing Osmosis mechanism should be able to get down to around 5 second
intervals.  I'm not keen to publish files publicly with that type of
interval, but a second process consuming them on the same server (or one in
close proximity) should be fine.


>
> As to the closing of changesets I thought of another small nicety
> (which I've not yet implemented so this might still fall apart): I
> planed a tool that subscribes to all the API messages and just holds a
> list of timers (even with a million open changesets at a time it
> shouldn't be a problem) which are adjusted accordingly. So when it
> gets a "changeset open" message it creates a timer that expires on the
> "closed at" time of the changeset and that is reset on every change to
> the changeset (of course factoring in the limitations like 24h, 50000
> elements). At some time the timer will fire and it can check the API
> to fetch all the associated metadata and send a message "changeset
> close". This would have the benefit that there is only one call to the
> API for every opened changeset (which I hope would be okay, load wise)
> and all consumers can profit from this information.
>

I didn't understand all of this to be honest.  I think I understand how
existing changesets work, and how they're auto-closed after intervals.  But
I'm not sure where your idea fits into that.


>
> > I also think you'll run into a fair bit of
> > resistance trying to incorporate changes into the Ruby API, it's simpler
> at
> > least to remain independent where possible.  Unless you want to achieve
> > sub-second replication, the current approach could be run with a very
> short
> > replication interval.  The main restriction on replication interval now
> is
> > downloading large numbers of files from the planet server, not the
> > extraction of data from the database.
>
> Yes I think that's going to be the problem (incorporate it into the
> API). That's why I currently implement it by running of the diff files
> you produce. That generates the same messages and allows for a very
> realistic testing scenario while at the same time not having to change
> anything in the API for now.
>

Yep, it will be easier to include something if you have a working prototype
first.  Having a client base ready to go will also help.


>
> As I've said before: We thought about this solely for the dev servers
> (and the wikimedia toolserver(s)) but I've since come to believe that
> something(!) like this would be very nice and an innovative step for
> OpenStreetMap's future.
>
> > I guess something to consider is who are the clients of the mechanism.
> > Somebody wanting to see activity in a geographical area may not care
> about
> > reliability and perhaps something like XMPP is appropriate here.  But
> > anybody wanting reliable replication (ie. TRAPI) will need something
> robust
> > that guarantees delivery and data ordering.
>
> I agree. AMQP is the solution I chose but there are of course other
> ways. AMQP allows for robust, reliable and ordered delivery of
> messages (it even has transactions if needed) and XMPP and others can
> be (easily from what I've read, not done it myself) integrated into
> this.
>
> > Anyway, it's good to hear that some fresh minds are interested in the
> > problem of changeset distribution.  I'm very interested to hear what
> comes
> > out of it.
>
> Me too :)
> Thanks for your comments. It's always good to get a set of fresh eyes
> on an idea and I'd be glad to listen to any further input you (and
> others) might have.
>

Good luck :-)

Cheers,
Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100314/33a14aad/attachment.html>