[OSM-dev] C++ implementation of the API

Alex Wilson alex_wilson at pobox.com
Mon May 26 16:28:57 BST 2008


Re-implementing the API in C++ will hopefully give a constant-time speedup
(x5?), serving the data from a cluster of machines offers a more scalable
solution. To me both seem worthwhile. The former is (imho) a fairly easy
task and will buy some time. The latter seems perhaps harder, but clearly if
the contributing userbase of OSM continues to grow it sounds like it will
also be needed. However I can't see that the two are mutually exclusive -
particularly if the latter can be achieved with mirrors and url rewriting
when presumably any implementation of the API would do the job.

Alex

2008/5/26 Ludwig <ludwigbrinckmann at gmail.com>:

> My concern is that even if a reimplementation of the API in some other form
> will not fundamentally solve the scalability problem if all data requests
> will have to be served by a single machine.
>
> Will not the introduction of the 0.6 API offer a solution in the sense that
> will will with far greater reliability allow the separation of read and
> write requests? With versioned map elements all read requests could be
> redirected to some mirror server, while only write requests will continue to
> hit the real (master) server. With read requests removed from the main
> server, it should be able to handle serve more active mappers better than at
> the moment.
>
> Mirror servers, in the form of OSMXAPI (which claims to be only a few
> minutes behind the real DB nowadays), could then handle the bulk of the
> requests.
>
> Could the effort for a API reimplementation be directed to some sort of
> smart mirror service using HTTP redirects, which would shield client
> applications from the existence of the mirror services?
> A first version could redirect read-request to a single read-only server,
> while future versions could smarten up and be specific on bounding boxes,
> allowing for a regionalisation of mirror servers, which would not have to
> host the entire world...
> Much of that could probably be achieved by URL rewriting in Apache.
>
>
> Ludwig
>
>
>
>
> 2008/5/26 Tom Hughes <tom at compton.nu>:
>
> In message <1211658532.3899.18.camel at test.asus>
>>           Joachim Zobel <jzobel at heute-morgen.de> wrote:
>>
>> > Am Donnerstag, den 22.05.2008, 00:06 +0100 schrieb Tom Hughes:
>> > > > Writing Apache modules in C is hard, and I don't think using mod_cpp
>> > > > will make it much easier. Doing Apache modules in Perl (mod_perl has
>> > > API
>> > > > access including filters) is a lot easier.
>> > >
>> > > Ye gads no. We want to keep the memory footprint under some sort of
>> > > control so mod_perl is a non-starter.
>> >
>> > Everybody seems obsessed with low memory footprint. Why? Memory is
>> > cheap, and we are not planning an embedded system. What is the number of
>> > concurrent requests we need to handle?
>>
>> I'm not obsesses with memory footprint, but with our current code
>> memory is far and away the biggest problem.
>>
>> Currently we can handle about six simultaneous requests to the data
>> heavy API calls - that is basically map data downloads, gps data
>> downloads and potlatch calls.
>>
>> The reason is that we have to allow about 600Mb or so for each call
>> and that quickly mounts up as you try and add extra simultaneous
>> accesses.
>>
>> Factor in the fact that many of these calls may take a minute or
>> more and you have a serious limitation to the load we can support.
>>
>> Of course we can just add hardware, but we should be able to reduce
>> the menory footprint dramatically as well.
>>
>> Tom
>>
>> --
>> Tom Hughes (tom at compton.nu)
>> http://www.compton.nu/
>>
>> _______________________________________________
>> dev mailing list
>> dev at openstreetmap.org
>> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20080526/52ab38f3/attachment.html>


More information about the dev mailing list